从txt文件解析url

Question

我试图解析一个如下所示的txt文件:

Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html

我需要读取文件并在"Disallow"之后用url提取部分,但也忽略了注释.提前致谢.

Answer 1

如果您尝试解析robots.txt文件,则应使用robotparser模块:

>>> import robotparser

>>> r = robotparser.RobotFileParser()
>>> r.set_url("http://www.your_url.com/robots.txt")
>>> r.read()

然后检查:

>>> r.can_fetch("*", "/foo.html")
False