在beautifulsoup的上下文中,lxml和html5lib解析器的功能有区别吗?我正在尝试学习使用BS4并使用以下代码构造 -
ret = requests.get('http://www.olivegarden.com')
soup = BeautifulSoup(ret.text, 'html5lib')
for item in soup.find_all('a'):
print item['href']
Run Code Online (Sandbox Code Playgroud)
我开始使用lxml作为解析器,但注意到对于某些网站,即使页面中有有效链接,也永远不会输入for循环.同一页面适用于html5ib解析器.是否有任何特定类型的页面可能无法与lxml一起使用?
我在Ubuntu上使用python-lxml 2.3.2-1和libxml2 2.7.8.dfsg-5.1ubunt和html5lib-1.0b3
编辑:我更新到lxml 3.1.2仍然看到相同的问题.在运行3.0.x的mac上,正在正确解析同一页面.有问题的网站是www.olivegarden.com
我正在使用html5lib并在将其更新到最新版本后,我不断收到此错误:
Traceback (most recent call last):
File "/home/travis/build/freelawproject/juriscraper/tests/test_everything.py", line 119, in test_scrape_all_example_files
site.parse()
File "/home/travis/build/freelawproject/juriscraper/juriscraper/AbstractSite.py", line 95, in parse
self.html = self._download()
File "/home/travis/build/freelawproject/juriscraper/juriscraper/AbstractSite.py", line 384, in _download
html_tree = self._make_html_tree(text)
File "/home/travis/build/freelawproject/juriscraper/juriscraper/opinions/united_states/federal_appellate/ca11_u.py", line 26, in _make_html_tree
e = html5parser.document_fromstring(text)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/lxml/html/html5parser.py", line 64, in document_fromstring
return parser.parse(html, useChardet=guess_charset).getroot()
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/html5lib/html5parser.py", line 235, in parse
self._parse(stream, False, None, *args, **kwargs)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/html5lib/html5parser.py", line 85, in _parse
self.tokenizer = _tokenizer.HTMLTokenizer(stream, parser=self, **kwargs)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/html5lib/_tokenizer.py", line 36, in __init__
self.stream = HTMLInputStream(stream, …Run Code Online (Sandbox Code Playgroud) 我正在尝试使用BeautifulSoup从页面获取结果:
req_url = 'http://www.xscores.com/soccer/livescores/25-02'
request = requests.get(req_url)
content = request.content
soup = BeautifulSoup(content, "html.parser")
scores = soup.find_all('tr', {'style': 'height:18px;'}, limit=None)
print(len(scores))
>50
Run Code Online (Sandbox Code Playgroud)
我读了之前的解决方案:美丽的汤findAll找不到所有 ,我尝试了html.parser,lxml和html5lib,但没有一个返回超过50个结果.有什么建议?
谢谢
我已经安装了 html5lib 包。我确定,因为当我尝试安装它时,我收到一条消息,表明它已安装。
pip install html5lib
Requirement already satisfied: html5lib in ./anaconda/lib/python3.5/site-packages
Run Code Online (Sandbox Code Playgroud)
我也能够成功导入该包。
import html5lib
Run Code Online (Sandbox Code Playgroud)
然而,当我尝试使用解析器作为 BeautifulSoup 构造函数的一部分时
soup = BeautifulSoup(response.data, 'html5lib')
Run Code Online (Sandbox Code Playgroud)
我不成功
FeatureNotFound: Couldn't find a tree builder with the features you requested: html5lib. Do you need to install a parser library?
Run Code Online (Sandbox Code Playgroud)
是否可以强制 bs4 查找已安装的解析器并成功找到 html5lib?
我有这个不起作用的基本代码。如何在 html5lib php 中使用 Xpath?或以任何其他方式使用 HTML5 的 Xpath。
$url = 'http://en.wikipedia.org/wiki/PHP';
$response = GuzzleHttp\get($url);
$html5 = new Masterminds\HTML5();
$dom = $html5->loadHTML($response);
$xpath = new DOMXPath($dom);
$elements = $xpath->query('//h1');
//$elements = $dom->getElementsByTagName('h1');
foreach ($elements as $element)
{
var_dump($element);
}
Run Code Online (Sandbox Code Playgroud)
未找到任何元素。使用$xpath->query('.')作品来获取根元素(通常 xpath 似乎有效)。$dom->getElementsByTagName('h1')正在工作。
我正在尝试安装html5lib.起初我试图安装最新版本(8或9个9),但它与我的BeautifulSoup发生冲突,所以我决定尝试更老的版本(0.9999999,7 个9).我安装了它,但是当我尝试使用它时:
>>> with urlopen("http://example.com/") as f:
document = html5lib.parse(f, encoding=f.info().get_content_charset())
Run Code Online (Sandbox Code Playgroud)
我收到一个错误:
Traceback (most recent call last):
File "<pyshell#11>", line 2, in <module>
document = html5lib.parse(f, encoding=f.info().get_content_charset())
File "C:\Python\Python35-32\lib\site-packages\html5lib\html5parser.py", line 35, in parse
return p.parse(doc, **kwargs)
File "C:\Python\Python35-32\lib\site-packages\html5lib\html5parser.py", line 235, in parse
self._parse(stream, False, None, *args, **kwargs)
File "C:\Python\Python35-32\lib\site-packages\html5lib\html5parser.py", line 85, in _parse
self.tokenizer = _tokenizer.HTMLTokenizer(stream, parser=self, **kwargs)
File "C:\Python\Python35-32\lib\site-packages\html5lib\_tokenizer.py", line 36, in __init__
self.stream = HTMLInputStream(stream, **kwargs)
File "C:\Python\Python35-32\lib\site-packages\html5lib\_inputstream.py", line 151, in HTMLInputStream
return HTMLBinaryInputStream(source, **kwargs) …Run Code Online (Sandbox Code Playgroud) 我突然无法启动Django服务器了,运行检查:
python manage.py check
Run Code Online (Sandbox Code Playgroud)
显示以下错误:
apps.populate(settings.INSTALLED_APPS)
File "/home/aym/.virtualenvs/prod_env/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
app_config.import_models(all_models)
File "/home/aym/.virtualenvs/prod_env/local/lib/python2.7/site-packages/django/apps/config.py", line 198, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/aym/workspace/CompanyP/apps/account/models.py", line 18, in <module>
from apps.groups.models import SubscriptionGroup
File "/home/aym/workspace/CompanyP/apps/subscription/models.py", line 8, in <module>
from apps.member.models import Member
File "/home/aym/.virtualenvs/prod_env/local/lib/python2.7/site-packages/bleach/__init__.py", line 5, in <module>
from bleach.linkifier import
File "/home/aym/.virtualenvs/prod_env/local/lib/python2.7/site-packages/bleach/linkifier.py", line 5, in <module>
from html5lib.filters.base import Filter
ImportError: No module named base
Run Code Online (Sandbox Code Playgroud)
你知道吗?
在尝试通过pip安装任何python包时,我在parse()中得到了意外的arg:keyword编码.
我遇到了这个问题,因为我为python 3.6安装了tensorflow,这可能导致了html5lib和setuptools的一些问题.使用tar.gz文件(管理员安装)重新安装了html5lib1.0b10,但问题仍然存在.请帮忙!!
pip install spacy
Collecting spacy
Exception:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
status = self.run(options, args)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\commands\install.py", line 335, in run
wb.build(autobuilding=True)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_set.py", line 554, in _prepare_file
require_hashes
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\index.py", line 423, in find_all_candidates
for page in …Run Code Online (Sandbox Code Playgroud) 我一直在使用优秀的漂白库来删除错误的HTML.
我有一堆HTML文档已经从Microsoft Word粘贴,包含以下内容:
<STYLE> st1:*{behavior:url(#ieooui) } </STYLE>
Run Code Online (Sandbox Code Playgroud)
使用漂白(style隐藏不允许标记),让我:
st1:*{behavior:url(#ieooui) }
Run Code Online (Sandbox Code Playgroud)
哪个没用.Bleach似乎只有以下选项:
我正在寻找第三种选择 - 删除标签及其内容.
我正在运行需要的python3程序,html5lib但收到错误消息No module named 'html5lib'。
这是终端的两个会话:
sam@pc ~ $ python
Python 2.7.9 (default, Mar 1 2015, 12:57:24)
[GCC 4.9.2] on linux2
>>> import html5lib
>>> html5lib.__file__
'/usr/local/lib/python2.7/dist-packages/html5lib/__init__.pyc'
>>> quit()
sam@pc ~ $ python3
Python 3.4.2 (default, Oct 8 2014, 10:45:20)
[GCC 4.9.1] on linux
>>> import html5lib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'html5lib'
>>>
Run Code Online (Sandbox Code Playgroud)
问题出在哪里?
html5lib ×10
python ×6
django ×2
lxml ×2
python-3.x ×2
html ×1
importerror ×1
php ×1
pip ×1
setuptools ×1
web-scraping ×1
xpath ×1