jay*_*ing 5 python scrapy parsel
我正在使用response.xpath('//*')re_first()提取js数据,然后将其转换为python本机数据.问题是extract/re方法似乎没有提供一种不取消引用html的方法
原始html:
{my_fields:['O'Connor Park'], }
提取输出:
{my_fields:['O'Connor Park'], }
将此输出转换为json将无法正常工作.
最简单的方法是什么？
pau*_*rth 10
简短回答:
.re()和.re_first()方法取代HTML实体(除<,&).extract()或.extract_first()获取原始HTML(或原始JavaScript指令)并re在提取的字符串上使用Python的模块答案很长:
让我们看一下示例输入和从HTML中提取Javascript数据的各种方法.
示例HTML:
<html lang="en">
<body>
<div>
<script type="text/javascript">
var i = {a:['O'Connor Park']}
</script>
</div>
</body>
</html>
使用scrapy Selector,它使用下面的parsel库,你有几种方法可以解压缩Javascript片段:
>>> import scrapy
>>> t = """<html lang="en">
... <body>
... <div>
... <script type="text/javascript">
... var i = {a:['O'Connor Park']}
... </script>
...
... </div>
... </body>
... </html>
... """
>>> selector = scrapy.Selector(text=t, type="html")
>>>
>>> # extracting the <script> element as raw HTML
>>> selector.xpath('//div/script').extract_first()
u'<script type="text/javascript">\n var i = {a:[\'O'Connor Park\']}\n </script>'
>>>
>>> # only getting the text node inside the <script> element
>>> selector.xpath('//div/script/text()').extract_first()
u"\n var i = {a:['O'Connor Park']}\n "
>>>
现在,使用.re(或.re_first)你得到不同的结果:
>>> # I'm using a very simple "catch-all" regex
>>> # you are probably using a regex to extract
>>> # that specific "O'Connor Park" string
>>> selector.xpath('//div/script/text()').re_first('.+')
u" var i = {a:['O'Connor Park']}"
>>>
>>> # .re() on the element itself, one needs to handle newlines
>>> selector.xpath('//div/script').re_first('.+')
u'<script type="text/javascript">' # only first line extracted
>>> import re
>>> selector.xpath('//div/script').re_first(re.compile('.+', re.DOTALL))
u'<script type="text/javascript">\n var i = {a:[\'O\'Connor Park\']}\n </script>'
>>>
HTML实体'已被撇号取代.这是由于实现中的w3lib.html.replace_entities()调用.re/re_first(参见函数中的parsel源代码extract_regex),仅在调用extract()或时使用extract_first()
| 归档时间： |
|
| 查看次数： |
5708 次 |
| 最近记录： |