如何从Scrapy网站获取所有纯文本？

Question

在呈现HTML之后,我希望从网站上看到所有文本.我在Python中使用Scrapy框架.随着xpath('//body//text()')我能够得到它,但与HTML标记,而我只想要的文字.对此有何解决方案？谢谢 !

Answer 1

最简单的办法是与一切发现:extract //body//text()join

''.join(sel.select("//body//text()").extract()).strip()

哪个sel是Selector实例.

另一个选择是使用nltk's clean_html():

>>> import nltk
>>> html = """
... <div class="post-text" itemprop="description">
... 
...         <p>I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
... With <code>xpath('//body//text()')</code> I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !</p>
... 
...     </div>"""
>>> nltk.clean_html(html)
"I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.\nWith xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !"

另一个选择是使用BeautifulSoup's get_text():

get_text()

如果您只需要文档或标记的文本部分,则可以使用该get_text()方法.它返回文档中或标记下的所有文本,作为单个Unicode字符串.

>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(html)
>>> print soup.get_text().strip()
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !

另一个选择是使用lxml.html's text_content():

.text_content()

返回元素的文本内容,包括其子元素的文本内容,没有标记.

>>> import lxml.html
>>> tree = lxml.html.fromstring(html)
>>> print tree.text_content().strip()
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !