如何从Python中的网站中提取表格

Question

这里,

http://www.ffiec.gov/census/report.aspx?year=2011&state=01&report=demographic&msa=11500

这里有张桌子.我的目标是提取表并将其保存到csv文件中.我写了一个代码:

import urllib
import os

web = urllib.urlopen("http://www.ffiec.gov/census/report.aspx?year=2011&state=01&report=demographic&msa=11500")

s = web.read()
web.close()

ff = open(r"D:\ex\python_ex\urllib\output.txt", "w")
ff.write(s)
ff.close()

我从这里输了.谁可以提供帮助呢？谢谢!

Answer 1

Pandas可以直接执行此操作,从而使您不必自己解析HTML.to_html()从html中提取所有表,并将它们放在数据帧列表中. to_csv()可用于将每个数据帧转换为csv文件.对于您示例中的网页,相关表格是最后一个,这就是我df_list[-1]在下面的代码中使用的原因.

import requests
import pandas as pd

url = 'http://www.ffiec.gov/census/report.aspx?year=2011&state=01&report=demographic&msa=11500'
html = requests.get(url).content
df_list = pd.read_html(html)
df = df_list[-1]
print(df)
df.to_csv('my data.csv')

如果您愿意,可以在一行中做到这一点很简单:

pd.read_html(requests.get(<url>).content)[-1].to_csv(<csv file>)

Answer 2

所以基本上你想要解析html文件以获取元素.您可以使用BeautifulSoup或lxml执行此任务.

您已经有使用的解决方案BeautifulSoup.我将使用lxml以下方法发布解决方案:

from lxml import etree
import urllib

web = urllib.urlopen("http://www.ffiec.gov/census/report.aspx?year=2011&state=01&report=demographic&msa=11500")
s = web.read()

html = etree.HTML(s)

## Get all 'tr'
tr_nodes = html.xpath('//table[@id="Report1_dgReportDemographic"]/tr')

## 'th' is inside first 'tr'
header = [i[0].text for i in tr_nodes[0].xpath("th")]

## Get text from rest all 'tr'
td_content = [[td.text for td in tr.xpath('td')] for tr in tr_nodes[1:]]