如何打开此XML文件以在Python中创建数据框？

Question

有没有人建议在下面的网站上打开xml数据的最佳方法是将它放在python中的数据框(我更喜欢使用pandas)？该文件位于此站点上的"Data - XML(sdmx/zip)"链接中:

http://www.federalreserve.gov/pubs/feds/2006/200628/200628abs.html

我试过从http://timhomelab.blogspot.com/2014/01/how-to-read-xml-file-into-dataframe.html复制使用以下内容,似乎我越来越近了:

from lxml import objectify
import pandas as pd

path = 'feds200628.xml'
xml = objectify.parse(open(path))
root = xml.getroot()
root.getchildren()[0].getchildren()
df = pd.DataFrame(columns=('id', 'name'))

for i in range(0,4):
    obj = root.getchildren()[i].getchildren()
    row = dict(zip(['id', 'name'], [obj[0].text, obj[1].text]))
    row_s = pd.Series(row)
    row_s.name = i
    df = df.append(row_s)

尽管如此,我还不太了解xml让我完成剩下的工作.

任何帮助都很棒 - 我甚至不需要它在数据框中,我只需要弄清楚如何以某种方式在python中解析这个内容.

Answer 1

XML是一种树状结构,而Pandas DataFrame是一种类似于2D的表结构.所以没有自动方式在两者之间进行转换.您必须了解XML结构并知道如何将其数据映射到2D表.因此,每个XML到DataFrame的问题都是不同的.

您的XML有2个DataSet,每个包含许多Series.每个系列都包含许多Obs元素.

每个Series都有一个NAME属性,每个Obs都有OBS_STATUS,TIME_PERIOD和OBS_VALUE属性.因此,使用NAME,OBS_STATUS,TIME_PERIOD和OBS_VALUE列创建表可能是合理的.

我发现从XML中提取所需的数据有点复杂,这使我怀疑我找到了最好的方法.但这是一种方式(PS.Thomas Maloney的想法从2D表格式XLS数据开始应该更简单):

import lxml.etree as ET
import pandas as pd

path = 'feds200628.xml'

def fast_iter(context, func, *args, **kwargs):
    """
    http://lxml.de/parsing.html#modifying-the-tree
    Based on Liza Daly's fast_iter
    http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
    See also http://effbot.org/zone/element-iterparse.htm
    http://stackoverflow.com/a/7171543/190597 (unutbu)
    """
    for event, elem in context:
        func(elem, *args, **kwargs)
        # It's safe to call clear() here because no descendants will be
        # accessed
        elem.clear()
        # Also eliminate now-empty references from the root node to elem
        for ancestor in elem.xpath('ancestor-or-self::*'):
            while ancestor.getprevious() is not None:
                del ancestor.getparent()[0]
    del context

data = list()
obs_keys = ['OBS_STATUS', 'TIME_PERIOD', 'OBS_VALUE']
columns = ['NAME'] + obs_keys

def process_obs(elem, name):
    dct = elem.attrib
    # print(dct)
    data.append([name] + [dct[key] for key in obs_keys])

def process_series(elem):
    dct = elem.attrib
    # print(dct)
    context = ET.iterwalk(
        elem, events=('end', ),
        tag='{http://www.federalreserve.gov/structure/compact/common}Obs'
        )
    fast_iter(context, process_obs, dct['SERIES_NAME'])

def process_dataset(elem):
    nsmap = elem.nsmap
    # print(nsmap)
    context = ET.iterwalk(
        elem, events=('end', ),
        tag='{{{prefix}}}Series'.format(prefix=elem.nsmap['kf'])
        )
    fast_iter(context, process_series)

with open(path, 'rb') as f:
    context = ET.iterparse(
        f, events=('end', ),
        tag='{http://www.federalreserve.gov/structure/compact/common}DataSet'
        )
    fast_iter(context, process_dataset)
    df = pd.DataFrame(data, columns=columns)

产量

            NAME OBS_STATUS TIME_PERIOD   OBS_VALUE
0        SVENY01          A  1961-06-14      2.9825
1        SVENY01          A  1961-06-15      2.9941
2        SVENY01          A  1961-06-16      3.0012
3        SVENY01          A  1961-06-19      2.9949
4        SVENY01          A  1961-06-20      2.9833
5        SVENY01          A  1961-06-21      2.9993
6        SVENY01          A  1961-06-22      2.9837
...
1029410     TAU2          A  2014-09-19  3.72896779
1029411     TAU2          A  2014-09-22  3.12836171
1029412     TAU2          A  2014-09-23  3.20146575
1029413     TAU2          A  2014-09-24  3.29972110

Answer 2

我会将XLS格式的文件导出为CSV文件(使用免费的程序,如Gnumeric或LibreOffice,或者如果你有,Excel),然后将CSV文件读入pandas.我知道这不是你最后一个问题的答案,但解析XML对你正在尝试做的事情来说是一个过于复杂的解决方案.

关于在Python中解析XML,lxml库是我最喜欢的库.我发现使用XPath查询语言和lxml解析器是最佳路由.