将分层（树状）XML 读入 Pandas 数据帧，保留层次结构

Question

将分层（树状）XML 读入 Pandas 数据帧，保留层次结构

Dir*_*irk 5 python xml tree hierarchical-data pandas

我有一个包含分层树状结构的 XML 文档，请参见下面的示例。

该文档包含多个<Message>标签（为方便起见，我只复制了其中一个）。

每个<Message>都有自己的一些关联数据 ( id, status, priority)。

此外，每个都<Message>可以包含一个或多个子节点，这些<Street>子节点又具有一些相关数据 ( <name>, <length>)。

此外，每个人都<Street>可以有一个或多个<Link>孩子，这些孩子又拥有自己的相关数据 ( <id>, <direction>)。

XML 文档示例：

<?xml version="1.0" encoding="ISO-8859-1"?>
<Root xmlns="someNamespace">
<Messages>
<Message id='12345'>
   <status>Active</status>
   <priority>Low</priority>
   <Area>
    <Streets>
     <Street>
      <name>King Street</name>
      <length>Short</length>
       <Link>
        <id>75838745</id>
        <direction>North</direction>
       </Link>
       <Link>
        <id>168745</id>
        <direction>South</direction>
       </Link>
       <Link>
        <id>975416</id>
        <direction>North</direction>
       </Link>
     </Street>
     <Street>
      <name>Queen Street</name>
      <length>Long</length>
       <Link>
        <id>366248</id>
         <direction>West</direction>
       </Link>
       <Link>
        <id>745812</id>
         <direction>East</direction>
       </Link>
     </Street>
    </Streets>
   </Area>
</Message>
</Messages>
</Root>

Run Code Online (Sandbox Code Playgroud)

用 Python 解析 XML 并将相关数据存储在变量中不是问题 - 我可以使用例如lxml库并读取整个文档，然后执行一些xpath表达式以获取相关字段，或者使用iterparse方法逐行读取.

但是，我想将数据放入熊猫数据框中，同时保留其中的层次结构。目标是查询单个消息（例如通过布尔表达式，如if status == Active then get the Message with all its streets and its streets' links）并获取属于特定消息的所有数据（其街道及其街道的链接）。这最好怎么做？

我尝试了不同的方法，但都遇到了问题。

如果我创建了一个包含信息，然后设置一个多指标上的每个XML列一个数据帧行[MessageID, StreetName, LinkID]，我得到一个指标，有很多NaN在它（这是一般不提倡），因为MessageID不知道它的孩子streets和links还没有。此外，我不知道如何通过布尔条件选择一些子数据集，而不是只获取一些没有子行的单行。

在对 GroupBy 执行 GroupBy 时[MessageID, StreetName, LinkID]，我不知道如何从 Pandas GroupBy 对象取回（可能是 MultiIndex）数据帧，因为这里没有任何可聚合的内容（没有mean/std/sum/whatsoever，值应该保持不变）。

任何建议如何有效处理？

Answer 1

Dir*_*irk 7

我终于设法解决了上述问题，这就是方法。

我扩展了上面给出的 XML 文档以包含两条消息而不是一条消息。这就是它作为有效 Python 字符串的样子（当然也可以从文件加载）：

xmlDocument = '''<?xml version="1.0" encoding="ISO-8859-1"?> \
<Root> \
<Messages> \
<Message id='12345'> \
   <status>Active</status> \
   <priority>Low</priority> \
   <Area> \
    <Streets> \
     <Street> \
      <name>King Street</name> \
      <length>Short</length> \
       <Link> \
        <id>75838745</id> \
        <direction>North</direction> \
       </Link> \
       <Link> \
        <id>168745</id> \
        <direction>South</direction> \
       </Link> \
       <Link> \
        <id>975416</id> \
        <direction>North</direction> \
       </Link> \
     </Street> \
     <Street> \
      <name>Queen Street</name> \
      <length>Long</length> \
       <Link> \
        <id>366248</id> \
         <direction>West</direction> \
       </Link> \
       <Link> \
        <id>745812</id> \
         <direction>East</direction> \
       </Link> \
     </Street> \
    </Streets> \
   </Area> \
</Message> \
<Message id='54321'> \
   <status>Inactive</status> \
   <priority>High</priority> \
   <Area> \
    <Streets> \
     <Street> \
      <name>Princess Street</name> \
      <length>Mid</length> \
       <Link> \
        <id>744154</id> \
        <direction>West</direction> \
       </Link> \
       <Link> \
        <id>632214</id> \
        <direction>South</direction> \
       </Link> \
       <Link> \
        <id>654785</id> \
        <direction>East</direction> \
       </Link> \
     </Street> \
     <Street> \
      <name>Prince Street</name> \
      <length>Very Long</length> \
       <Link> \
        <id>1022444</id> \
         <direction>North</direction> \
       </Link> \
       <Link> \
        <id>4474558</id> \
         <direction>South</direction> \
       </Link> \
     </Street> \
    </Streets> \
   </Area> \
</Message> \
</Messages> \
</Root>'''

Run Code Online (Sandbox Code Playgroud)

为了将分层的 XML 结构解析为扁平的 Pandas 数据帧，我使用了 Python 的 ElementTreeiterparse方法，该方法提供了一个类似 SAX 的接口来逐行遍历 XML 文档并在特定的 XML 标记开始或结束时触发事件。

对于每个解析的 XML 行，给定的信息都存储在字典中。使用了三个字典，一个用于以某种方式属于一起的每组数据（消息、街道、链接），稍后将存储在其自己的数据帧行中。当收集到一个这样的行的所有信息时，字典被附加到一个列表中，以适当的顺序存储所有行。

这是 XML 解析的样子（请参阅内联注释以获取进一步说明）：

# imports
import xml.etree.ElementTree as ET
import pandas as pd

# initialize parsing from Bytes buffer
from io import BytesIO
xmlDocument = BytesIO(xmlDocument.encode('utf-8'))

# initialize dictionaries storing the information to each type of row
messageRow, streetRow, linkRow = {}, {}, {}

# initialize list that stores the single dataframe rows
listOfRows = []

# read the xml file line by line and throw signal when specific tags start or end
for event, element in ET.iterparse(xmlDocument, events=('start', 'end')):

    ##########
    # get all information on the current message and store in the appropriate dictionary
    ##########

    # get current message's id attribute
    if event == 'start' and element.tag == 'Message':
        messageRow = {} # re-initialize the dictionary for the current row
        messageRow['messageId'] = element.get('id')

    # get current message's status
    if event == 'end' and element.tag == 'status':
        messageRow['status'] = element.text

    # get current message's priority
    if event == 'end' and element.tag == 'priority':
        messageRow['priority'] = element.text

    # when no more information on the current message is expected, append it to the list of rows
    if event == 'end' and element.tag == 'priority':
        listOfRows.append(messageRow)

    ##########
    # get all information on the current street and store in row dictionary
    ##########

    if event == 'end' and element.tag == 'name':
        streetRow = {} # re-initialize the dictionary for the current street row
        streetRow['streetName'] = element.text

    if event == 'end' and element.tag == 'length':
        streetRow['streetLength'] = element.text

    # when no more information on the current street is expected, append it to the list of rows
    if event == 'end' and element.tag == 'length':

        # link the street to the message it belongs to, then append
        streetRow['messageId'] = messageRow['messageId']
        listOfRows.append(streetRow)

    ##########
    # get all information on the current link and store in row dictionary
    ##########

    if event == 'end' and element.tag == 'id':
        linkRow = {} # re-initialize the dictionary for the current link row
        linkRow['linkId'] = element.text

    if event == 'end' and element.tag == 'direction':
        linkRow['direction'] = element.text

    # when no more information on the current link is expected, append it to the list of rows
    if event == 'end' and element.tag == 'direction':

        # link the link to the message it belongs to, then append
        linkRow['messageId'] = messageRow['messageId']
        listOfRows.append(linkRow)

Run Code Online (Sandbox Code Playgroud)

listOfRows现在是一个字典列表，其中每个字典存储要放入一个数据帧行的信息。可以使用此列表作为数据源创建数据框

# create dataframe from list of rows and pass column order (would be random otherwise)
df = pd.DataFrame.from_records(listOfRows, columns=['messageId', 'status', 'priority', 'streetName', 'streetLength', 'linkId', 'direction'])
print(df)

Run Code Online (Sandbox Code Playgroud)

并给出“原始”数据框：

   messageId    status priority       streetName streetLength    linkId  \
0      12345    Active      Low              NaN          NaN       NaN   
1      12345       NaN      NaN      King Street        Short       NaN   
2      12345       NaN      NaN              NaN          NaN  75838745   
3      12345       NaN      NaN              NaN          NaN    168745   
4      12345       NaN      NaN              NaN          NaN    975416   
5      12345       NaN      NaN     Queen Street         Long       NaN   
6      12345       NaN      NaN              NaN          NaN    366248   
7      12345       NaN      NaN              NaN          NaN    745812   
8      54321  Inactive     High              NaN          NaN       NaN   
9      54321       NaN      NaN  Princess Street          Mid       NaN   
10     54321       NaN      NaN              NaN          NaN    744154   
11     54321       NaN      NaN              NaN          NaN    632214   
12     54321       NaN      NaN              NaN          NaN    654785   
13     54321       NaN      NaN    Prince Street    Very Long       NaN   
14     54321       NaN      NaN              NaN          NaN   1022444   
15     54321       NaN      NaN              NaN          NaN   4474558   

   direction  
0        NaN  
1        NaN  
2      North  
3      South  
4      North  
5        NaN  
6       West  
7       East  
8        NaN  
9        NaN  
10      West  
11     South  
12      East  
13       NaN  
14     North  
15     South

Run Code Online (Sandbox Code Playgroud)

我们现在可以在该数据帧上将感兴趣的列（messageId、streetName、linkId）设置为 MultiIndex：

# set the columns of interest as MultiIndex
df = df.set_index(['messageId', 'streetName', 'linkId'])
print(df)

Run Code Online (Sandbox Code Playgroud)

这使：

                                      status priority streetLength direction
messageId streetName      linkId                                            
12345     NaN             NaN         Active      Low          NaN       NaN
          King Street     NaN            NaN      NaN        Short       NaN
          NaN             75838745       NaN      NaN          NaN     North
                          168745         NaN      NaN          NaN     South
                          975416         NaN      NaN          NaN     North
          Queen Street    NaN            NaN      NaN         Long       NaN
          NaN             366248         NaN      NaN          NaN      West
                          745812         NaN      NaN          NaN      East
54321     NaN             NaN       Inactive     High          NaN       NaN
          Princess Street NaN            NaN      NaN          Mid       NaN
          NaN             744154         NaN      NaN          NaN      West
                          632214         NaN      NaN          NaN     South
                          654785         NaN      NaN          NaN      East
          Prince Street   NaN            NaN      NaN    Very Long       NaN
          NaN             1022444        NaN      NaN          NaN     North
                          4474558        NaN      NaN          NaN     South

Run Code Online (Sandbox Code Playgroud)

尽管NaN在一般情况下应该忽略索引中的内容，但对于这个用例，我认为它没有任何问题。

最后，为了获得通过它们的访问单个消息的预期效果messageId，包括其所有“子”街道和链接，MultiIndexed 数据框必须按最外层索引级别进行分组：

# group by the most outer index
groups = df.groupby(level='messageId')

Run Code Online (Sandbox Code Playgroud)

现在，您可以例如循环遍历所有消息（并对它们执行任何操作）

# iterate over all groups
for key, group in groups:
    print('key: ' + key)
    print('group:')
    print(group)
    print('\n')

Run Code Online (Sandbox Code Playgroud)

返回

key: 12345
group:
                                 status priority streetLength direction
messageId streetName   linkId                                          
12345     NaN          NaN       Active      Low          NaN       NaN
          King Street  NaN          NaN      NaN        Short       NaN
          NaN          75838745     NaN      NaN          NaN     North
                       168745       NaN      NaN          NaN     South
                       975416       NaN      NaN          NaN     North
          Queen Street NaN          NaN      NaN         Long       NaN
          NaN          366248       NaN      NaN          NaN      West
                       745812       NaN      NaN          NaN      East


key: 54321
group:
                                     status priority streetLength direction
messageId streetName      linkId                                           
54321     NaN             NaN      Inactive     High          NaN       NaN
          Princess Street NaN           NaN      NaN          Mid       NaN
          NaN             744154        NaN      NaN          NaN      West
                          632214        NaN      NaN          NaN     South
                          654785        NaN      NaN          NaN      East
          Prince Street   NaN           NaN      NaN    Very Long       NaN
          NaN             1022444       NaN      NaN          NaN     North
                          4474558       NaN      NaN          NaN     South

Run Code Online (Sandbox Code Playgroud)

或者您可以通过 messageId 访问特定消息，返回包含 messageId 的行及其所有专用街道和链接：

# get groups by key
print('specific group only:')
print(groups.get_group('54321'))

Run Code Online (Sandbox Code Playgroud)

给

specific group only:
                                     status priority streetLength direction
messageId streetName      linkId                                           
54321     NaN             NaN      Inactive     High          NaN       NaN
          Princess Street NaN           NaN      NaN          Mid       NaN
          NaN             744154        NaN      NaN          NaN      West
                          632214        NaN      NaN          NaN     South
                          654785        NaN      NaN          NaN      East
          Prince Street   NaN           NaN      NaN    Very Long       NaN
          NaN             1022444       NaN      NaN          NaN     North
                          4474558       NaN      NaN          NaN     South

Run Code Online (Sandbox Code Playgroud)

希望这会在某个时候对某人有所帮助。

归档时间：	11 年，7 月前
查看次数：	3117 次
最近记录：	11 年，6 月前