如何在 Pandas 中分解嵌套的 json 结构？

Question

我正在处理一个每行包含一个 json 块的文件。每一行看起来像这样：

{"a":3,"b":10,"unnecessaryList":[{"value":12,"colName":"c"},{"value":792,"colName":"d"},{"value":645,"colName":"e"}],"index":"-1417561653"}

json 的生产者选择了一个不必要的嵌套结构，而平面结构就完全足够了。也就是说，我想以更明显的扁平结构将数据读入 Pandas DataFrame，该结构具有列“a”、“b”、“c”、“d”、“e”、“index”。到目前为止，我想出的最好的方法是以不同的方式处理文件两次：

import pandas as pd
from pandas.io.json import json_normalize, loads

raw_json = pd.read_json('sample.json', lines=True)
raw_json.set_index('index', inplace=True)

with open('sample.json') as f:
    lines = f.readlines()
    exploded_columns = pd.concat([json_normalize(loads(l), 'unnecessaryList', 'index').pivot(index='index', columns='colName', values='value') for l in lines])

data = pd.merge(raw_json[['a', 'b']], exploded_columns, left_index=True, right_index=True)

有没有办法避免像这样读取数据两次？Pandas 是否提供了一些可以避免我想出的 concat/normalize/pivot/merge 垃圾的功能？

Answer 1

您可以使用：

df = pd.read_json('sample.json', lines=True)
#create Multiindex from 3 columns and select unnecessaryList for Series
s = df.set_index(['a','b','index'])['unnecessaryList']
print (s)
a  b   index      
3  10  -1417561653    [{'value': 12, 'colName': 'c'}, {'value': 792,...
       -1417561655    [{'value': 13, 'colName': 'c'}, {'value': 794,...
       -1417561658    [{'value': 14, 'colName': 'c'}, {'value': 795,...
Name: unnecessaryList, dtype: object

#create DataFrame for each dict and concat, transpose
L = [pd.DataFrame(x).set_index('colName')['value'] for x in s]
df = (pd.concat(L, axis=1, keys=s.index)
       .T
       .reset_index(level=[0,1])
       .rename(columns={'level_0':'a','level_1':'b'})
       .rename_axis(None, 1))
print (df)
             a   b   c    d    e
-1417561653  3  10  12  792  645
-1417561655  3  10  13  794  645
-1417561658  3  10  14  795  645

在json中输入数据：

{"a":3,"b":10,"unnecessaryList":[{"value":12,"colName":"c"},{"value":792,"colName":"d"},{"value":645,"colName":"e"}],"index":"-1417561653"}
{"a":3,"b":10,"unnecessaryList":[{"value":13,"colName":"c"},{"value":794,"colName":"d"},{"value":645,"colName":"e"}],"index":"-1417561655"}
{"a":3,"b":10,"unnecessaryList":[{"value":14,"colName":"c"},{"value":795,"colName":"d"},{"value":645,"colName":"e"}],"index":"-1417561658"}