将字典的字典转换为 pandas DataFrame - 内存问题

Asa*_*eed 3 dictionary dataframe python-2.7 pandas

我有一个由三级嵌套字典组成的数据结构,该字典保存三部分对象的出现次数。我想用它构建一个具有特定形状的 DataFrame,但我想不出一种不消耗大量工作内存的方法来实现它——因为表很大(几个完整的 GB)。

基本功能如下所示:

class SparseCubeTable:
    def __init__(self):
        self.table = {}
        self.dim1 = []
        self.dim2 = []
        self.dim3 = []

    def increment(self, dim1, dim2, dim3):
        if dim1 in self.table:
            if dim2 in self.table[dim1]:
                if dim3 in self.table[dim1][dim2]:
                    self.table[dim1][dim2][dim3] += 1
                else:
                    self.dim3.append(dim3)
                    self.table[dim1][dim2][dim3] = 1
            else:
                self.dim2.append(dim2)
                self.dim3.append(dim3)
                self.table[dim1][dim2] = {dim3:1}
        else:
            self.dim1.append(dim1)
            self.dim2.append(dim2)
            self.dim3.append(dim3)
            self.table[dim1] = {dim2:{dim3:1}}
Run Code Online (Sandbox Code Playgroud)

这样做的目的是为了让密钥求和等变得更容易。SparseCubeTable 的使用方式如下:

In [23]: example = SparseCubeTable()

In [24]: example.increment("thing1", "thing2", "thing3")

In [25]: example.increment("thing1", "thing2", "thing3")

In [26]: example.increment("thing4", "thing5", "thing6")

In [27]: example.increment("thing1", "thing3", "thing5")
Run Code Online (Sandbox Code Playgroud)

你可以这样获取数据:

In [29]: example.table['thing1']['thing2']['thing3']
Out[29]: 2
Run Code Online (Sandbox Code Playgroud)

我想要的 DataFrame 类型如下所示:

1 2 3 4
thing1 thing2 thing3 2
thing1 thing3 thing5 1
thing4 thing5 thing6 1
Run Code Online (Sandbox Code Playgroud)

DataFrame 将保存为 HDF5 数据库,其中第 1-3 列进行索引,第 4 列进行统计转换(这需要将整个表暂时存储在内存中)。

问题在于,pandas.DataFrame.from_dict据我所知,该函数使用用作行标签的键构建了一种完全其他类型的结构。然而,尝试使用from_records迫使我将整个数据结构复制到列表中,这意味着我现在需要担心双倍的内存大小。

我尝试在以下位置实施该解决方案:

从生成器创建 pandas DataFrame?

但在 0.12.0 中,它最终所做的是首先构建一个巨大的字符串列表,这甚至更糟糕。我认为将结构写到 csv 中并读回它也会对内存造成很大的影响。

有更好的方法吗?或者我应该尝试以某种方式进一步压缩内存SparseCubeTable?必须构建一个中间列表数据结构来使用似乎很浪费from_records。

Jef*_*eff 5

这是有效解决方案的代码。

创建一些与您的数据相似的数据。这是 1000 个 3 元组的列表

In [1]: import random

In [2]: tags = [ 'thing{0}'.format(i) for i in xrange(100) ]

In [3]: data = [ (random.choice(tags),random.choice(tags),random.choice(tags)) for i in range(1000) ]
Run Code Online (Sandbox Code Playgroud)

我们的写入函数确保我们写入时索引是全局唯一的(实际上不是必需的,但由于索引实际上是写入的,因此它“更好”)

In [4]: def write(store,c):
   ...:     df = DataFrame(c,columns=['dim1','dim2','dim3'])
   ...:     try:
   ...:         nrows = store.get_storer('df').nrows
   ...:     except:
   ...:         nrows = 0
   ...:     df.index += nrows
   ...:     store.append('df',df,data_columns=True)
   ...:     return []
   ...: 

In [5]: collector = []

In [6]: store = pd.HDFStore('data.h5',mode='w')
Run Code Online (Sandbox Code Playgroud)

迭代您的数据(或来自流或其他任何数据),然后写入。

In [7]: for i, d in enumerate(data):
   ...:     collector.append(d)
   ...:     if i % 100 == 0 and i:
   ...:         collector = write(store,collector)
   ...:         

In [8]: write(store,collector)
Out[8]: []
Run Code Online (Sandbox Code Playgroud)

商店

In [9]: store
Out[9]: 
<class 'pandas.io.pytables.HDFStore'>
File path: data.h5
/df            frame_table  (typ->appendable,nrows->1000,ncols->3,indexers->[index],dc->[dim1,dim2,dim3])
In [9]: store
Out[9]: 
<class 'pandas.io.pytables.HDFStore'>
File path: data.h5
/df            frame_table  (typ->appendable,nrows->1000,ncols->3,indexers->[index],dc->[dim1,dim2,dim3])

In [10]: store.select('df')
Out[10]: 
       dim1     dim2     dim3
0   thing28  thing87  thing29
1   thing62  thing70  thing50
2   thing64  thing12  thing98
3   thing33  thing98  thing46
4   thing46   thing5  thing76
5    thing2   thing9  thing21
6    thing1  thing63  thing68
7   thing42  thing30  thing45
8   thing56  thing71  thing77
9   thing99  thing10  thing91
10  thing40   thing9  thing10
11  thing70  thing54  thing59
12  thing94  thing65   thing3
13  thing93  thing24  thing25
14  thing95  thing94  thing86
15  thing41  thing55   thing3
16  thing88  thing10  thing47
17  thing89  thing58  thing33
18  thing16  thing66  thing55
19  thing68  thing20  thing99
20  thing34  thing71  thing28
21  thing67  thing87  thing97
22  thing77  thing74   thing6
23  thing63  thing41  thing30
24  thing14  thing62  thing66
25  thing20  thing36  thing67
26  thing33  thing19  thing58
27   thing0  thing71  thing24
28   thing1  thing48  thing42
29  thing18  thing12   thing4
30  thing85  thing97  thing20
31  thing73  thing71  thing70
32  thing91  thing43  thing48
33  thing45   thing6  thing87
34   thing0  thing28   thing8
35  thing56  thing38  thing61
36  thing39  thing92  thing35
37  thing69  thing26  thing22
38  thing16  thing16  thing79
39   thing4  thing16  thing12
40  thing81  thing79   thing1
41  thing77  thing90  thing83
42  thing53  thing17  thing89
43  thing53  thing15  thing37
44  thing25   thing7  thing20
45  thing44  thing14  thing25
46  thing62  thing84  thing23
47  thing83  thing50  thing60
48  thing68  thing64  thing24
49  thing73  thing53  thing43
50  thing86  thing67  thing31
51  thing75  thing63  thing82
52   thing8  thing10  thing90
53  thing34  thing23  thing12
54  thing66  thing97  thing26
55  thing66  thing53  thing27
56  thing79  thing22  thing37
57  thing43  thing82  thing66
58  thing87  thing53  thing92
59  thing33  thing71  thing97
        ...      ...      ...

[1000 rows x 3 columns]

In [11]: store.close()
Run Code Online (Sandbox Code Playgroud)

然后你就可以做一些有趣的事情了。如果您没有阅读整套内容,您可能需要将其分块(如果您正在计算内容,则涉及更多内容)。

In [56]: pd.read_hdf('data.h5','df').apply(lambda x: x.value_counts())
Out[56]: 
         dim1  dim2  dim3
thing0     12     6     8
thing1     14     7     8
thing10    10    10     7
thing11     8    10    14
thing12    11    14    11
thing13    11    12     7
thing14     8    14     3
thing15    12    11    11
thing16     7    10    11
thing17    16     9    13
thing18    13     8    10
thing19    11     7     8
thing2      9     5    17
thing20     6     7    11
thing21     7     8     8
thing22     4    17    14
thing23    14    11     7
thing24    10     5    14
thing25    11    11    12
thing26    13    10    15
thing27    12    15    16
thing28    11    10     8
thing29     7     7     8
thing3     11    14    14
thing30    11    16     8
thing31     7     6    12
thing32     8    12     9
thing33    13    12    12
thing34    12     8     5
thing35     6    10     8
thing36     6     9    13
thing37     8    10    12
thing38     7    10     4
thing39    14    11     7
thing4      9     7    10
thing40    12     8     9
thing41     8    16    11
thing42     9    11    13
thing43     8     6    13
thing44     9    13    11
thing45     7    13     7
thing46    12     8    13
thing47     9    10     9
thing48     8     9     9
thing49     4     8     7
thing5     13     7     7
thing50    14    12     9
thing51     5     7    11
thing52     9    11    12
thing53     9    15    15
thing54     7     9    13
thing55     6    10    10
thing56    12    11    11
thing57    12     9    11
thing58    12    12    10
thing59     6    13    10
thing6      8     5     7
thing60    12     9     6
thing61     5     9     9
thing62     8    10     8
          ...   ...   ...

[100 rows x 3 columns]
Run Code Online (Sandbox Code Playgroud)

然后你可以像这样进行“groupby”:

In [69]: store = pd.HDFStore('data.h5')

In [61]: dim1 = Index(store.select_column('df','dim1').unique())
In [66]: store.close()

In [67]: groups = dim1[0:10]

In [68]: groups
Out[68]: Index([u'thing28', u'thing62', u'thing64', u'thing33', u'thing46', u'thing2', u'thing1', u'thing42', u'thing56', u'thing99'], dtype='object')

In [70]: pd.read_hdf('data.h5','df',where='dim1=groups').apply(lambda x: x.value_counts())
Out[70]: 
         dim1  dim2  dim3
thing1     14     2     1
thing10   NaN     1     1
thing11   NaN     1     2
thing12   NaN     5   NaN
thing13   NaN     1   NaN
thing14   NaN     1     1
thing15   NaN     1     1
thing16   NaN     1     3
thing17   NaN   NaN     2
thing18   NaN     1     1
thing19   NaN     1     2
thing2      9     1     1
thing20   NaN     2   NaN
thing21   NaN   NaN     1
thing22   NaN     2     2
thing23   NaN     2     3
thing24   NaN     2     1
thing25   NaN     3     2
thing26   NaN     2     2
thing27   NaN     3     1
thing28    11   NaN   NaN
thing29   NaN     1     2
thing30   NaN     2   NaN
thing31   NaN     1     1
thing32   NaN     1     1
thing33    13     1     2
thing34   NaN     1   NaN
thing35   NaN     1   NaN
thing36   NaN     1     1
thing37   NaN     1     2
thing38   NaN     3   NaN
thing39   NaN     3     1
thing4    NaN     2   NaN
thing41   NaN   NaN     1
thing42     9     1     1
thing43   NaN   NaN     1
thing44   NaN     1     2
thing45   NaN   NaN     2
thing46    12   NaN     1
thing47   NaN     1     1
thing48   NaN     1   NaN
thing49   NaN     1   NaN
thing5    NaN     2     2
thing50   NaN   NaN     3
thing51   NaN     2     2
thing52   NaN     1     3
thing53   NaN     2     4
thing55   NaN   NaN     2
thing56    12     1     1
thing57   NaN   NaN     3
thing58   NaN     1     2
thing6    NaN   NaN     1
thing60   NaN     1     1
thing61   NaN     1     4
thing62     8     2     1
thing63   NaN     1     1
thing64    15   NaN     1
thing66   NaN     1     2
thing67   NaN     2   NaN
thing68   NaN     1     1
          ...   ...   ...

[90 rows x 3 columns]
Run Code Online (Sandbox Code Playgroud)