使用pandas,如何以有效的方式按组对大型DataFrame进行二次采样？

Question

我正在尝试根据分组对DataFrame的行进行子采样.这是一个例子.假设我定义了以下数据:

from pandas import *
df = DataFrame({'group1' : ["a","b","a","a","b","c","c","c","c",
                            "c","a","a","a","b","b","b","b"],
                'group2' : [1,2,3,4,1,3,5,6,5,4,1,2,3,4,3,2,1],
                'value'  : ["apple","pear","orange","apple",
                            "banana","durian","lemon","lime",
                            "raspberry","durian","peach","nectarine",
                            "banana","lemon","guava","blackberry","grape"]})

如果我按group1和分组group2,则每组中的行数如下:

In [190]: df.groupby(['group1','group2'])['value'].agg({'count':len})
Out[190]: 
      count
a  1  2    
   2  1    
   3  2    
   4  1    
b  1  2    
   2  2    
   3  1    
   4  1    
c  3  1    
   4  1    
   5  2    
   6  1    

(如果有更简洁的方法来计算它,请告诉.)

我现在想构建一个DataFrame,每个组中有一个随机选择的行.我的建议就是这样做:

In [215]: from random import choice
In [216]: grouped = df.groupby(['group1','group2'])
In [217]: subsampled = grouped.apply(lambda x: df.reindex(index=[choice(range(len(x)))]))
In [218]: subsampled.index = range(len(subsampled))
In [219]: subsampled
Out[219]: 
    group1  group2  value
0   b       2       pear 
1   a       1       apple
2   b       2       pear 
3   a       1       apple
4   a       1       apple
5   a       1       apple
6   a       1       apple
7   a       1       apple
8   a       1       apple
9   a       1       apple
10  a       1       apple
11  a       1       apple

哪个有效.但是,我的真实数据有大约250万行和12列.如果我通过构建自己的数据结构来做到这一点,我可以在几秒钟内完成此操作.但是,我上面的实现没有在30分钟内完成(并且似乎没有内存限制).作为旁注,当我尝试在R中实现这一点时,我首先尝试了plyr,但也没有在合理的时间内完成; 然而,使用data.table完成的解决方案非常迅速.

如何快速使用它pandas？我想要喜欢这个包,所以请帮忙!

Answer 1

我用apply测试过,似乎当有很多子组时,它很慢.groups的groups属性是一个dict,你可以直接从它中选择索引:

subsampled = df.ix[(choice(x) for x in grouped.groups.itervalues())]

编辑:从pandas版本0.18.1开始,itervalues不再适用于groupby对象 - 您可以使用.values:

subsampled = df.ix[(choice(x) for x in grouped.groups.values())]