熊猫:在每组中按平均值填充缺失值比转换更快

Question

我需要通过每组中的平均值填充pandas DataFrame中的缺失值.根据这个问题 transform可以实现这一点.

但是,transform对我来说太慢了.

例如,对具有100个不同组和70%NaN值的大型DataFrame进行以下设置:

import pandas as pd
import numpy as np

size = 10000000  # DataFrame length
ngroups = 100  # Number of Groups

randgroups = np.random.randint(ngroups, size=size)  # Creation of groups
randvals = np.random.rand(size) * randgroups * 2    # Random values with mean like group number
nan_indices = np.random.permutation(range(size))    # NaN indices
nanfrac = 0.7                                       # Fraction of NaN values
nan_indices = nan_indices[:int(nanfrac*size)]       # Take fraction of NaN indices
randvals[nan_indices] = np.NaN                      # Set NaN values

df = pd.DataFrame({'value': randvals, 'group': randgroups})  # Create data frame

使用transformvia

df.groupby("group").transform(lambda x: x.fillna(x.mean())) # Takes too long

我的电脑上已经超过3秒钟了.我需要更快一些数量级的东西(购买更大的机器不是一种选择:-D).

那么我怎样才能更快地填补缺失值？

Answer 1

这是一种 NumPy 方法，np.bincount对于这种基于 bin 的求和/平均操作非常有效 -

ids = df.group.values                    # Extract 2 columns as two arrays
vals = df.value.values

m = np.isnan(vals)                             # Mask of NaNs
grp_sums = np.bincount(ids,np.where(m,0,vals)) # Group sums with NaNs as 0s
avg_vals = grp_sums*(1.0/np.bincount(ids,~m))        # Group averages
vals[m] = avg_vals[ids[m]]              # Set avg values into NaN positions

请注意，这将更新该value列。

运行时测试

数据大小：

size = 1000000  # DataFrame length
ngroups = 10  # Number of Groups

时间：

In [17]: %timeit df.groupby("group").transform(lambda x: x.fillna(x.mean()))
1 loops, best of 3: 276 ms per loop

In [18]: %timeit bincount_based(df)
100 loops, best of 3: 13.6 ms per loop

In [19]: 276.0/13.6  # Speedup
Out[19]: 20.294117647058822

20x+那里加速！