快速替代在Pandas DataFrame中的所有行上运行基于numpy的函数

Question

我有一个以下列方式创建的Pandas数据框:

import pandas as pd
def create(n):  
    df = pd.DataFrame({ 'gene':["foo",
                            "bar",
                            "qux",
                            "woz"], 
                            'cell1':[433.96,735.62,483.42,10.33], 
                            'cell2':[94.93,2214.38,97.93,1205.30],
                            'cell3':[1500,90,100,80]})
    df = df[["gene","cell1","cell2","cell3"]]
    df = pd.concat([df]*n)
    df = df.reset_index(drop=True)
    return df

它看起来像这样:

In [108]: create(1)
Out[108]:
  gene   cell1    cell2  cell3
0  foo  433.96    94.93   1500
1  bar  735.62  2214.38     90
2  qux  483.42    97.93    100
3  woz   10.33  1205.30     80

然后我有一个函数,它取每个基因(行)的值来计算某个分数:

import numpy as np 
def sparseness(xvec):
    n = len(xvec)
    xvec_sum = np.sum(np.abs(xvec))
    xvecsq_sum = np.sum(np.square(xvec))

    denom = np.sqrt(n) - (xvec_sum / np.sqrt(xvecsq_sum))
    enum  = np.sqrt(n) - 1
    sparseness_x = denom/enum

    return sparseness_x

实际上我需要在40K行上应用此功能.目前使用Pandas'apply'运行速度非常慢:

In [109]: df = create(10000)
In [110]: express_df   = df.ix[:,1:]
In [111]: %timeit express_df.apply(sparseness, axis=1)
1 loops, best of 3: 8.32 s per loop

实现它的更快的替代方案是什么？

Answer 1

更快的方法是实现函数的矢量化版本,该函数直接在二维ndarray上运行.这是非常可行的,因为numpy中的许多函数可以在二维ndarray上操作,使用axis参数控制.可能的实施:

def sparseness2(xs):
    nr = np.sqrt(xs.shape[1])
    a = np.sum(np.abs(xs), axis=1)
    b = np.sqrt(np.sum(np.square(xs), axis=1))
    sparseness = (nr - a/b) / (nr - 1)
    return sparseness

res_arr = sparseness2(express_df.values)
res2 = pd.Series(res_arr, index=express_df.index)

一些测试:

from pandas.util.testing import assert_series_equal
res1 = express_df.apply(sparseness, axis=1)
assert_series_equal(res1, res2) #OK
%timeit sparseness2(express_df.values)
# 1000 loops, best of 3: 655 µs per loop

Answer 2

这是一种矢量化方法,np.einsum用于在整个数据帧中一次性执行所有这些操作.现在,这np.einsum对于这种乘法和求和目的来说非常有效.在我们的例子中,我们可以使用它来对xvec_sum案例进行一维求和,并对xvecsq_sum案例进行平方和求和.这个项目看起来像这样 -

def sparseness_vectorized(A):
    nsqrt = np.sqrt(A.shape[1])
    B = np.einsum('ij->i',np.abs(A))/np.sqrt(np.einsum('ij,ij->i',A,A))    
    denom = nsqrt - B
    enum  = nsqrt - 1
    return denom/enum

运行时测试 -

本节比较了迄今为止列出的所有方法,以解决问题,包括问题中的问题.

In [235]: df = create(1000)
     ...: express_df  = df.ix[:,1:]
     ...: 

In [236]: %timeit express_df.apply(sparseness, axis=1)
1 loops, best of 3: 1.36 s per loop

In [237]: %timeit sparseness2(express_df.values)
1000 loops, best of 3: 247 µs per loop

In [238]: %timeit sparseness_vectorized(express_df.values)
1000 loops, best of 3: 231 µs per loop



In [239]: df = create(5000)
     ...: express_df  = df.ix[:,1:]
     ...: 

In [240]: %timeit express_df.apply(sparseness, axis=1)
1 loops, best of 3: 6.66 s per loop

In [241]: %timeit sparseness2(express_df.values)
1000 loops, best of 3: 1.14 ms per loop

In [242]: %timeit sparseness_vectorized(express_df.values)
1000 loops, best of 3: 1.06 ms per loop