有效地逐行比较两列中的列表

Question

当有这样的 Pandas DataFrame 时：

import pandas as pd
import numpy as np
df = pd.DataFrame({'today': [['a', 'b', 'c'], ['a', 'b'], ['b']], 
                   'yesterday': [['a', 'b'], ['a'], ['a']]})

                 today        yesterday
0      ['a', 'b', 'c']       ['a', 'b']
1           ['a', 'b']            ['a']
2                ['b']            ['a']                          
... etc

但是有大约 100 000 个条目，我希望在两列中逐行找到这些列表的添加和删除。

它类似于这个问题：Pandas: How to Compare Columns of Lists of List Row-wise in a DataFrame with Pandas (not for loop)? 但我正在研究差异，Pandas.apply对于这么多条目，方法似乎并没有那么快。这是我目前使用的代码。Pandas.apply用numpy's setdiff1d方法：

additions = df.apply(lambda row: np.setdiff1d(row.today, row.yesterday), axis=1)
removals  = df.apply(lambda row: np.setdiff1d(row.yesterday, row.today), axis=1)

这工作正常，但是 120 000 个条目需要大约一分钟。那么有没有更快的方法来实现这一点？

Answer 1

不确定性能，但由于缺乏更好的解决方案，这可能适用：

temp = df[['today', 'yesterday']].applymap(set)
removals = temp.diff(periods=1, axis=1).dropna(axis=1)
additions = temp.diff(periods=-1, axis=1).dropna(axis=1) 

移除：

  yesterday
0        {}
1        {}
2       {a}

补充：

  today
0   {c}
1   {b}
2   {b}

Answer 2

df['today'].apply(set) - df['yesterday'].apply(set)

Answer 3

我会建议你计算additions并removals在同一个应用中。

生成一个更大的例子

import pandas as pd
import numpy as np
df = pd.DataFrame({'today': [['a', 'b', 'c'], ['a', 'b'], ['b']], 
                   'yesterday': [['a', 'b'], ['a'], ['a']]})
df = pd.concat([df for i in range(10_000)], ignore_index=True)

您的解决方案

%%time
additions = df.apply(lambda row: np.setdiff1d(row.today, row.yesterday), axis=1)
removals  = df.apply(lambda row: np.setdiff1d(row.yesterday, row.today), axis=1)
CPU times: user 10.9 s, sys: 29.8 ms, total: 11 s
Wall time: 11 s

您一次申请的解决方案

%%time
df["out"] = df.apply(lambda row: [np.setdiff1d(row.today, row.yesterday),
                                  np.setdiff1d(row.yesterday, row.today)], axis=1)
df[['additions','removals']] = pd.DataFrame(df['out'].values.tolist(), 
                                            columns=['additions','removals'])
df = df.drop("out", axis=1)

CPU times: user 4.97 s, sys: 16 ms, total: 4.99 s
Wall time: 4.99 s

使用 set

除非你的清单很大，否则你可以避免 numpy

def fun(x):
    a = list(set(x["today"]).difference(set(x["yesterday"])))
    b = list((set(x["yesterday"])).difference(set(x["today"])))
    return [a,b]

%%time
df["out"] = df.apply(fun, axis=1)
df[['additions','removals']] = pd.DataFrame(df['out'].values.tolist(), 
                                            columns=['additions','removals'])
df = df.drop("out", axis=1)

CPU times: user 1.56 s, sys: 0 ns, total: 1.56 s
Wall time: 1.56 s

@r.ook 的解决方案

如果您很高兴将集合而不是列表作为输出，您可以使用@r.ook 的代码

%%time
temp = df[['today', 'yesterday']].applymap(set)
removals = temp.diff(periods=1, axis=1).dropna(axis=1)
additions = temp.diff(periods=-1, axis=1).dropna(axis=1) 
CPU times: user 93.1 ms, sys: 12 ms, total: 105 ms
Wall time: 104 ms

@Andreas K. 的解决方案

%%time
df['additions'] = (df['today'].apply(set) - df['yesterday'].apply(set))
df['removals'] = (df['yesterday'].apply(set) - df['today'].apply(set))

CPU times: user 161 ms, sys: 28.1 ms, total: 189 ms
Wall time: 187 ms

你最终可以添加.apply(list)以获得相同的输出