比较两个数据帧并获得差异

Question

我有两个数据帧.例子:

df1:
Date       Fruit  Num  Color 
2013-11-24 Banana 22.1 Yellow
2013-11-24 Orange  8.6 Orange
2013-11-24 Apple   7.6 Green
2013-11-24 Celery 10.2 Green

df2:
Date       Fruit  Num  Color 
2013-11-24 Banana 22.1 Yellow
2013-11-24 Orange  8.6 Orange
2013-11-24 Apple   7.6 Green
2013-11-24 Celery 10.2 Green
2013-11-25 Apple  22.1 Red
2013-11-25 Orange  8.6 Orange

每个数据帧都以Date作为索引.两个数据帧都具有相同的结构.

我想要做的是比较这两个数据帧,并找出df2中哪些行不在df1中.我想比较日期(索引)和第一列(Banana,APple等),看看它们是否存在于df2和df1中.

我尝试过以下方法:

• 并排输出两个Pandas数据帧的差异 - 突出显示差异
• 比较两个pandas数据帧的差异

对于第一种方法,我得到了这个错误:"异常:只能比较标记相同的DataFrame对象".我已经尝试删除日期作为索引,但得到相同的错误.

在第三种方法中,我得到断言返回False但无法弄清楚如何实际看到不同的行.

任何指针都会受到欢迎

Answer 1

此方法df1 != df2仅适用于具有相同行和列的数据帧.实际上,所有数据帧轴都与_indexed_same方法进行比较,如果发现差异,则会引发异常,即使在列/索引顺序中也是如此.

如果我找到了你,你想要找不到变化,而是对称差异.为此,一种方法可能是连接数据帧:

>>> df = pd.concat([df1, df2])
>>> df = df.reset_index(drop=True)

通过...分组

>>> df_gpby = df.groupby(list(df.columns))

获取唯一记录的索引

>>> idx = [x[0] for x in df_gpby.groups.values() if len(x) == 1]

过滤

>>> df.reindex(idx)
         Date   Fruit   Num   Color
9  2013-11-25  Orange   8.6  Orange
8  2013-11-25   Apple  22.1     Red

Answer 2

更新和放置，在其他人更容易找到的地方，ling对jur上面的回复的评论。

df_diff = pd.concat([df1,df2]).drop_duplicates(keep=False)

使用这些 DataFrame 进行测试：

# with import pandas as pd

df1 = pd.DataFrame({
    'Date':['2013-11-24','2013-11-24','2013-11-24','2013-11-24'],
    'Fruit':['Banana','Orange','Apple','Celery'],
    'Num':[22.1,8.6,7.6,10.2],
    'Color':['Yellow','Orange','Green','Green'],
    })

df2 = pd.DataFrame({
    'Date':['2013-11-24','2013-11-24','2013-11-24','2013-11-24','2013-11-25','2013-11-25'],
    'Fruit':['Banana','Orange','Apple','Celery','Apple','Orange'],
    'Num':[22.1,8.6,7.6,10.2,22.1,8.6],
    'Color':['Yellow','Orange','Green','Green','Red','Orange'],
    })

结果如下：

# for df1

         Date   Fruit   Num   Color
0  2013-11-24  Banana  22.1  Yellow
1  2013-11-24  Orange   8.6  Orange
2  2013-11-24   Apple   7.6   Green
3  2013-11-24  Celery  10.2   Green


# for df2

         Date   Fruit   Num   Color
0  2013-11-24  Banana  22.1  Yellow
1  2013-11-24  Orange   8.6  Orange
2  2013-11-24   Apple   7.6   Green
3  2013-11-24  Celery  10.2   Green
4  2013-11-25   Apple  22.1     Red
5  2013-11-25  Orange   8.6  Orange


# for df_diff

         Date   Fruit   Num   Color
4  2013-11-25   Apple  22.1     Red
5  2013-11-25  Orange   8.6  Orange

Answer 3

将数据帧传递到字典中的concat会产生一个多索引数据框,您可以从中轻松删除重复项,从而产生具有数据帧之间差异的多索引数据框:

import sys
if sys.version_info[0] < 3:
    from StringIO import StringIO
else:
    from io import StringIO
import pandas as pd

DF1 = StringIO("""Date       Fruit  Num  Color 
2013-11-24 Banana 22.1 Yellow
2013-11-24 Orange  8.6 Orange
2013-11-24 Apple   7.6 Green
2013-11-24 Celery 10.2 Green
""")
DF2 = StringIO("""Date       Fruit  Num  Color 
2013-11-24 Banana 22.1 Yellow
2013-11-24 Orange  8.6 Orange
2013-11-24 Apple   7.6 Green
2013-11-24 Celery 10.2 Green
2013-11-25 Apple  22.1 Red
2013-11-25 Orange  8.6 Orange""")


df1 = pd.read_table(DF1, sep='\s+')
df2 = pd.read_table(DF2, sep='\s+')
#%%
dfs_dictionary = {'DF1':df1,'DF2':df2}
df=pd.concat(dfs_dictionary)
df.drop_duplicates(keep=False)

结果:

             Date   Fruit   Num   Color
DF2 4  2013-11-25   Apple  22.1     Red
    5  2013-11-25  Orange   8.6  Orange

Answer 4

# THIS WORK FOR ME

# Get all diferent values
df3 = pd.merge(df1, df2, how='outer', indicator='Exist')
df3 = df3.loc[df3['Exist'] != 'both']


# If you like to filter by a common ID
df3  = pd.merge(df1, df2, on="Fruit", how='outer', indicator='Exist')
df3  = df3.loc[df3['Exist'] != 'both']

Answer 5

由于pandas >= 1.1.0我们有DataFrame.compare和Series.compare。

注意：该方法只能比较标记相同的 DataFrame 对象，这意味着具有相同行和列标签的 DataFrame。

df1 = pd.DataFrame({'A': [1, 2, 3],
                    'B': [4, 5, 6],
                    'C': [7, np.NaN, 9]})

df2 = pd.DataFrame({'A': [1, 99, 3],
                    'B': [4, 5, 81],
                    'C': [7, 8, 9]})

   A  B    C
0  1  4  7.0
1  2  5  NaN
2  3  6  9.0 

    A   B  C
0   1   4  7
1  99   5  8
2   3  81  9

df1.compare(df2)

     A          B          C      
  self other self other self other
1  2.0  99.0  NaN   NaN  NaN   8.0
2  NaN   NaN  6.0  81.0  NaN   NaN

Answer 6

df2从into获取现有数据df1：

dfe = df2[df2["Fruit"].isin(df1["Fruit"])]

df2从into获取不存在的数据df1：

dfn = df2[~ df2["Fruit"].isin(df1["Fruit"])]

您可以使用多个比较。

Answer 7

基于 alko 几乎对我有用的答案，除了过滤步骤（我得到的地方：）ValueError: cannot reindex from a duplicate axis，这是我使用的最终解决方案：

# join the dataframes
united_data = pd.concat([data1, data2, data3, ...])
# group the data by the whole row to find duplicates
united_data_grouped = united_data.groupby(list(united_data.columns))
# detect the row indices of unique rows
uniq_data_idx = [x[0] for x in united_data_grouped.indices.values() if len(x) == 1]
# extract those unique values
uniq_data = united_data.iloc[uniq_data_idx]