检查dataframe列中的重复值

Question

有没有办法检查数据帧列是否具有重复值而不丢弃行？我有一个将删除重复行的函数,但是,如果特定列中实际存在重复项,我只希望它运行.

目前,我将列中唯一值的数量与行数进行比较:如果唯一值少于行,则存在重复值并运行代码.

 if len(df['Student'].unique()) < len(df.index):
    # Code to remove duplicates based on Date column runs

是否有更简单或更有效的方法使用pandas检查特定列中是否存在重复值？

我正在使用的一些示例数据(仅显示了两列).如果找到重复项,则另一个函数标识要保留的行(具有最早日期的行):

    Student Date
0   Joe     December 2017
1   James   January 2018
2   Bob     April 2018
3   Joe     December 2017
4   Jack    February 2018
5   Jack    March 2018

Answer 1

这是一个完整的例子:

???????????????????????????
? Student ? Date          ?
???????????????????????????
? Joe     ? December 2017 ?
???????????????????????????
? Bob     ? April 2018    ?
???????????????????????????
? Joe     ? December 2018 ?
???????????????????????????

返回

boolean = any(df['Student'].duplicated()) # True

我经常在熊猫中说我们有兴趣创建布尔掩码,即我们可以用来选择帧子集的True或False数组.在这种情况下,我们可以例如想要创建:

boolean = any(df.duplicated(subset=['Student'])) # True
# We were expecting True, as Joe can be seen twice.

~m等于np.invert(m),它将True翻转为False,反之亦然.在这种情况下获得非重复的非常有用.如果我们现在将其作为选择器传递给数据帧:

boolean = any(df.duplicated()) # False
boolean = any(df.duplicated(subset=['Student','Date'])) # False
# We were expecting False here - no duplicates row-wise 
# ie. Joe Dec 2017, Joe Dec 2018

我们又来了:

import pandas as pd
import io

data = '''\
Student,Date
Joe,December 2017
Bob,April 2018
Joe,December 2018'''

df = pd.read_csv(io.StringIO(data), sep=',')

# Approach 1: Simple True/False
boolean = any(df.duplicated(subset=['Student']))
print(boolean, end='\n\n') # True

# Approach 2: First store boolean array, check then remove
duplicate_in_student = df.duplicated(subset=['Student'])
if any(duplicate_in_student):
    print(df.loc[~duplicate_in_student], end='\n\n')

# Approach 3: Use drop_duplicates method
df.drop_duplicates(subset=['Student'], inplace=True)
print(df)

之前我打扰将Date列更改为datetime.在进行任何类型的日期排序或任何日期操作之前,这是很好的实践.

Answer 2

如果您想知道有多少重复项以及它们的用途：

df.pivot_table(index=['ColumnName'], aggfunc='size')

df.pivot_table(index=['ColumnName1',.., 'ColumnNameN'], aggfunc='size')

Answer 3

您可以使用is_unique：

df['Student'].is_unique

# equals true in case of no duplicates

需要较旧的熊猫版本：

pd.Series(df['Student']).is_unique