在Pandas中将Date列转换为工作日名称的更快方法

Question

这是我通过pd.read_csv()读取的输入csv文件

ProductCode,Date,Receipt,Total
x1,07/29/15,101790,17.35
x2,07/29/15,103601,8.89
x3,07/29/15,103601,8.58
x4,07/30/15,101425,11.95
x5,07/29/15,101422,1.09
x6,07/29/15,101422,0.99
x7,07/29/15,101422,3
y7,08/05/15,100358,7.29
x8,08/05/15,100358,2.6
z3,08/05/15,100358,2.99


import pandas as pd
df = pd.read_csv('product.csv')

#I have to add some columns to the data:

df['Receipt_Count'] = df.groupby(['Date','Receipt'])['Receipt'].transform('count')
df['Day_of_Week'] = pd.to_datetime(df['Date']).dt.weekday_name

我的csv文件中有大约800K的行.当我运行将日期转换为weekday_name的代码行时,我需要大约2分钟.我知道我首先将我的'Date'列转换为datetime,因为它被视为来自csv的字符串,然后它被转换为它的工作日等价物.有什么办法可以缩短转换时间吗？

我对Pandas/Python很新,所以我不确定我是否错过了这里的内容.

Answer 1

指定日期字符串的格式将大大加快转换速度:

df['Day_of_Week'] = pd.to_datetime(df['Date'], format='%m/%d/%y').dt.weekday_name

以下是一些基准测试:

import io
import pandas as pd

data = io.StringIO('''\
ProductCode,Date,Receipt,Total
x1,07/29/15,101790,17.35
x2,07/29/15,103601,8.89
x3,07/29/15,103601,8.58
x4,07/30/15,101425,11.95
x5,07/29/15,101422,1.09
x6,07/29/15,101422,0.99
x7,07/29/15,101422,3
y7,08/05/15,100358,7.29
x8,08/05/15,100358,2.6
z3,08/05/15,100358,2.99
''')

df = pd.read_csv(data)
%timeit pd.to_datetime(df['Date']).dt.weekday_name
# => 100 loops, best of 3: 2.48 ms per loop
%timeit pd.to_datetime(df['Date'], format='%m/%d/%y').dt.weekday_name
# => 1000 loops, best of 3: 507 µs per loop

large_df = pd.concat([df] * 1000)
%timeit pd.to_datetime(large_df['Date']).dt.weekday_name
# => 1 loop, best of 3: 1.62 s per loop
%timeit pd.to_datetime(large_df['Date'], format='%m/%d/%y').dt.weekday_name
# => 10 loops, best of 3: 45.9 ms per loop

即使您在OP中提供的小样本,性能也提高了5倍 - 对于更大的数据帧,它会变得更好,更好.