在 pandas 中每个组的开头插入多个日期

Question

我有一个包含数百万组的数据框。我尝试在每个组的顶部添加 3 个月的日期（月结束日期）。因此，如果某个组的第一次观察是 2019 年 12 月，我想在该观察之前填充 3 行，日期为 2019 年 9 月到 2019 年 11 月。我还想用相关组 ID 填充组列，其他列可以保留作为空值。

希望尽可能避免循环，因为这是一个非常大的数据集

这是我之前的数据框：

import pandas as pd

before = pd.DataFrame({'Group':[1,1,1,1,1,2,2,2,2,2],
    'Date':['31/10/2018','30/11/2018','31/12/2018','31/01/2019','28/02/2019','30/03/2001','30/04/2001','31/05/2001','30/06/2001','31/07/2001'],
    'value':[1.1,1.7,1.9,2.3,1.5,2.8,2,2,2,2]})

这是我的数据框之后

import pandas as pd

after = pd.DataFrame({'Group':[1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2],
'Date':['31/07/2018','31/08/2018','30/09/2018','31/10/2018','30/11/2018','31/12/2018','31/01/2019','28/02/2019','31/12/2000','31/01/2001','28/02/2001','30/03/2001','30/04/2001','31/05/2001','30/06/2001','31/07/2001'],
'value':[np.nan,np.nan,np.nan,1.1,1.7,1.9,2.3,1.5,np.nan,np.nan,np.nan,2.8,2,2,2,2]})

Answer 1

因为如果多个组解决方案不能非常快，则单独处理每个组 - 想法是获取Groupby的第一行DataFrame.drop_duplicates，按 移动月份offsets.MonthOffset，连接在一起并在之间添加所有缺失的日期：

before['Date'] = pd.to_datetime(before['Date'], dayfirst=True)

df1 = before.drop_duplicates('Group')

#first and last shifted months - by 1 and by 3 months
df11 = df1[['Group','Date']].assign(Date = lambda x: x['Date'] - pd.offsets.MonthOffset(3))
df12 = df1[['Group','Date']].assign(Date = lambda x: x['Date'] - pd.offsets.MonthOffset(1))

df = (pd.concat([df11, df12], sort=False, ignore_index=True)
       .set_index('Date')
       .groupby('Group')
       .resample('m')
       .size()
       .reset_index(name='value')
       .assign(value = np.nan))
print (df)
   Group       Date  value
0      1 2018-07-31    NaN
1      1 2018-08-31    NaN
2      1 2018-09-30    NaN
3      2 2000-12-31    NaN
4      2 2001-01-31    NaN
5      2 2001-02-28    NaN

最后添加到原始并排序：

df = pd.concat([before, df], ignore_index=True).sort_values(['Group','Date'])

print (df)
    Group       Date  value
10      1 2018-07-31    NaN
11      1 2018-08-31    NaN
12      1 2018-09-30    NaN
0       1 2018-10-31    1.1
1       1 2018-11-30    1.7
2       1 2018-12-31    1.9
3       1 2019-01-31    2.3
4       1 2019-02-28    1.5
13      2 2000-12-31    NaN
14      2 2001-01-31    NaN
15      2 2001-02-28    NaN
5       2 2001-03-30    2.8
6       2 2001-04-30    2.0
7       2 2001-05-31    2.0
8       2 2001-06-30    2.0
9       2 2001-07-31    2.0

如果新月份只有几个，您可以省略groupby部分：

before['Date'] = pd.to_datetime(before['Date'], dayfirst=True)

df1 = before.drop_duplicates('Group')

df11 = df1[['Group','Date']].assign(Date = lambda x: x['Date'] - pd.offsets.MonthOffset(3))
df12 = df1[['Group','Date']].assign(Date = lambda x: x['Date'] - pd.offsets.MonthOffset(2))
df13 = df1[['Group','Date']].assign(Date = lambda x: x['Date'] - pd.offsets.MonthOffset(1))

df = (pd.concat([df11, df12, df13, before], ignore_index=True, sort=False)
        .sort_values(['Group','Date']))

print (df)
    Group       Date  value
0       1 2018-07-31    NaN
2       1 2018-08-31    NaN
4       1 2018-09-30    NaN
6       1 2018-10-31    1.1
7       1 2018-11-30    1.7
8       1 2018-12-31    1.9
9       1 2019-01-31    2.3
10      1 2019-02-28    1.5
1       2 2000-12-30    NaN
3       2 2001-01-30    NaN
5       2 2001-02-28    NaN
11      2 2001-03-30    2.8
12      2 2001-04-30    2.0
13      2 2001-05-31    2.0
14      2 2001-06-30    2.0
15      2 2001-07-31    2.0