pandas:合并条件的时间范围

Question

我想将一个数据框与另一个数据框合并,其中合并的条件是日期/时间落在特定范围内.

例如,假设我有以下两个数据框.

import pandas as pd
import datetime

# Create main data frame.
data = pd.DataFrame()
time_seq1 = pd.DataFrame(pd.date_range('1/1/2016', periods=3, freq='H'))
time_seq2 = pd.DataFrame(pd.date_range('1/2/2016', periods=3, freq='H'))
data = data.append(time_seq1, ignore_index=True)
data = data.append(time_seq1, ignore_index=True)
data = data.append(time_seq1, ignore_index=True)
data = data.append(time_seq2, ignore_index=True)
data['myID'] = ['001','001','001','002','002','002','003','003','003','004','004','004']
data.columns = ['Timestamp', 'myID']

# Create second data frame.
data2 = pd.DataFrame()
data2['time'] = [pd.to_datetime('1/1/2016 12:06 AM'), pd.to_datetime('1/1/2016 1:34 AM'), pd.to_datetime('1/2/2016 12:25 AM')]
data2['myID'] = ['002', '003', '004']
data2['specialID'] = ['foo_0', 'foo_1', 'foo_2']

# Show data frames.
data
             Timestamp myID
0  2016-01-01 00:00:00  001
1  2016-01-01 01:00:00  001
2  2016-01-01 02:00:00  001
3  2016-01-01 00:00:00  002
4  2016-01-01 01:00:00  002
5  2016-01-01 02:00:00  002
6  2016-01-01 00:00:00  003
7  2016-01-01 01:00:00  003
8  2016-01-01 02:00:00  003
9  2016-01-02 00:00:00  004
10 2016-01-02 01:00:00  004
11 2016-01-02 02:00:00  004

data2
                 time myID specialID
0 2016-01-01 00:06:00  002     foo_0
1 2016-01-01 01:34:00  003     foo_1
2 2016-01-02 00:25:00  004     foo_2

我想构造以下输出.

# Desired output.
             Timestamp myID special_ID
0  2016-01-01 00:00:00  001        NaN
1  2016-01-01 01:00:00  001        NaN
2  2016-01-01 02:00:00  001        NaN
3  2016-01-01 00:00:00  002        NaN
4  2016-01-01 01:00:00  002      foo_0
5  2016-01-01 02:00:00  002        NaN
6  2016-01-01 00:00:00  003        NaN
7  2016-01-01 01:00:00  003        NaN
8  2016-01-01 02:00:00  003      foo_1
9  2016-01-02 00:00:00  004        NaN
10 2016-01-02 01:00:00  004      foo_2
11 2016-01-02 02:00:00  004        NaN

特别是,我想合并special_ID到data这样Timestamp的第一次发生在值之后time.例如,foo_0将在对应于行2016-01-01 01:00:00与myID = 002自认为是在未来的时间data后立即2016-01-01 00:06:00(在time的special_ID = foo_0)含有的行中myID = 002.

注意,Timestamp不是索引data而time不是索引data2.大多数其他相关帖子似乎依赖于使用datetime对象作为数据框的索引.

Answer 1

您可以使用merge_asofPandas 0.19中的新功能来完成大部分工作.然后,组合loc并duplicated删除辅助匹配:

# Data needs to be sorted for merge_asof.
data = data.sort_values(by='Timestamp')

# Perform the merge_asof.
df = pd.merge_asof(data, data2, left_on='Timestamp', right_on='time', by='myID').drop('time', axis=1)

# Make the additional matches null.
df.loc[df['specialID'].duplicated(), 'specialID'] = np.nan

# Get the original ordering.
df = df.set_index(data.index).sort_index()

结果输出:

             Timestamp myID specialID
0  2016-01-01 00:00:00  001       NaN
1  2016-01-01 01:00:00  001       NaN
2  2016-01-01 02:00:00  001       NaN
3  2016-01-01 00:00:00  002       NaN
4  2016-01-01 01:00:00  002     foo_0
5  2016-01-01 02:00:00  002       NaN
6  2016-01-01 00:00:00  003       NaN
7  2016-01-01 01:00:00  003       NaN
8  2016-01-01 02:00:00  003     foo_1
9  2016-01-02 00:00:00  004       NaN
10 2016-01-02 01:00:00  004     foo_2
11 2016-01-02 02:00:00  004       NaN