Python:pandas合并多个数据帧

Question

我有不同的数据帧,需要根据日期列将它们合并在一起.如果我只有两个数据帧,我可以使用df1.merge(df2, on='date')它来做三个数据帧df1.merge(df2.merge(df3, on='date'), on='date'),但是,使用多个数据帧,它变得非常复杂和难以理解.

所有数据帧都有一个共同的列 - date但它们没有相同数量的行或列,我只需要每个日期对每个数据帧都是通用的那些行.

所以,我正在尝试编写一个递归函数,它返回一个包含所有数据的数据帧,但它不起作用.那么我应该如何合并多个数据帧呢？

我试图diferent的方式,得到了类似的错误out of range,keyerror 0/1/2/3和can not merge DataFrame with instance of type <class 'NoneType'>.

这是我写的脚本:

dfs = [df1, df2, df3] # list of dataframes

def mergefiles(dfs, countfiles, i=0):
    if i == (countfiles - 2): # it gets to the second to last and merges it with the last
        return

    dfm = dfs[i].merge(mergefiles(dfs[i+1], countfiles, i=i+1), on='date')
    return dfm

print(mergefiles(dfs, len(dfs)))

一个例子:df_1:

May 19, 2017;1,200.00;0.1%
May 18, 2017;1,100.00;0.1%
May 17, 2017;1,000.00;0.1%
May 15, 2017;1,901.00;0.1%

df_2:

May 20, 2017;2,200.00;1000000;0.2%
May 18, 2017;2,100.00;1590000;0.2%
May 16, 2017;2,000.00;1230000;0.2%
May 15, 2017;2,902.00;1000000;0.2%

df_3:

May 21, 2017;3,200.00;2000000;0.3%
May 17, 2017;3,100.00;2590000;0.3%
May 16, 2017;3,000.00;2230000;0.3%
May 15, 2017;3,903.00;2000000;0.3%

预期的合并结果:

May 15, 2017;  1,901.00;0.1%;  2,902.00;1000000;0.2%;   3,903.00;2000000;0.3%   

Answer 1

如果不涉及复杂查询,下面是合并多个数据帧的最简洁,最易理解的方法.

只需简单地将DATE与DATE合并为索引,并使用OUTER方法进行合并(以获取所有数据).

import pandas as pd
from functools import reduce

df1 = pd.read_table('file1.csv', sep=',')
df2 = pd.read_table('file2.csv', sep=',')
df3 = pd.read_table('file3.csv', sep=',')

因此,基本上将您拥有的所有文件作为数据框加载.然后使用merge或reduce函数合并文件.

# compile the list of dataframes you want to merge
data_frames = [df1, df2, df3]

您可以在上面的代码中添加任意数量的数据帧.这是这种方法的好处.不涉及复杂的查询.

要保留属于同一日期的值,您需要将其合并到 DATE

df_merged = reduce(lambda  left,right: pd.merge(left,right,on=['DATE'],
                                            how='outer'), data_frames)

# if you want to fill the values that don't exist in the lines of merged dataframe simply fill with required strings as

df_merged = reduce(lambda  left,right: pd.merge(left,right,on=['DATE'],
                                            how='outer'), data_frames).fillna('void')

• 因此,同一日期的值在同一行.
• 您可以使用fillna()从不同的帧填充不同列的非现有数据.

然后,如果需要,将合并的数据写入csv文件.

pd.DataFrame.to_csv(df_merged, 'merged.txt', sep=',', na_rep='.', index=False)

这应该给你

DATE VALUE1 VALUE2 VALUE3 ....

Answer 2

另一种组合方式：functools.reduce

来自文档：

例如，reduce(lambda x, y: x+y, [1, 2, 3, 4, 5])计算 ((((1+2)+3)+4)+5)。左侧参数 x 是累积值，右侧参数 y 是可迭代的更新值。

所以：

from functools import reduce
dfs = [df1, df2, df3, df4, df5, df6]
df_final = reduce(lambda left,right: pd.merge(left,right,on='some_common_column_name'), dfs)

Answer 3

有两种解决方案，但它分别返回所有列：

import functools

dfs = [df1, df2, df3]

df_final = functools.reduce(lambda left,right: pd.merge(left,right,on='date'), dfs)
print (df_final)
          date     a_x   b_x       a_y      b_y   c_x         a        b   c_y
0  May 15,2017  900.00  0.2%  1,900.00  1000000  0.2%  2,900.00  2000000  0.2%

k = np.arange(len(dfs)).astype(str)
df = pd.concat([x.set_index('date') for x in dfs], axis=1, join='inner', keys=k)
df.columns = df.columns.map('_'.join)
print (df)
                0_a   0_b       1_a      1_b   1_c       2_a      2_b   2_c
date                                                                       
May 15,2017  900.00  0.2%  1,900.00  1000000  0.2%  2,900.00  2000000  0.2%

Answer 4

看起来数据具有相同的列，因此您可以：

df1 = pd.DataFrame(data1)
df2 = pd.DataFrame(data2)

merged_df = pd.concat([df1, df2])

Answer 5

functools.reduce和 pd.concat是很好的解决方案，但是就执行时间而言，pd.concat是最好的。

from functools import reduce
import pandas as pd

dfs = [df1, df2, df3, ...]
nan_value = 0

# solution 1 (fast)
result_1 = pd.concat(dfs, join='outer', axis=1).fillna(nan_value)

# solution 2
result_2 = reduce(lambda left,right: pd.merge(df_left, df_right, 
                                              left_index=True, right_index=True, 
                                              how='outer'), 
                  dfs).fillna(nan_value)

Answer 6

您也可以像这样使用dataframe.merge

df = df1.merge(df2).merge(df3)

更新

将此方法的性能与当前接受的答案进行比较

import timeit

setup = '''import pandas as pd
from functools import reduce
df_1 = pd.DataFrame({'date': {0: 'May 19, 2017', 1: 'May 18, 2017', 2: 'May 17, 2017', 3: 'May 15, 2017'}, 'a': {0: '1,200.00', 1: '1,100.00', 2: '1,000.00', 3: '1,901.00'}, 'b': {0: '0.1%', 1: '0.1%', 2: '0.1%', 3: '0.1%'}})
df_2 = pd.DataFrame({'date': {0: 'May 20, 2017', 1: 'May 18, 2017', 2: 'May 16, 2017', 3: 'May 15, 2017'}, 'a': {0: '2,200.00', 1: '2,100.00', 2: '2,000.00', 3: '2,902.00'}, 'b': {0: 1000000, 1: 1590000, 2: 1230000, 3: 1000000}, 'c': {0: '0.2%', 1: '0.2%', 2: '0.2%', 3: '0.2%'}})
df_3 = pd.DataFrame({'date': {0: 'May 21, 2017', 1: 'May 17, 2017', 2: 'May 16, 2017', 3: 'May 15, 2017'}, 'a': {0: '3,200.00', 1: '3,100.00', 2: '3,000.00', 3: '3,903.00'}, 'b': {0: 2000000, 1: 2590000, 2: 2230000, 3: 2000000}, 'c': {0: '0.3%', 1: '0.3%', 2: '0.3%', 3: '0.3%'}})
dfs = [df_1, df_2, df_3]'''


#methods from currently accepted answer
>>> timeit.timeit(setup=setup, stmt="reduce(lambda  left,right: pd.merge(left,right,on=['date'], how='outer'), dfs)", number=1000)
3.3471919000148773
>>> timeit.timeit(setup=setup, stmt="df_merged = reduce(lambda  left,right: pd.merge(left,right,on=['date'], how='outer'), dfs).fillna('void')", number=1000)
4.079146400094032

#method demonstrated in this answer
>>> timeit.timeit(setup=setup, stmt="df = df_1.merge(df_2, on='date').merge(df_3, on='date')", number=1000)
2.7787032001651824

Answer 7

@dannyeuu 的答案是正确的。如果将 axis 选项设置为 1，pd.concat 自然会对索引列进行联接。默认值是外联接，但您也可以指定内联接。这是一个例子：

x = pd.DataFrame({'a': [2,4,3,4,5,2,3,4,2,5], 'b':[2,3,4,1,6,6,5,2,4,2], 'val': [1,4,4,3,6,4,3,6,5,7], 'val2': [2,4,1,6,4,2,8,6,3,9]})
x.set_index(['a','b'], inplace=True)
x.sort_index(inplace=True)

y = x.__deepcopy__()
y.loc[(14,14),:] = [3,1]
y['other']=range(0,11)

y.sort_values('val', inplace=True)

z = x.__deepcopy__()
z.loc[(15,15),:] = [3,4]
z['another']=range(0,22,2)
z.sort_values('val2',inplace=True)


pd.concat([x,y,z],axis=1)

Answer 8

看看这个pandas 三向连接列上的多个数据框

filenames = ['fn1', 'fn2', 'fn3', 'fn4',....]
dfs = [pd.read_csv(filename, index_col=index_col) for filename in filenames)]
dfs[0].join(dfs[1:])