Bra*_*ley 129 python dataframe pandas
我正在使用大型csv文件,最后一列的下一行有一个文本字符串,我希望通过特定的分隔符进行拆分.我想知道是否有一种简单的方法可以使用pandas或python来做到这一点？
CustNum CustomerName ItemQty Item Seatblocks ItemExt
32363 McCartney, Paul 3 F04 2:218:10:4,6 60
31316 Lennon, John 25 F01 1:13:36:1,12 1:13:37:1,13 300
我想的空间分割(' '),然后结肠(':')在Seatblocks列,但每个单元格将导致不同的列数.我有一个重新排列列的功能,所以Seatblocks列位于工作表的末尾,但我不知道该怎么做.我可以使用内置text-to-columns函数和快速宏在excel中完成它,但我的数据集有太多的记录供excel处理.
最终,我想记录约翰列侬的记录并创建多条线,每组座位的信息都在一条单独的线上.
Dan*_*lan 186
这会按空格分割Seatblocks并为每个子行分配自己的行.
In [43]: df
Out[43]:
CustNum CustomerName ItemQty Item Seatblocks ItemExt
0 32363 McCartney, Paul 3 F04 2:218:10:4,6 60
1 31316 Lennon, John 25 F01 1:13:36:1,12 1:13:37:1,13 300
In [44]: s = df['Seatblocks'].str.split(' ').apply(Series, 1).stack()
In [45]: s.index = s.index.droplevel(-1) # to line up with df's index
In [46]: s.name = 'Seatblocks' # needs a name to join
In [47]: s
Out[47]:
0 2:218:10:4,6
1 1:13:36:1,12
1 1:13:37:1,13
Name: Seatblocks, dtype: object
In [48]: del df['Seatblocks']
In [49]: df.join(s)
Out[49]:
CustNum CustomerName ItemQty Item ItemExt Seatblocks
0 32363 McCartney, Paul 3 F04 60 2:218:10:4,6
1 31316 Lennon, John 25 F01 300 1:13:36:1,12
1 31316 Lennon, John 25 F01 300 1:13:37:1,13
或者,将每个以冒号分隔的字符串放在其自己的列中:
In [50]: df.join(s.apply(lambda x: Series(x.split(':'))))
Out[50]:
CustNum CustomerName ItemQty Item ItemExt 0 1 2 3
0 32363 McCartney, Paul 3 F04 60 2 218 10 4,6
1 31316 Lennon, John 25 F01 300 1 13 36 1,12
1 31316 Lennon, John 25 F01 300 1 13 37 1,13
这有点难看,但也许有人会用一个更漂亮的解决方案.
Pie*_*ton 51
与丹不同,我认为他的答案相当优雅......但不幸的是,这也是非常低效的.所以,既然问题提到了"一个大的csv文件",那么让我建议尝试一个shell Dan的解决方案:
time python -c "import pandas as pd;
df = pd.DataFrame(['a b c']*100000, columns=['col']);
print df['col'].apply(lambda x : pd.Series(x.split(' '))).head()"
...与此替代方案相比:
time python -c "import pandas as pd;
from scipy import array, concatenate;
df = pd.DataFrame(['a b c']*100000, columns=['col']);
print pd.DataFrame(concatenate(df['col'].apply( lambda x : [x.split(' ')]))).head()"
... 还有这个:
time python -c "import pandas as pd;
df = pd.DataFrame(['a b c']*100000, columns=['col']);
print pd.DataFrame(dict(zip(range(3), [df['col'].apply(lambda x : x.split(' ')[i]) for i in range(3)]))).head()"
第二种只是避免分配100 000系列,这足以使它快10倍左右.但是第三个解决方案,有点讽刺的是浪费了很多对str.split()的调用(每行每列调用一次,所以比其他两个解决方案多三倍),比第一个快大约40倍,因为它甚至可以避免实例化10万个列表.是的,它肯定有点难看......
编辑: 这个答案建议如何使用"to_list()"并避免需要lambda.结果是这样的
time python -c "import pandas as pd;
df = pd.DataFrame(['a b c']*100000, columns=['col']);
print pd.DataFrame(df.col.str.split().tolist()).head()"
这比第三种解决方案更有效,当然也更优雅.
编辑:更简单
time python -c "import pandas as pd;
df = pd.DataFrame(['a b c']*100000, columns=['col']);
print pd.DataFrame(list(df.col.str.split())).head()"
也起作用,几乎同样有效.
编辑: 更简单!并处理NaN(但效率较低):
time python -c "import pandas as pd;
df = pd.DataFrame(['a b c']*100000, columns=['col']);
print df.col.str.split(expand=True).head()"
jez*_*ael 11
import pandas as pd
import numpy as np
df = pd.DataFrame({'ItemQty': {0: 3, 1: 25},
'Seatblocks': {0: '2:218:10:4,6', 1: '1:13:36:1,12 1:13:37:1,13'},
'ItemExt': {0: 60, 1: 300},
'CustomerName': {0: 'McCartney, Paul', 1: 'Lennon, John'},
'CustNum': {0: 32363, 1: 31316},
'Item': {0: 'F04', 1: 'F01'}},
columns=['CustNum','CustomerName','ItemQty','Item','Seatblocks','ItemExt'])
print (df)
CustNum CustomerName ItemQty Item Seatblocks ItemExt
0 32363 McCartney, Paul 3 F04 2:218:10:4,6 60
1 31316 Lennon, John 25 F01 1:13:36:1,12 1:13:37:1,13 300
链接的另一个类似解决方案是使用reset_index和rename:
print (df.drop('Seatblocks', axis=1)
.join
(
df.Seatblocks
.str
.split(expand=True)
.stack()
.reset_index(drop=True, level=1)
.rename('Seatblocks')
))
CustNum CustomerName ItemQty Item ItemExt Seatblocks
0 32363 McCartney, Paul 3 F04 60 2:218:10:4,6
1 31316 Lennon, John 25 F01 300 1:13:36:1,12
1 31316 Lennon, John 25 F01 300 1:13:37:1,13
如果在列中不是 NaN值,则最快的解决方案是使用构造函数的list理解DataFrame:
df = pd.DataFrame(['a b c']*100000, columns=['col'])
In [141]: %timeit (pd.DataFrame(dict(zip(range(3), [df['col'].apply(lambda x : x.split(' ')[i]) for i in range(3)]))))
1 loop, best of 3: 211 ms per loop
In [142]: %timeit (pd.DataFrame(df.col.str.split().tolist()))
10 loops, best of 3: 87.8 ms per loop
In [143]: %timeit (pd.DataFrame(list(df.col.str.split())))
10 loops, best of 3: 86.1 ms per loop
In [144]: %timeit (df.col.str.split(expand=True))
10 loops, best of 3: 156 ms per loop
In [145]: %timeit (pd.DataFrame([ x.split() for x in df['col'].tolist()]))
10 loops, best of 3: 54.1 ms per loop
但是,如果列NaN只包含返回(文档)的str.split参数,它解释了为什么它更慢:expand=TrueDataFrame
df = pd.DataFrame(['a b c']*10, columns=['col'])
df.loc[0] = np.nan
print (df.head())
col
0 NaN
1 a b c
2 a b c
3 a b c
4 a b c
print (df.col.str.split(expand=True))
0 1 2
0 NaN None None
1 a b c
2 a b c
3 a b c
4 a b c
5 a b c
6 a b c
7 a b c
8 a b c
9 a b c
回答这个问题可能为时已晚，但我希望记录 Pandas 的 2 个好的特性：pandas.Series.str.split()使用正则表达式和pandas.Series.explode().
import pandas as pd
import numpy as np
df = pd.DataFrame(
{'CustNum': [32363, 31316],
'CustomerName': ['McCartney, Paul', 'Lennon, John'],
'ItemQty': [3, 25],
'Item': ['F04', 'F01'],
'Seatblocks': ['2:218:10:4,6', '1:13:36:1,12 1:13:37:1,13'],
'ItemExt': [60, 360]
}
)
print(df)
print('-'*80+'\n')
df['Seatblocks'] = df['Seatblocks'].str.split('[ :]')
df = df.explode('Seatblocks').reset_index(drop=True)
cols = list(df.columns)
cols.append(cols.pop(cols.index('CustomerName')))
df = df[cols]
print(df)
print('='*80+'\n')
print(df[df['CustomerName'] == 'Lennon, John'])
输出是：
CustNum CustomerName ItemQty Item Seatblocks ItemExt
0 32363 McCartney, Paul 3 F04 2:218:10:4,6 60
1 31316 Lennon, John 25 F01 1:13:36:1,12 1:13:37:1,13 360
--------------------------------------------------------------------------------
CustNum ItemQty Item Seatblocks ItemExt CustomerName
0 32363 3 F04 2 60 McCartney, Paul
1 32363 3 F04 218 60 McCartney, Paul
2 32363 3 F04 10 60 McCartney, Paul
3 32363 3 F04 4,6 60 McCartney, Paul
4 31316 25 F01 1 360 Lennon, John
5 31316 25 F01 13 360 Lennon, John
6 31316 25 F01 36 360 Lennon, John
7 31316 25 F01 1,12 360 Lennon, John
8 31316 25 F01 1 360 Lennon, John
9 31316 25 F01 13 360 Lennon, John
10 31316 25 F01 37 360 Lennon, John
11 31316 25 F01 1,13 360 Lennon, John
================================================================================
CustNum ItemQty Item Seatblocks ItemExt CustomerName
4 31316 25 F01 1 360 Lennon, John
5 31316 25 F01 13 360 Lennon, John
6 31316 25 F01 36 360 Lennon, John
7 31316 25 F01 1,12 360 Lennon, John
8 31316 25 F01 1 360 Lennon, John
9 31316 25 F01 13 360 Lennon, John
10 31316 25 F01 37 360 Lennon, John
11 31316 25 F01 1,13 360 Lennon, John