Pandas从单列字符串生成列

Question

我有一个csv文件,如下所示:

index,labels
1,created the tower
2,destroyed the tower
3,created the swimming pool
4,destroyed the swimming pool

现在,如果我传递我想要的列的列表代替标签列(不包含标签列中的所有单词)

['created','tower','destroyed','swimming pool']

我想获取数据帧为:

index,created,destroyed,tower,swimming pool
1,1,0,1,0
2,0,1,1,0
3,1,0,0,1
4,0,1,0,1

我调查了get_dummies,但那并没有帮助

Answer 1

import re
import pandas as pd
df = pd.DataFrame({'index': [1, 2, 3, 4], 'labels': ['created the tower', 'destroyed the tower', 'created the swimming pool', 'destroyed the swimming pool']})

columns = ['created','destroyed','tower','swimming pool']
pat = '|'.join(['({})'.format(re.escape(c)) for c in columns])
result = (df['labels'].str.extractall(pat)).groupby(level=0).count()
result.columns = columns
print(result)

产量

   created  destroyed  tower  swimming pool
0        1          0      1              0
1        0          1      1              0
2        1          0      0              1
3        0          1      0              1

大部分工作是通过str.extractall以下方式完成的:

In [808]: df['labels'].str.extractall(r'(created)|(destroyed)|(tower)|(swimming pool)')
Out[808]: 
               0          1      2              3
  match                                          
0 0      created        NaN    NaN            NaN
  1          NaN        NaN  tower            NaN
1 0          NaN  destroyed    NaN            NaN
  1          NaN        NaN  tower            NaN
2 0      created        NaN    NaN            NaN
  1          NaN        NaN    NaN  swimming pool
3 0          NaN  destroyed    NaN            NaN
  1          NaN        NaN    NaN  swimming pool

由于每个匹配都放在它自己的行上,因此可以通过执行groupby/count我们按索引的第一级(原始索引)分组的操作来获得所需的结果.

请注意,Python re模块对允许的命名组数量有一个硬编码限制:

/usr/lib/python3.4/sre_compile.py in compile(p, flags)
    577     if p.pattern.groups > 100:
    578         raise AssertionError(
--> 579             "sorry, but this version only supports 100 named groups"
    580             )
    581 

AssertionError: sorry, but this version only supports 100 named groups

这将extractall上面使用的方法限制为最多100个关键字.

这是一个基准测试,表明cᴏʟᴅsᴘᴇᴇᴅ的解决方案(至少在某些用例范围内)可能是最快的:

In [76]: %timeit using_contains(ser, keywords)
10 loops, best of 3: 63.4 ms per loop

In [77]: %timeit using_defchararray(ser, keywords)
10 loops, best of 3: 90.6 ms per loop

In [78]: %timeit using_extractall(ser, keywords)
10 loops, best of 3: 126 ms per loop

这是我使用的设置:

import string
import numpy as np
import pandas as pd

def using_defchararray(ser, keywords):
    """
    https://stackoverflow.com/a/46046558/190597 (piRSquared)
    """
    v = ser.values.astype(str)
    # >>> (np.core.defchararray.find(v[:, None], columns) >= 0)
    # array([[ True, False,  True, False],
    #        [False,  True,  True, False],
    #        [ True, False, False,  True],
    #        [False,  True, False,  True]], dtype=bool)

    result = pd.DataFrame(
        (np.core.defchararray.find(v[:, None], keywords) >= 0).astype(int),
        index=ser.index, columns=keywords)
    return result

def using_extractall(ser, keywords):
    """
    https://stackoverflow.com/a/46046417/190597 (unutbu)
    """
    pat = '|'.join(['({})'.format(re.escape(c)) for c in keywords])
    result = (ser.str.extractall(pat)).groupby(level=0).count()
    result.columns = keywords
    return result

def using_contains(ser, keywords):
    """
    https://stackoverflow.com/a/46046142/190597 (c???s????)
    """
    return (pd.concat([ser.str.contains(x) for x in keywords], 
                      axis=1, keys=keywords).astype(int))

def make_random_str_array(letters=string.ascii_letters, strlen=10, size=100):
    return (np.random.choice(list(letters), size*strlen)
            .view('|U{}'.format(strlen)))

keywords = make_random_str_array(size=99)
arr = np.random.choice(keywords, size=(1000, 5),replace=True)
ser = pd.Series([' '.join(row) for row in arr])

请务必在您自己的机器上检查基准测试,并使用类似于您的用例的设置.结果可能由于许多因素,如系列的大小而变化,ser,的长度keywords,硬件,操作系统,NumPy的,熊猫和Python的版本,以及它们是如何被编译.

Answer 2

你可以str.contains循环调用.

print(df)

                        labels
0            created the tower
1          destroyed the tower
2    created the swimming pool
3  destroyed the swimming pool

req = ['created', 'destroyed', 'tower', 'swimming pool']

out = pd.concat([df['labels'].str.contains(x) for x in req], 1, keys=req).astype(int)
print(out)

   created  destroyed  tower  swimming pool
0        1          0      1              0
1        0          1      1              0
2        1          0      0              1
3        0          1      0              1

Answer 3

使用numpy.core.defchararray.find和numpybraodcasting

from numpy.core.defchararray import find

v = df['labels'].values.astype(str)
l = ['created','tower','destroyed','swimming pool']

pd.DataFrame(
    (find(v[:, None], l) >= 0).astype(int),
    df.index, l
)

       created  tower  destroyed  swimming pool
index                                          
1            1      1          0              0
2            0      1          1              0
3            1      0          0              1
4            0      0          1              1

find将在str.find我们提供的字符串数组的维度上广播该函数. find从第一个数组返回字符串中的位置,该数组首先找到第二个数组的字符串.如果找不到,则返回-1.因此,我们可以通过评估返回值find是否大于或等于来评估是否找到字符串0.