布尔当量的pandas to_numeric()

Question

我正在寻找pandas to_numeric()的布尔等价物我希望函数将列转换为True/False/nan,如果可能的话,如果没有抛出错误.

我的动机是我需要在数据集中自动识别和转换大约1000列的布尔列.我可以使用以下代码使用浮点数/整数执行类似的操作:

df = df_raw.apply(pd.to_numeric, errors='ignore')

Answer 1

由于pd.to_numeric主要用于将字符串转换为数值,我将假设您要转换字符串布尔值的字符串.

考虑数据帧 df

df = pd.DataFrame([
        ['1', None, 'True'],
        ['False', 2, True]
    ])

print(df)

       0    1     2
0      1  NaN  True
1  False  2.0  True

我的选择
这就是我的建议.在下面,我将其分解,试图解释发生了什么.

def try_eval2(x):
    if type(x) is str:
        try:
            x = literal_eval(x)
        except:
            x = np.nan

    if type(x) is not bool:
        x = np.nan

    return x

vals = df.values
v = vals.ravel()
a = np.array([try_eval2(x) for x in v.tolist()], dtype=object)
pd.DataFrame(a.reshape(vals.shape), df.index, df.columns)

       0    1     2
0    NaN  NaN  True
1  False  NaN  True

时间
你会注意到我提出的解决方案非常快

%%timeit
vals = df.values
v = vals.ravel()
a = np.array([try_eval2(x) for x in v.tolist()], dtype=object)
pd.DataFrame(a.reshape(vals.shape), df.index, df.columns)
10000 loops, best of 3: 149 µs per loop

%timeit df.astype(str).applymap(to_boolean)
1000 loops, best of 3: 1.28 ms per loop

%timeit df.astype(str).stack().map({'True':True, 'False':False}).unstack()
1000 loops, best of 3: 1.27 ms per loop

说明

步骤1
现在我将创建一个简单的函数,ast.literal_eval用于将字符串转换为值

from ast import literal_eval

def try_eval(x):
    try:
        x = literal_eval(x)
    except:
        pass
    return x

第2步
applymap使用我的新功能.它会看起来一样!

d1 = df.applymap(try_eval)
print(d1)

       0    1     2
0      1  NaN  True
1  False  2.0  True

步骤3
使用where和applymap再次查找值的实际位置bool

d2 = d1.where(d1.applymap(type).eq(bool))
print(d2)

       0   1     2
0    NaN NaN  True
1  False NaN  True

步骤4
您可以删除所有列NaN

print(d2.dropna(1, 'all'))

       0     2
0    NaN  True
1  False  True

Answer 2

你需要replace在where哪里替换NaN所有不是boolean:

df = df.replace({'True':True,'False':False})
df = df.where(df.applymap(type) == bool)

旧解决方案(非常慢):

您可以astype为字符串,如果一些布尔中df,applymap使用自定义功能和ast.literal_eval转换:

from ast import literal_eval

def to_boolean(x):
    try:
        x = literal_eval(x)
        if type(x) == bool:
            return x
        else:
            return np.nan
    except:
        x = np.nan
    return x

print (df.astype(str).applymap(to_boolean))
#with borrowing sample from piRSquared
       0   1     2
0    NaN NaN  True
1  False NaN  True

时间:

In [76]: %timeit (jez(df))
1 loop, best of 3: 488 ms per loop

In [77]: %timeit (jez2(df))
1 loop, best of 3: 527 ms per loop

#piRSquared fastest solution
In [78]: %timeit (pir(df))
1 loop, best of 3: 5.42 s per loop

#maxu solution
In [79]: %timeit df.astype(str).stack().map({'True':True, 'False':False}).unstack()
1 loop, best of 3: 1.88 s per loop

#jezrael ols solution
In [80]: %timeit df.astype(str).applymap(to_boolean)
1 loop, best of 3: 13.3 s per loop

时间代码:

df = pd.DataFrame([
        ['True', False, '1', 0, None, 5.2],
        ['False', True, '0', 1, 's', np.nan]])

#[20000 rows x 60 columns]
df = pd.concat([df]*10000).reset_index(drop=True)
df = pd.concat([df]*10, axis=1).reset_index(drop=True)
df.columns = pd.RangeIndex(len(df.columns))
#print (df)

def to_boolean(x):
    try:
        x = literal_eval(x)
        if type(x) == bool:
            return x
        else:
            return np.nan
    except:
        x = np.nan
    return x


def try_eval2(x):
    if type(x) is str:
        try:
            x = literal_eval(x)
        except:
            x = np.nan

    if type(x) is not bool:
        x = np.nan

    return x

def pir(df):
    vals = df.values
    v = vals.ravel()
    a = np.array([try_eval2(x) for x in v.tolist()], dtype=object)
    df2 = pd.DataFrame(a.reshape(vals.shape), df.index, df.columns)
    return (df2)

def jez(df):
    df = df.replace({'True':True,'False':False})
    df = df.where(df.applymap(type) == bool)
    return (df)

def jez2(df):
    df = df.replace({'True':True,'False':False})
    df = df.where(df.applymap(type).eq(bool))
    return (df)