检查scikitlearn不当丢弃的功能

Question

scikit-learn的插补变压器的文档说

当axis = 0时，仅包含适合的缺失值的列在转换时将被丢弃。

由于imputer返回一个numpy数组，我如何检查在插补过程中丢弃了哪些要素，或者相应地检查了在插补之后保留了哪些要素？

这是一个简单的示例：

import pandas as pd
import numpy as np
from sklearn.preprocessing import Imputer

df = pd.DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e'])
df['f'] = len(df3)*['NaN']

这是数据框：

>>> df
      a         b         c         d         e    f
0 -1.284658  0.246541 -1.120987  0.559911 -1.189870  NaN
1  0.773717  0.430597 -0.004346 -1.292080  1.993266  NaN
2  1.418761 -0.004749 -0.181932 -0.305756 -0.135870  NaN
3  0.418673 -0.376318 -0.860783  0.074135 -1.034095  NaN
4 -0.019873  0.006210  0.364384  1.029895 -0.188727  NaN
5  0.903661  0.123575 -0.556970  1.344985 -1.109806  NaN
6 -0.069168 -0.385597  0.684345  0.645920  1.159898  NaN
7  0.695782  0.030239 -0.777304 -0.037102  2.053028  NaN
8 -0.256409  0.106735 -0.729710  0.254626  1.064925  NaN
9  0.235507 -0.087767  0.626121  1.391286  0.449158  NaN

现在，我创造了一个麻烦者imp：

imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit(df)
imputed = imp.transform(df)

这是从插补返回的numpy数组。

>>> imputed
array([[-1.28465763,  0.24654083, -1.12098675,  0.55991059, -1.18986998],
   [ 0.77371694,  0.43059674, -0.0043461 , -1.29208032,  1.99326594],
   [ 1.41876145, -0.0047488 , -0.18193164, -0.30575631, -0.13586974],
   [ 0.41867326, -0.37631792, -0.86078293,  0.07413458, -1.03409532],

Answer 1

如何检查插补过程中丢弃了哪些功能？

包含所有NaN的列将被丢弃。您可以检查此内容，而无需通过fit和进行transform处理df.isnull().all()。在哪里True，这些是将被丢弃的“功能”。

确切的答案是verbose=1像这样添加到您的计算机中：

imp = Imputer(verbose=1)

为了使本示例更加清楚发生了什么，请在df包含all的另一列中添加一个NaN。

df.insert(2, 'g', np.nan)

df 现在看起来像这样：

          a         b   g         c         d         e   f
0 -1.284658  0.246541 NaN -1.120987  0.559911 -1.189870 NaN
1  0.773717  0.430597 NaN -0.004346 -1.292080  1.993266 NaN
2  1.418761 -0.004749 NaN -0.181932 -0.305756 -0.135870 NaN
3  0.418673 -0.376318 NaN -0.860783  0.074135 -1.034095 NaN
4 -0.019873  0.006210 NaN  0.364384  1.029895 -0.188727 NaN
5  0.903661  0.123575 NaN -0.556970  1.344985 -1.109806 NaN
6 -0.069168 -0.385597 NaN  0.684345  0.645920  1.159898 NaN
7  0.695782  0.030239 NaN -0.777304 -0.037102  2.053028 NaN
8 -0.256409  0.106735 NaN -0.729710  0.254626  1.064925 NaN
9  0.235507 -0.087767 NaN  0.626121  1.391286  0.449158 NaN

跑步...

imp.fit(df)
imp.transform(df)

现在输出以下“详细”消息，告诉您哪些列已删除[2 6]：

Warning (from warnings module):
  File "C:\Python34\lib\site-packages\sklearn\preprocessing\imputation.py", line 347
    "observed values: %s" % missing)
UserWarning: Deleting features without observed values: [2 6]
array([[-1.284658,  0.246541, -1.120987,  0.559911, -1.18987 ],
       [ 0.773717,  0.430597, -0.004346, -1.29208 ,  1.993266],
       [ 1.418761, -0.004749, -0.181932, -0.305756, -0.13587 ],
       [ 0.418673, -0.376318, -0.860783,  0.074135, -1.034095],
       [-0.019873,  0.00621 ,  0.364384,  1.029895, -0.188727],
       [ 0.903661,  0.123575, -0.55697 ,  1.344985, -1.109806],
       [-0.069168, -0.385597,  0.684345,  0.64592 ,  1.159898],
       [ 0.695782,  0.030239, -0.777304, -0.037102,  2.053028],
       [-0.256409,  0.106735, -0.72971 ,  0.254626,  1.064925],
       [ 0.235507, -0.087767,  0.626121,  1.391286,  0.449158]])

插补后保留了哪些特征？

插补后剩余的列和值。

使用我以前的df，如果我们添加一些NaN到混合：

df.loc[[1, 7, 3], ['a', 'c', 'e']] = np.nan

df 看起来像这样：

          a         b   g         c         d         e   f
0 -1.284658  0.246541 NaN -1.120987  0.559911 -1.189870 NaN
1       NaN  0.430597 NaN       NaN -1.292080       NaN NaN
2  1.418761 -0.004749 NaN -0.181932 -0.305756 -0.135870 NaN
3       NaN -0.376318 NaN       NaN  0.074135       NaN NaN
4 -0.019873  0.006210 NaN  0.364384  1.029895 -0.188727 NaN
5  0.903661  0.123575 NaN -0.556970  1.344985 -1.109806 NaN
6 -0.069168 -0.385597 NaN  0.684345  0.645920  1.159898 NaN
7       NaN  0.030239 NaN       NaN -0.037102       NaN NaN
8 -0.256409  0.106735 NaN -0.729710  0.254626  1.064925 NaN
9  0.235507 -0.087767 NaN  0.626121  1.391286  0.449158 NaN

重要的是要了解所使用的插补策略。的默认Imputer值为mean。这意味着它将NaN用该给定列的平均值替换这些值。

为了证明这一点，请首先检查每列的均值：

>>> df.mean()
a    0.132546
b    0.008947
g         NaN
c   -0.130678
d    0.366582
e    0.007101
f         NaN
dtype: float64

然后，您可以进行拟合和转换，并查看转换后的估算数据中是否有任何值在imp.statistics_超参数中。

imp = Imputer(verbose=1)
imp.fit(df)
imp.transform(df)

返回以下内容-再次要注意的关键是，这些NaN值已替换mean为给定列的。例如，无论您0.13254586在第一栏中看到的是什么，它们都会出现在第1、3和7行（以前为NaNs）上：

Warning (from warnings module):
  File "C:\Python34\lib\site-packages\sklearn\preprocessing\imputation.py", line 347
    "observed values: %s" % missing)
UserWarning: Deleting features without observed values: [2 6]
array([[-1.284658  ,  0.246541  , -1.120987  ,  0.559911  , -1.18987   ],
       [ 0.13254586,  0.430597  , -0.13067843, -1.29208   ,  0.00710114],
       [ 1.418761  , -0.004749  , -0.181932  , -0.305756  , -0.13587   ],
       [ 0.13254586, -0.376318  , -0.13067843,  0.074135  ,  0.00710114],
       [-0.019873  ,  0.00621   ,  0.364384  ,  1.029895  , -0.188727  ],
       [ 0.903661  ,  0.123575  , -0.55697   ,  1.344985  , -1.109806  ],
       [-0.069168  , -0.385597  ,  0.684345  ,  0.64592   ,  1.159898  ],
       [ 0.13254586,  0.030239  , -0.13067843, -0.037102  ,  0.00710114],
       [-0.256409  ,  0.106735  , -0.72971   ,  0.254626  ,  1.064925  ],
       [ 0.235507  , -0.087767  ,  0.626121  ,  1.391286  ,  0.449158  ]])

如果您想进行布尔比较以了解要估算的值，则可以执行以下操作（不是万无一失，而是一种最可靠的方法）：

np.reshape(np.in1d(imp.transform(df), imp.statistics_), imp.transform(df).shape)
array([[False, False, False, False, False],
       [ True, False,  True, False,  True],
       [False, False, False, False, False],
       [ True, False,  True, False,  True],
       [False, False, False, False, False],
       [False, False, False, False, False],
       [False, False, False, False, False],
       [ True, False,  True, False,  True],
       [False, False, False, False, False],
       [False, False, False, False, False]], dtype=bool)