Jar*_*rad 5 python scikit-learn
这个问题不像有人建议的那样重复.为什么？因为在该示例中,所有可能的值都是已知的.在这个例子中,它们不是.此外,这个问题 - 除了在未知值上使用自定义转换器 - 还要具体询问如何以与初始变换相同的方式执行变换.我再一次可以告诉我最终必须回答我自己的问题.
在创建自定义scikit-learn变换器时,如何保证或"强制"转换方法仅输出最初安装的列？
下面说明.这是我的示例变压器.
import numpy as np
import pandas as pd
from sklearn.base import TransformerMixin
from sklearn.linear_model import LogisticRegression
class DFTransformer(TransformerMixin):
def fit(self, df, y=None, **fit_params):
return self
def transform(self, df, **trans_params):
self.df = df
self.STACKER = pd.DataFrame()
for col in self.df:
dtype = self.df[col].dtype.name
if dtype == 'object':
self.STACKER = pd.concat([self.STACKER, self.get_dummies(col)], axis=1)
elif dtype == 'int64':
self.STACKER = pd.concat([self.STACKER, self.cut_it(col)], axis=1)
return self.STACKER
def get_dummies(self, name):
return pd.get_dummies(self.df[name], prefix=name)
def cut_it(self, name, bins=5):
s = self.df[name].copy()
return pd.get_dummies(pd.cut(s, bins), prefix=name)
这是一些虚拟数据.我的一种方法是使用pd.cutin-effort来存储大范围的整数或浮点数.另一种方法使用pd.get_dummies将唯一值转换为列.
df = pd.DataFrame({'integers': np.random.randint(2000, 20000, 30, dtype='int64'),
'categorical': np.random.choice(list('ABCDEFGHIJKLMNOP'), 30)},
columns=['integers', 'categorical'])
trans = DFTransformer()
X = trans.fit_transform(df)
y = np.random.binomial(1, 0.5, 30)
lr = LogisticRegression()
lr.fit(X, y)
X_test = pd.DataFrame({'integers': np.random.randint(2000, 60000, 30, dtype='int64'),
'categorical': np.random.choice(list('ABGIOPXYZ'), 30)},
columns=['integers', 'categorical'])
lr.predict(trans.transform(X_test))
我遇到的问题是,当我转换"测试"数据(我想要预测的数据)时,由于不同的分类,转换很可能不会输出相同的精确列值(例如:可能出现一次然后再也看不到或再次听到的模糊值).
例如,上面的代码会产生以下错误:
Traceback (most recent call last):
File "C:/Users/myname/Downloads/SO009949884.py", line 44, in <module>
lr.predict(trans.transform(X_test))
File "C:\python36\lib\site-packages\sklearn\linear_model\base.py", line 324, in predict
scores = self.decision_function(X)
File "C:\python36\lib\site-packages\sklearn\linear_model\base.py", line 305, in decision_function
% (X.shape[1], n_features))
ValueError: X has 14 features per sample; expecting 20
问题:如何确保我的转换方法以相同的方式转换我的测试数据？
我能想到的一个不好的解决方案是:转换训练数据,转换测试数据,查看列相交的位置,修改我的转换函数以限制输出到这些列.或者,为缺少的列填写空白列.这不可扩展.当然有更好的方法吗？我不想知道输出列必须是什么.
我的总体目标是在列车和测试数据集之间以一致的方式转换分类变量.我有150多个专栏要改造!
我发了一篇博文来解决这个问题.下面是我建的变压器.
class CategoryGrouper(BaseEstimator, TransformerMixin):
"""A tranformer for combining low count observations for categorical features.
This transformer will preserve category values that are above a certain
threshold, while bucketing together all the other values. This will fix issues
where new data may have an unobserved category value that the training data
did not have.
"""
def __init__(self, threshold=0.05):
"""Initialize method.
Args:
threshold (float): The threshold to apply the bucketing when
categorical values drop below that threshold.
"""
self.d = defaultdict(list)
self.threshold = threshold
def transform(self, X, **transform_params):
"""Transforms X with new buckets.
Args:
X (obj): The dataset to pass to the transformer.
Returns:
The transformed X with grouped buckets.
"""
X_copy = X.copy()
for col in X_copy.columns:
X_copy[col] = X_copy[col].apply(lambda x: x if x in self.d[col] else 'CategoryGrouperOther')
return X_copy
def fit(self, X, y=None, **fit_params):
"""Fits transformer over X.
Builds a dictionary of lists where the lists are category values of the
column key for preserving, since they meet the threshold.
"""
df_rows = len(X.index)
for col in X.columns:
calc_col = X.groupby(col)[col].agg(lambda x: (len(x) * 1.0) / df_rows)
self.d[col] = calc_col[calc_col >= self.threshold].index.tolist()
return self
基本上,动机最初来自我必须处理稀疏类别值,但后来我意识到这可以应用于未知值.在给定阈值的情况下,变换器基本上将稀疏类别值组合在一起,因此,由于未知值将继承值空间的0%,因此它们将被CategoryGrouperOther分组到一个组中.
这只是变压器的演示:
# dfs with 100 elements in cat1 and cat2
# note how df_test has elements 'g' and 't' in the respective categories (unknown values)
df_train = pd.DataFrame({'cat1': ['a'] * 20 + ['b'] * 30 + ['c'] * 40 + ['d'] * 3 + ['e'] * 4 + ['f'] * 3,
'cat2': ['z'] * 25 + ['y'] * 25 + ['x'] * 25 + ['w'] * 20 +['v'] * 5})
df_test = pd.DataFrame({'cat1': ['a'] * 10 + ['b'] * 20 + ['c'] * 5 + ['d'] * 50 + ['e'] * 10 + ['g'] * 5,
'cat2': ['z'] * 25 + ['y'] * 55 + ['x'] * 5 + ['w'] * 5 + ['t'] * 10})
catgrouper = CategoryGrouper()
catgrouper.fit(df_train)
df_test_transformed = catgrouper.transform(df_test)
df_test_transformed
cat1 cat2
0 a z
1 a z
2 a z
3 a z
4 a z
5 a z
6 a z
7 a z
8 a z
9 a z
10 b z
11 b z
12 b z
13 b z
14 b z
15 b z
16 b z
17 b z
18 b z
19 b z
20 b z
21 b z
22 b z
23 b z
24 b z
25 b y
26 b y
27 b y
28 b y
29 b y
... ... ...
70 CategoryGrouperOther y
71 CategoryGrouperOther y
72 CategoryGrouperOther y
73 CategoryGrouperOther y
74 CategoryGrouperOther y
75 CategoryGrouperOther y
76 CategoryGrouperOther y
77 CategoryGrouperOther y
78 CategoryGrouperOther y
79 CategoryGrouperOther y
80 CategoryGrouperOther x
81 CategoryGrouperOther x
82 CategoryGrouperOther x
83 CategoryGrouperOther x
84 CategoryGrouperOther x
85 CategoryGrouperOther w
86 CategoryGrouperOther w
87 CategoryGrouperOther w
88 CategoryGrouperOther w
89 CategoryGrouperOther w
90 CategoryGrouperOther CategoryGrouperOther
91 CategoryGrouperOther CategoryGrouperOther
92 CategoryGrouperOther CategoryGrouperOther
93 CategoryGrouperOther CategoryGrouperOther
94 CategoryGrouperOther CategoryGrouperOther
95 CategoryGrouperOther CategoryGrouperOther
96 CategoryGrouperOther CategoryGrouperOther
97 CategoryGrouperOther CategoryGrouperOther
98 CategoryGrouperOther CategoryGrouperOther
99 CategoryGrouperOther CategoryGrouperOther
甚至可以在我将阈值设置为0时工作(这将专门将未知值设置为"其他"组,同时保留所有其他类别值).我会提醒您不要将阈值设置为0,因为您的训练数据集不会有"其他"类别,因此调整阈值以将至少一个值标记为"其他"组:
catgrouper = CategoryGrouper(threshold=0)
catgrouper.fit(df_train)
df_test_transformed = catgrouper.transform(df_test)
df_test_transformed
cat1 cat2
0 a z
1 a z
2 a z
3 a z
4 a z
5 a z
6 a z
7 a z
8 a z
9 a z
10 b z
11 b z
12 b z
13 b z
14 b z
15 b z
16 b z
17 b z
18 b z
19 b z
20 b z
21 b z
22 b z
23 b z
24 b z
25 b y
26 b y
27 b y
28 b y
29 b y
... ... ...
70 d y
71 d y
72 d y
73 d y
74 d y
75 d y
76 d y
77 d y
78 d y
79 d y
80 d x
81 d x
82 d x
83 d x
84 d x
85 e w
86 e w
87 e w
88 e w
89 e w
90 e CategoryGrouperOther
91 e CategoryGrouperOther
92 e CategoryGrouperOther
93 e CategoryGrouperOther
94 e CategoryGrouperOther
95 CategoryGrouperOther CategoryGrouperOther
96 CategoryGrouperOther CategoryGrouperOther
97 CategoryGrouperOther CategoryGrouperOther
98 CategoryGrouperOther CategoryGrouperOther
99 CategoryGrouperOther CategoryGrouperOther
| 归档时间： |
|
| 查看次数： |
5339 次 |
| 最近记录： |