Hal*_*lee 4 python machine-learning pandas scikit-learn
我开发了一个用于多标签分类的文本模型。该OneVsRestClassifier LinearSVC模型使用sklearnsPipeline并FeatureUnion为模型准备。
主要输入特征由一个名为的文本列和response5 个主题概率（从先前的 LDA 主题模型生成）组成t1_prob，t5_prob用于预测 5 个可能的标签。在管道中还有其他特征创建步骤用于生成TfidfVectorizer.
我最终使用ItemSelector调用每一列并在这些主题概率列上分别执行 ArrayCaster（请参阅下面的代码以了解函数定义）5 次。有没有更好的方法来使用FeatureUnion来选择管道中的多个列？（所以我不必做5次）
我想知道是否有必要复制topic1_feature-topic5_feature代码或者是否可以以更简洁的方式选择多列？
我输入的数据是 Pandas 数据帧：
id response label_1 label_2 label3 label_4 label_5 t1_prob t2_prob t3_prob t4_prob t5_prob
1 Text from response... 0.0 0.0 0.0 0.0 0.0 0.0 0.0625 0.0625 0.1875 0.0625 0.1250
2 Text to model with... 0.0 0.0 0.0 0.0 0.0 0.0 0.1333 0.1333 0.0667 0.0667 0.0667
3 Text to work with ... 0.0 0.0 0.0 0.0 0.0 0.0 0.1111 0.0938 0.0393 0.0198 0.2759
4 Free text comments ... 0.0 0.0 1.0 1.0 0.0 0.0 0.2162 0.1104 0.0341 0.0847 0.0559
x_train isresponse和 5 个主题概率列（t1_prob、t2_prob、t3_prob、t4_prob、t5_prob）。
y_train 是label我调用的 5列，.values用于返回 DataFrame 的 numpy 表示。(label_1, label_2, label3, label_4, label_5)
示例数据帧：
import pandas as pd
column_headers = ["id", "response",
"label_1", "label_2", "label3", "label_4", "label_5",
"t1_prob", "t2_prob", "t3_prob", "t4_prob", "t5_prob"]
input_data = [
[1, "Text from response",0.0,0.0,1.0,0.0,0.0,0.0625,0.0625,0.1875,0.0625,0.1250],
[2, "Text to model with",0.0,0.0,0.0,0.0,0.0,0.1333,0.1333,0.0667,0.0667,0.0667],
[3, "Text to work with",0.0,0.0,0.0,0.0,0.0,0.1111,0.0938,0.0393,0.0198,0.2759],
[4, "Free text comments",0.0,0.0,1.0,1.0,1.0,0.2162,0.1104,0.0341,0.0847,0.0559]
]
df = pd.DataFrame(input_data, columns = column_headers)
df = df.set_index('id')
df
我认为我的实现有点绕，因为 FeatureUnion 在组合它们时只会处理二维数组，所以像 DataFrame 这样的任何其他类型对我来说都是有问题的。然而，这个例子是有效的——我只是在寻找改进它的方法，让它更干燥。
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.base import BaseEstimator, TransformerMixin
class ItemSelector(BaseEstimator, TransformerMixin):
def __init__(self, column):
self.column = column
def fit(self, X, y=None):
return self
def transform(self, X, y=None):
return X[self.column]
class ArrayCaster(BaseEstimator, TransformerMixin):
def fit(self, x, y=None):
return self
def transform(self, data):
return np.transpose(np.matrix(data))
def basic_text_model(trainX, testX, trainY, testY, classLabels, plotPath):
'''OneVsRestClassifier for multi-label prediction'''
pipeline = Pipeline([
('features', FeatureUnion([
('topic1_feature', Pipeline([
('selector', ItemSelector(column='t1_prob')),
('caster', ArrayCaster())
])),
('topic2_feature', Pipeline([
('selector', ItemSelector(column='t2_prob')),
('caster', ArrayCaster())
])),
('topic3_feature', Pipeline([
('selector', ItemSelector(column='t3_prob')),
('caster', ArrayCaster())
])),
('topic4_feature', Pipeline([
('selector', ItemSelector(column='t4_prob')),
('caster', ArrayCaster())
])),
('topic5_feature', Pipeline([
('selector', ItemSelector(column='t5_prob')),
('caster', ArrayCaster())
])),
('word_features', Pipeline([
('vect', CountVectorizer(analyzer="word", stop_words='english')),
('tfidf', TfidfTransformer(use_idf = True)),
])),
])),
('clf', OneVsRestClassifier(svm.LinearSVC(random_state=random_state)))
])
# Fit the model
pipeline.fit(trainX, trainY)
predicted = pipeline.predict(testX)
我将 ArrayCaster 纳入流程源于这个答案。
我使用受@Marcus V 对这个问题的解决方案启发的FunctionTransformer找出了这个问题的答案。修订后的管道更加简洁。
from sklearn.preprocessing import FunctionTransformer
get_numeric_data = FunctionTransformer(lambda x: x[['t1_prob', 't2_prob', 't3_prob', 't4_prob', 't5_prob']], validate=False)
pipeline = Pipeline(
[
(
"features",
FeatureUnion(
[
("numeric_features", Pipeline([("selector", get_numeric_data)])),
(
"word_features",
Pipeline(
[
("vect", CountVectorizer(analyzer="word", stop_words="english")),
("tfidf", TfidfTransformer(use_idf=True)),
]
),
),
]
),
),
("clf", OneVsRestClassifier(svm.LinearSVC(random_state=10))),
]
)