相关疑难解决方法(0)

我正在尝试在PySpark中运行线性回归,我想创建一个包含汇总统计信息的表,例如我的数据集中每列的系数,P值和t值.但是,为了训练线性回归模型,我必须使用Spark创建一个特征向量VectorAssembler,现在对于每一行我都有一个特征向量和目标列.当我尝试访问Spark的内置回归摘要统计信息时,它们会为每个统计信息提供一个非常原始的数字列表,并且无法知道哪个属性对应于哪个值,这很难通过手动计算出来大量的列.如何将这些值映射回列名？

例如,我的当前输出是这样的:

系数:[ - 187.807832407,-187.058926726,85.1716641376,10595.3352802,-127.258892837,-39.2827730493,-1206.47228704,33.7078197705,99.9956812528]

P值:[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.18589731365614548,0.275173571416679,0.0]

t统计量:[ - 23.348593508995318,-44.72813283953004,19.836508234714472,144.49248881747755,-16.547272230754242,-9.560681351483941,-19.563547400189073,1.3232383890822680,1.0912415361190977,20.383256127350474]

系数标准误差:[8.043646497811427,4.182131353367049,4.293682291754585,73.32793120907755,7.690626652102948,4.108783841348964,61.669402913526625,25.481445101737247,91.63478289909655,609.7007361468519]

除非我知道它们对应哪个属性,否则这些数字毫无意义.但在我看来,DataFrame我只有一个名为"features"的列,其中包含稀疏向量行.

当我有一个热编码特征时,这是一个更大的问题,因为如果我有一个长度为n的编码变量,我会得到n个相应的系数/ p值/ t值等.

我正在尝试提取我使用PySpark训练的随机森林对象的要素重要性.但是,我没有看到在文档中的任何地方执行此操作的示例,也不是RandomForestModel的方法.

如何从RandomForestModelPySpark中的回归器或分类器中提取要素重要性？

以下是文档中提供的示例代码,以帮助我们开始; 但是,没有提到其中的特征重要性.

from pyspark.mllib.tree import RandomForest
from pyspark.mllib.util import MLUtils

# Load and parse the data file into an RDD of LabeledPoint.
data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt')
# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a RandomForest model.
#  Empty categoricalFeaturesInfo indicates all features are continuous.
#  Note: Use larger numTrees in practice.
#  Setting featureSubsetStrategy="auto" lets the algorithm choose.
model = RandomForest.trainClassifier(trainingData, numClasses=2, …

我在spark中使用标准(字符串索引器+一个热编码器+ randomForest)管道,如下所示

labelIndexer = StringIndexer(inputCol = class_label_name, outputCol="indexedLabel").fit(data)

string_feature_indexers = [
   StringIndexer(inputCol=x, outputCol="int_{0}".format(x)).fit(data)
   for x in char_col_toUse_names
]

onehot_encoder = [
   OneHotEncoder(inputCol="int_"+x, outputCol="onehot_{0}".format(x))
   for x in char_col_toUse_names
]
all_columns = num_col_toUse_names + bool_col_toUse_names + ["onehot_"+x for x in char_col_toUse_names]
assembler = VectorAssembler(inputCols=[col for col in all_columns], outputCol="features")
rf = RandomForestClassifier(labelCol="indexedLabel", featuresCol="features", numTrees=100)
labelConverter = IndexToString(inputCol="prediction", outputCol="predictedLabel", labels=labelIndexer.labels)
pipeline = Pipeline(stages=[labelIndexer] + string_feature_indexers + onehot_encoder + [assembler, rf, labelConverter])

crossval = CrossValidator(estimator=pipeline,
                          estimatorParamMaps=paramGrid,
                          evaluator=evaluator,
                          numFolds=3)
cvModel = crossval.fit(trainingData)

现在,在拟合之后,我可以使用随机林和特征重要性cvModel.bestModel.stages[-2].featureImportances,但这不会给我功能/列名称,而只是功能号码. …