Spark CrossValidatorModel访问其他型号比bestModel？

Question

我正在使用Spark 1.6.1:

目前我正在使用CrossValidator来训练我的ML管道,其中包含各种参数.在训练过程之后,我可以使用CrossValidatorModel的bestModel属性来获取在交叉验证期间表现最佳的模型.是否会自动丢弃交叉验证的其他模型,还是可以选择性能比bestModel差的模型？

我问,因为我使用F1分数指标进行交叉验证,但我也对所有模型的weighedRecall感兴趣,而不仅仅是在交叉验证期间表现最佳的模型.

val folds = 6
val cv = new CrossValidator()
  .setEstimator(pipeline)
  .setEvaluator(new MulticlassClassificationEvaluator)
  .setEstimatorParamMaps(paramGrid)
  .setNumFolds(folds)

val avgF1Scores = cvModel.avgMetrics

val predictedDf = cvModel.bestModel.transform(testDf)

// Here I would like to predict as well with the other models of the cross validation

Answer 1

Spark> = 2.4.0(Scala中 > = 2.3.0)

SPARK- 21088 CrossValidator,TrainValidationSplit应该在装配时收集所有模型 - 增加对收集子模型的支持.

cv = CrossValidator(..., collectSubModels=True)

model = cv.fit(...)
model.subModels

Spark <2.4

如果要访问所有中间模型,则必须从头开始创建自定义交叉验证器.o.a.s.ml.tuning.CrossValidator丢弃其他模型,只有最好的一个和指标被复制到CrossValidatorModel.

另请参见Pyspark - 获取使用ParamGridBuilder创建的模型的所有参数

Answer 2

如果你只是想做这个实验而不是生产实现某些东西,我推荐猴子修补.以下是我打印出中级培训结果的方法.只需CrossValidatorVerbose用作替代品即可CrossValidator.

import numpy as np

from pyspark.ml.tuning import CrossValidator, CrossValidatorModel
from pyspark.sql.functions import rand


class CrossValidatorVerbose(CrossValidator):

    def _fit(self, dataset):
        est = self.getOrDefault(self.estimator)
        epm = self.getOrDefault(self.estimatorParamMaps)
        numModels = len(epm)

        eva = self.getOrDefault(self.evaluator)
        metricName = eva.getMetricName()

        nFolds = self.getOrDefault(self.numFolds)
        seed = self.getOrDefault(self.seed)
        h = 1.0 / nFolds

        randCol = self.uid + "_rand"
        df = dataset.select("*", rand(seed).alias(randCol))
        metrics = [0.0] * numModels

        for i in range(nFolds):
            foldNum = i + 1
            print("Comparing models on fold %d" % foldNum)

            validateLB = i * h
            validateUB = (i + 1) * h
            condition = (df[randCol] >= validateLB) & (df[randCol] < validateUB)
            validation = df.filter(condition)
            train = df.filter(~condition)

            for j in range(numModels):
                paramMap = epm[j]
                model = est.fit(train, paramMap)
                # TODO: duplicate evaluator to take extra params from input
                metric = eva.evaluate(model.transform(validation, paramMap))
                metrics[j] += metric

                avgSoFar = metrics[j] / foldNum
                print("params: %s\t%s: %f\tavg: %f" % (
                    {param.name: val for (param, val) in paramMap.items()},
                    metricName, metric, avgSoFar))

        if eva.isLargerBetter():
            bestIndex = np.argmax(metrics)
        else:
            bestIndex = np.argmin(metrics)

        bestParams = epm[bestIndex]
        bestModel = est.fit(dataset, bestParams)
        avgMetrics = [m / nFolds for m in metrics]
        bestAvg = avgMetrics[bestIndex]
        print("Best model:\nparams: %s\t%s: %f" % (
            {param.name: val for (param, val) in bestParams.items()},
            metricName, bestAvg))

        return self._copyValues(CrossValidatorModel(bestModel, avgMetrics))

注意:此解决方案还纠正了我在v2.0.0中看到的错误,其中CrossValidationModel.avgMetrics设置为指标的总和而不是平均值.

以下是一个简单的5倍验证的输出示例ALS:

Comparing models on fold 1
params: {'regParam': 0.1, 'rank': 5, 'maxIter': 10}     rmse: 1.122425  avg: 1.122425
params: {'regParam': 0.01, 'rank': 5, 'maxIter': 10}    rmse: 1.123537  avg: 1.123537
params: {'regParam': 0.001, 'rank': 5, 'maxIter': 10}   rmse: 1.123651  avg: 1.123651
Comparing models on fold 2
params: {'regParam': 0.1, 'rank': 5, 'maxIter': 10}     rmse: 0.992541  avg: 1.057483
params: {'regParam': 0.01, 'rank': 5, 'maxIter': 10}    rmse: 0.992541  avg: 1.058039
params: {'regParam': 0.001, 'rank': 5, 'maxIter': 10}   rmse: 0.992541  avg: 1.058096
Comparing models on fold 3
params: {'regParam': 0.1, 'rank': 5, 'maxIter': 10}     rmse: 1.141786  avg: 1.085584
params: {'regParam': 0.01, 'rank': 5, 'maxIter': 10}    rmse: 1.141786  avg: 1.085955
params: {'regParam': 0.001, 'rank': 5, 'maxIter': 10}   rmse: 1.141786  avg: 1.085993
Comparing models on fold 4
params: {'regParam': 0.1, 'rank': 5, 'maxIter': 10}     rmse: 0.954110  avg: 1.052715
params: {'regParam': 0.01, 'rank': 5, 'maxIter': 10}    rmse: 0.952955  avg: 1.052705
params: {'regParam': 0.001, 'rank': 5, 'maxIter': 10}   rmse: 0.952873  avg: 1.052713
Comparing models on fold 5
params: {'regParam': 0.1, 'rank': 5, 'maxIter': 10}     rmse: 1.140098  avg: 1.070192
params: {'regParam': 0.01, 'rank': 5, 'maxIter': 10}    rmse: 1.139589  avg: 1.070082
params: {'regParam': 0.001, 'rank': 5, 'maxIter': 10}   rmse: 1.139535  avg: 1.070077
Best model:
params: {'regParam': 0.001, 'rank': 5, 'maxIter': 10}   rmse: 1.070077