小编har*_*bcn的帖子

我是Spark和PySpark Data Frames和ML的新手.如何为ML库创建自定义交叉验证.我想例如改变训练折叠的形成方式,例如分层分裂.

这是我目前的代码

numFolds = 10
predictions = []

lr = LogisticRegression()\
     .setFeaturesCol("features")\
     .setLabelCol('label')

# Grid search on LR model
lrparamGrid = ParamGridBuilder()\
     .addGrid(lr.regParam, [0.01, 0.1, 0.5, 1.0, 2.0])\
     .addGrid(lr.elasticNetParam, [0.0, 0.1, 0.5, 0.8, 1.0])\
     .addGrid(lr.maxIter, [5, 10, 20])\
     .build()

pipelineModel = Pipeline(stages=[lr])
evaluator = BinaryClassificationEvaluator()

cv = CrossValidator()\
     .setEstimator(pipelineModel)\
     .setEvaluator(evaluator)\
     .setEstimatorParamMaps(lrparamGrid).setNumFolds(5)

# My own Cross-Validation with stratified splits
for i in range(numFolds):
     # Use Stratified indexOfStratifiedSplits
     trainingData = df[df.ID.isin(indexOfStratifiedSplits[i][0])]
     testingData = df[df.ID.isin(indexOfStratifiedSplits[i][1])]

# Training and Grid …

我是Spark的新手,我正在尝试将StandardScaler()应用于DataFrame中的列.

+---------------+
|      DF_column|
+---------------+
| 0.114285714286|
| 0.115702479339|
| 0.267893660532|
|0.0730337078652|
| 0.124309392265|
| 0.365714285714|
| 0.111747851003|
| 0.279538904899|
| 0.134670487106|
| 0.523287671233|
| 0.404011461318|
|          0.375|
| 0.125517241379|
|0.0143266475645|
| 0.313684210526|
| 0.381088825215|
| 0.411428571429|
| 0.327683615819|
| 0.153409090909|
| 0.344827586207|
+---------------+

问题是像这样应用它会给我一个错误,"要求失败:输入列DF_column必须是一个向量列".我尝试使用UDF,但仍然无法正常工作.

scaler = StandardScaler(inputCol='DF_column', 
    outputCol="scaledFeatures",withStd=True, withMean=False)

我做了LIBSVM的例子,但这很简单,TXT文件正在将功能加载为Vectors.