如何从管道中提取词汇

Question

我可以通过以下方式从CountVecotizerModel中提取词汇

fl = StopWordsRemover(inputCol="words", outputCol="filtered")
df = fl.transform(df)
cv = CountVectorizer(inputCol="filtered", outputCol="rawFeatures")
model = cv.fit(df)

print(model.vocabulary)

上面的代码将打印带有索引作为ID的词汇表。

现在，我创建了上述代码的管道，如下所示：

rm_stop_words = StopWordsRemover(inputCol="words", outputCol="filtered")
count_freq = CountVectorizer(inputCol=rm_stop_words.getOutputCol(), outputCol="rawFeatures")

pipeline = Pipeline(stages=[rm_stop_words, count_freq])
model = pipeline.fit(dfm)
df = model.transform(dfm)

print(model.vocabulary) # This won't work as it's not CountVectorizerModel

它将引发以下错误

print(len(model.vocabulary))

AttributeError：“ PipelineModel”对象没有属性“ vocabulary”

那么如何从管道中提取Model属性呢？

Answer 1

与其他任何舞台属性一样，方法是stages：

stages = model.stages

找到您感兴趣的一个：

from pyspark.ml.feature import CountVectorizerModel

vectorizers = [s for s in stages if isinstance(s, CountVectorizerModel)]

并获取所需的字段：

[v.vocabulary for v in vectorizers]