将功能名称更新为scikit TFIdfVectorizer

Question

我正在尝试这段代码

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

train_data = ["football is the sport","gravity is the movie", "education is imporatant"]
vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5,
                                                 stop_words='english')

print "Applying first train data"
X_train = vectorizer.fit_transform(train_data)
print vectorizer.get_feature_names()

print "\n\nApplying second train data"
train_data = ["cricket", "Transformers is a film","AIMS is a college"]
X_train = vectorizer.transform(train_data)
print vectorizer.get_feature_names()

print "\n\nApplying fit transform onto second train data"
X_train = vectorizer.fit_transform(train_data)
print vectorizer.get_feature_names()

这个的输出是

Applying first train data
[u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport']


Applying second train data
[u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport']


 Applying fit transform onto second train data
[u'aims', u'college', u'cricket', u'film', u'transformers']

我使用fit_transform向vectorizer提供了第一组数据,所以它给了我一些特征名称, [u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport']之后我将另一个火车组应用到同一个矢量器,但它给了我相同的特征名称,因为我没有使用fit或fit_transform.但我想知道如何在不覆盖以前的oncs的情况下更新矢量化器的功能.如果我再次使用fit_transform,之前的功能将被覆盖.所以我想更新矢量化器的功能列表.所以我想要一些像[u'education', u'football', u'gravity', u'imporatant', u'movie', u'sport',u'aims', u'college', u'cricket', u'film', u'transformers']我怎么能得到它.

Answer 1

在 sklearn 术语中，这称为部分拟合，您不能使用TfidfVectorizer. 有两种方法可以解决这个问题：

• 连接两个训练集并重新向量化
• 使用 a HashingVectorizer，它支持部分拟合。get_feature_names但是，由于散列功能，因此没有方法，因此不保留原始内容。另一个优点是内存效率更高。

第一种方法的示例：

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

train_data1 = ["football is the sport", "gravity is the movie", "education is important"]
vectorizer = TfidfVectorizer(stop_words='english')

print("Applying first train data")
X_train = vectorizer.fit_transform(train_data1)
print(vectorizer.get_feature_names())

print("\n\nApplying second train data")
train_data2 = ["cricket", "Transformers is a film", "AIMS is a college"]
X_train = vectorizer.transform(train_data2)
print(vectorizer.get_feature_names())

print("\n\nApplying fit transform onto second train data")
X_train = vectorizer.fit_transform(train_data1 + train_data2)
print(vectorizer.get_feature_names())

输出：

Applying first train data
['education', 'football', 'gravity', 'important', 'movie', 'sport']

Applying second train data
['education', 'football', 'gravity', 'important', 'movie', 'sport']

Applying fit transform onto second train data
['aims', 'college', 'cricket', 'education', 'film', 'football', 'gravity', 'important', 'movie', 'sport', 'transformers']