tf-idf 模型如何处理测试数据期间看不见的单词？

Question

我已经阅读了很多博客，但对答案并不满意，假设我在几个文档示例上训练 tf-idf 模型：

   " John like horror movie."
   " Ryan watches dramatic movies"
    ------------so on ----------

我使用这个功能：

   from sklearn.feature_extraction.text import TfidfTransformer
   count_vect = CountVectorizer()
   X_train_counts = count_vect.fit_transform(twenty_train.data)
   X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
   print((X_train_counts.todense()))
   # Gives count of words in each document

   But it doesn't tell which word? How to get words as headers in X_train_counts 
  outputs. Similarly in X_train_tfidf ?

所以 X_train_tfidf 输出将是具有 tf-idf 分数的矩阵：

     Horror  watch  movie  drama
doc1  score1  --    -----------
doc2   ------------------------

这样对吗？

做什么fit和做什么transformation？在 sklearn 中提到：

fit(..) 方法使我们的估计器适合数据，其次是 transform(..) 方法将我们的计数矩阵转换为 tf-idf 表示。是什么estimator to the data意思？

现在假设新的测试文件来了：

    " Ron likes thriller movies"

如何将此文档转换为 tf-idf？我们不能将其转换为 tf-idf 对吗？如何处理thriller训练文档中没有的单词。

Answer 1

以两个文本作为输入

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

text = ["John like horror movie","Ryan watches dramatic movies"]

count_vect = CountVectorizer()
tfidf_transformer = TfidfTransformer()
X_train_counts = count_vect.fit_transform(text)
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)

pd.DataFrame(X_train_tfidf.todense(), columns = count_vect.get_feature_names())

o/p

        dramatic    horror      john        like        movie       movies      ryan    watches
   0    0.000000    0.471078    0.471078    0.471078    0.471078    0.335176    0.000000    0.000000
   1    0.363788    0.000000    0.000000    0.000000    0.000000    0.776515    0.363788    0.363788

现在测试它的新评论，我们需要使用转换函数，在向量化时，词汇表外的单词将被忽略。

new_comment = ["ron don't like dramatic movie"]

pd.DataFrame(tfidf_transformer.transform(count_vect.transform(new_comment)).todense(), columns = count_vect.get_feature_names())


    dramatic    horror  john    like    movie   movies  ryan    watches
0   0.57735      0.0    0.0    0.57735  0.57735   0.0   0.0      0.0

如果你想使用某个单词的词汇表，那么准备你想要使用的单词列表，并不断地将新单词添加到这个列表中并将列表传递给 CountVectorizer

 vocabulary = ['dramatic', 'movie','horror']
 vocabulary.append('Thriller')
 count_vect = CountVectorizer(vocabulary = vocabulary)
 cont_vect.fit_transform(text)