CountVectorizer忽略'我'

Question

为什么sklearn中的CountVectorizer忽略了代词"I"？

ngram_vectorizer = CountVectorizer(analyzer = "word", ngram_range = (2,2), min_df = 1)
ngram_vectorizer.fit_transform(['HE GAVE IT TO I'])
<1x3 sparse matrix of type '<class 'numpy.int64'>'
ngram_vectorizer.get_feature_names()
['gave it', 'he gave', 'it to']

Answer 1

默认标记生成器仅考虑2个字符(或更多)字.

你可以通过适当改变这种行为token_pattern给您CountVectorizer.

默认模式是(请参阅文档中的签名):

'token_pattern': u'(?u)\\b\\w\\w+\\b'

您可以CountVectorizer通过更改默认值来删除单字母单词,例如:

from sklearn.feature_extraction.text import CountVectorizer
ngram_vectorizer = CountVectorizer(analyzer="word", ngram_range=(2,2), 
                                   token_pattern=u"(?u)\\b\\w+\\b",min_df=1)
ngram_vectorizer.fit_transform(['HE GAVE IT TO I'])
print(ngram_vectorizer.get_feature_names())

这使:

['gave it', 'he gave', 'it to', 'to i']