相关疑难解决方法(0)

我使用sklearn.feature_extraction.text.CountVectorizer来计算n-gram.例:

import sklearn.feature_extraction.text # FYI http://scikit-learn.org/stable/install.html
ngram_size = 4
string = ["I really like python, it's pretty awesome."]
vect = sklearn.feature_extraction.text.CountVectorizer(ngram_range=(ngram_size,ngram_size))
vect.fit(string)
print('{1}-grams: {0}'.format(vect.get_feature_names(), ngram_size))

输出:

4-grams: [u'like python it pretty', u'python it pretty awesome', u'really like python it']

删除标点符号:如何将它们作为单独的标记包含在内？

考虑

doc = ["i am a fellow student", "we both are the good student", "a student works hard"]

我有这个作为输入我只想打印整个列表中每个单词出现的次数：

例如 student 出现 3 次所以 预期输出student=3, a=2, etc

我能够打印文档中的唯一单词，但无法打印出现的次数。这是我使用的功能：

def fit(doc):    
    unique_words = set() 
    if isinstance(dataset, (list,)):
        for row in dataset:
            for word in row.split(" "): 
                if len(word) < 2:
                    continue
                unique_words.add(word)
        unique_words = sorted(list(unique_words))
        return (unique_words)
doc=fit(docs)

print(doc)

['am', 'are', 'both', 'fellow', 'good', 'hard', 'student', 'the', 'we', 'works']

我得到这个作为输出我只想要unique_words的出现次数。请问我该怎么做？