使用 NLTK 的独特词频

Question

使用 NLTK 获取以下内容的唯一词频的代码。

Seq Sentence
1 Let's try to be Good.
2 Being good doesn't make sense.
3 Good is always good.

输出：
{'good':3, 'let':1, 'try':1, 'to':1, 'be':1, 'being':1, 'doesn':1, 't':1, 'make':1, 'sense':1, 'is':1, 'always':1, '.':3, ''':2, 's':1}

Answer 1

如果您对使用 nltk 非常挑剔，请参考以下代码片段

import nltk

text1 = '''Seq Sentence 
1   Let's try to be Good.
2   Being good doesn't make sense.
3   Good is always good.'''

words = nltk.tokenize.word_tokenize(text1)
fdist1 = nltk.FreqDist(words)

filtered_word_freq = dict((word, freq) for word, freq in fdist1.items() if not word.isdigit())

print(filtered_word_freq)

希望能帮助到你。

参考了一些部分：

如何判断输入的字符串是否为数字？

从 NLTK 分布中删除除停用词之外的特定单词

Answer 2

尝试这个

from collections import Counter
import pandas as pd
import nltk

sno = nltk.stem.SnowballStemmer('english')
s = "1   Let's try to be Good. 2   Being good doesn't make sense. 3   Good is always good."
s1 = s.split(' ')
d = pd.DataFrame(s1)
s2 = d[0].apply(lambda x: sno.stem(x))
counts =  Counter(s2)
print(counts)

输出将是：

Counter({'': 6, 'be': 2, 'good.': 2, 'good': 2, '1': 1, 'let': 1, 'tri': 1, 'to': 1, '2': 1, "doesn't": 1, 'make': 1, 'sense.': 1, '3': 1, 'is': 1, 'alway': 1})