分词时结合单数和复数、动词和副词的 nltk 频率

Question

我想计算频率，但我想组合名词和动词的单数和复数形式及其副词形式。请原谅糟糕的句子。例如：“那个咄咄逼人的人走过那边的房子，这是众多房子之一，咄咄逼人。”

标记化和计数频率

import nltk
from nltk.tokenize import RegexpTokenizer
test = "That aggressive person walk by the house over there, one of many houses aggressively"
tokenizer = RegexpTokenizer(r'\w+')
tokens = tokenizer.tokenize(test)
fdist = nltk.FreqDist(tokens)
common=fdist.most_common(100)

输出： [('houses', 1), ('aggressively', 1), ('by', 1), ('That', 1), ('house', 1), ('over', 1), ('there', 1), ('walk', 1), ('person', 1), ('many', 1), ('of', 1), ('aggressive', 1), ('one', 1), ('the', 1)]

我希望house和houses被算作('house\houses', 2)和aggressive被aggressively算作('aggressive\aggressively',2)。这可能吗？如果不是，我该如何让它看起来像那样？

Answer 1

你需要进行词形还原。

NLTK 包括一个基于 WordNet 的词形还原器：

import nltk
tokenizer = nltk.tokenize.RegexpTokenizer(r'\w+')
lemmatizer = nltk.stem.WordNetLemmatizer()
test = "That aggressive person walk by the house over there, one of many houses aggressively"
tokens = tokenizer.tokenize(test)
lemmas = [lemmatizer.lemmatize(t) for t in tokens]
fdist = nltk.FreqDist(lemmas)
common = fdist.most_common(100)

这导致：

[('house', 2),
 ('aggressively', 1),
 ('by', 1),
 ('That', 1),
 ('over', 1),
 ('there', 1),
 ('walk', 1),
 ('person', 1),
 ('many', 1),
 ('of', 1),
 ('aggressive', 1),
 ('one', 1),
 ('the', 1)]

然而，WordNet 词形还原器不会合并aggressive和aggressively 。还有其他词形还原器，它们可能可以满足您的需求。不过，首先，您可能需要考虑词干：

stemmer = nltk.stem.PorterStemmer()
stems = [stemmer.stem(t) for t in tokens]
nltk.FreqDist(stems).most_common()

这给你：

[(u'aggress', 2),
 (u'hous', 2),
 (u'there', 1),
 (u'That', 1),
 (u'of', 1),
 (u'over', 1),
 (u'walk', 1),
 (u'person', 1),
 (u'mani', 1),
 (u'the', 1),
 (u'one', 1),
 (u'by', 1)]

现在计数看起来没问题！然而，您可能会因为词干不一定看起来像真实的单词而感到恼火......