无论其上下文如何,NLTK的NgramModel总是给出一个单词的相同概率

Question

我正在使用来自nltk的NgramModel来计算在句子中找到某个单词的概率.我的问题是每个单词每次都给出完全相同的概率,无论上下文如何!这是一些演示我的问题的示例代码.

from nltk.corpus import brown
from nltk.probability import LidstoneProbDist, WittenBellProbDist
from nltk.model import NgramModel

estimator = lambda fdist, bins: LidstoneProbDist(fdist, 0.2)

lm = NgramModel(3, brown.words(categories='news'), estimator=estimator)
>>> print lm.prob("word", ["This is a context which generates a word"])
0.00493261081006
>>> print lm.prob("word", ["This is a context of a word"])
0.00493261081006
>>> print lm.prob("word", ["This word"])
0.00493261081006
>>> print lm.prob("word", ["word"])
0.00493261081006
>>> print lm.prob("word", ["adnga"])
0.00493261081006

Answer 1

单词的上下文不应该包含单词本身,除非你有一个重复的单词.棕色语料库很小,所以除非你碰到数据中实际观察到的三元组,否则你会得到相同的答案.在我的例子中,我使用bigrams代替,以便我不会经常使用平滑模型.在您的示例中,您每次都会遇到平滑模型.第三,在实践中,LidstoneProbDist非常糟糕,它是平滑时最简单的东西,而不是你想要在实践中使用的东西.SimpleGoodTuringProbDist要好得多.

from nltk.corpus import brown
from nltk.probability import LidstoneProbDist
from nltk.model import NgramModel

estimator = lambda fdist, bins: LidstoneProbDist(fdist, 0.2)

lm = NgramModel(2, brown.words(categories='news'), estimator=estimator)

lm.prob("good", ["very"])          # 0.0024521936223426436
lm.prob("good", ["not"])           # 0.0019510849023145812
lm.prob("good", ["unknown_term"])  # 0.017437821314436573