Python NLTK pos_tag没有返回正确的词性标签

Question

有这个:

text = word_tokenize("The quick brown fox jumps over the lazy dog")

并运行:

nltk.pos_tag(text)

我明白了:

[('The', 'DT'), ('quick', 'NN'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'NN'), ('dog', 'NN')]

这是不正确的.quick brown lazy句子中的标签应为:

('quick', 'JJ'), ('brown', 'JJ') , ('lazy', 'JJ')

通过在线工具进行测试可以得到相同的结果; quick,brown而fox应该是形容词不是名词.

Answer 1

简而言之:

NLTK并不完美.事实上,没有任何模型是完美的.

注意:

从NLTK 3.1版开始,默认pos_tag功能不再是旧的MaxEnt英文泡菜.

它现在是来自@Honnibal实施的感知器标记器,请参阅nltk.tag.pos_tag

>>> import inspect
>>> print inspect.getsource(pos_tag)
def pos_tag(tokens, tagset=None):
    tagger = PerceptronTagger()
    return _pos_tag(tokens, tagset, tagger) 

它还是更好但不完美:

>>> from nltk import pos_tag
>>> pos_tag("The quick brown fox jumps over the lazy dog".split())
[('The', 'DT'), ('quick', 'JJ'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'VBZ'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN')]

在某些时候,如果有人想要TL;DR解决方案,请参阅https://github.com/alvations/nltk_cli

长期:

尝试使用其他标记器(请参阅https://github.com/nltk/nltk/tree/develop/nltk/tag),例如:

• HunPos
• 斯坦福POS
• 塞纳

使用NLTK的默认MaxEnt POS标记,即nltk.pos_tag:

>>> from nltk import word_tokenize, pos_tag
>>> text = "The quick brown fox jumps over the lazy dog"
>>> pos_tag(word_tokenize(text))
[('The', 'DT'), ('quick', 'NN'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'NN'), ('dog', 'NN')]

使用Stanford POS标签:

$ cd ~
$ wget http://nlp.stanford.edu/software/stanford-postagger-2015-04-20.zip
$ unzip stanford-postagger-2015-04-20.zip
$ mv stanford-postagger-2015-04-20 stanford-postagger
$ python
>>> from os.path import expanduser
>>> home = expanduser("~")
>>> from nltk.tag.stanford import POSTagger
>>> _path_to_model = home + '/stanford-postagger/models/english-bidirectional-distsim.tagger'
>>> _path_to_jar = home + '/stanford-postagger/stanford-postagger.jar'
>>> st = POSTagger(path_to_model=_path_to_model, path_to_jar=_path_to_jar)
>>> text = "The quick brown fox jumps over the lazy dog"
>>> st.tag(text.split())
[(u'The', u'DT'), (u'quick', u'JJ'), (u'brown', u'JJ'), (u'fox', u'NN'), (u'jumps', u'VBZ'), (u'over', u'IN'), (u'the', u'DT'), (u'lazy', u'JJ'), (u'dog', u'NN')]

使用HunPOS(注意:默认编码是ISO-8859-1而不是UTF8):

$ cd ~
$ wget https://hunpos.googlecode.com/files/hunpos-1.0-linux.tgz
$ tar zxvf hunpos-1.0-linux.tgz
$ wget https://hunpos.googlecode.com/files/en_wsj.model.gz
$ gzip -d en_wsj.model.gz 
$ mv en_wsj.model hunpos-1.0-linux/
$ python
>>> from os.path import expanduser
>>> home = expanduser("~")
>>> from nltk.tag.hunpos import HunposTagger
>>> _path_to_bin = home + '/hunpos-1.0-linux/hunpos-tag'
>>> _path_to_model = home + '/hunpos-1.0-linux/en_wsj.model'
>>> ht = HunposTagger(path_to_model=_path_to_model, path_to_bin=_path_to_bin)
>>> text = "The quick brown fox jumps over the lazy dog"
>>> ht.tag(text.split())
[('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN')]

使用Senna(确保您拥有最新版本的NLTK,对API进行了一些更改):

$ cd ~
$ wget http://ronan.collobert.com/senna/senna-v3.0.tgz
$ tar zxvf senna-v3.0.tgz
$ python
>>> from os.path import expanduser
>>> home = expanduser("~")
>>> from nltk.tag.senna import SennaTagger
>>> st = SennaTagger(home+'/senna')
>>> text = "The quick brown fox jumps over the lazy dog"
>>> st.tag(text.split())
[('The', u'DT'), ('quick', u'JJ'), ('brown', u'JJ'), ('fox', u'NN'), ('jumps', u'VBZ'), ('over', u'IN'), ('the', u'DT'), ('lazy', u'JJ'), ('dog', u'NN')]

或尝试构建更好的POS标记器:

• Ngram Tagger:http://streamhacker.com/2008/11/03/part-of-speech-tagging-with-nltk-part-1/
• Affix/Regex Tagger:http://streamhacker.com/2008/11/10/part-of-speech-tagging-with-nltk-part-2/
• 建立自己的Brill(阅读代码,这是一个非常有趣的标记,http: //www.nltk.org/_modules/nltk/tag/brill.html ),请参阅http://streamhacker.com/2008/12/03/部分的语音标记与- NLTK部分-3 /
• Perceptron Tagger:https://honnibal.wordpress.com/2013/09/11/a-good-part-of-speechpos-tagger-in-about-200-lines-of-python/
• LDA Tagger:http://scm.io/blog/hack/2015/02/lda-intentions/

pos_tag关于stackoverflow的准确性的抱怨包括:

• POS标记 - NLTK认为名词是形容词
• python NLTK POS标记符的行为不符合预期
• 如何使用NLTK pos标签获得更好的结果
• NLTK中的pos_tag不能正确标记句子

关于NLTK HunPos的问题包括:

• 如何在nltk中使用hunpos标记文本文件？
• 有谁知道如何在nltk上配置hunpos包装器类？

NLTK和Stanford POS标记器的问题包括:

• 麻烦将stanford pos tagger导入nltk
• NLTK Stanford POS Tagger中的Java命令失败
• 在NLTK Python中使用Stanford POS Tagger时出错
• 如何使用Stanford NLP Tagger和NLTK提高速度
• Nltk stanford pos tagger错误:Java命令失败
• 在NLTK中实例化和使用StanfordTagger
• 在NLTK中运行Stanford POS标记符导致Windows上"不是有效的Win32应用程序"