那些遇到过的问题

nltk:如何防止专有名词的堵塞

Abt*_*Pst 5 python nlp stemming nltk stanford-nlp

我正在尝试使用Stanford POS标记器和NER编写关键字提取程序.对于关键字提取,我只对专有名词感兴趣.这是基本方法

  1. 删除除字母之外的任何内容来清理数据
  2. 删除停用词
  3. 干每个字
  4. 确定每个单词的POS标签
  5. 如果POS标签是名词,则将其提供给NER
  6. 然后,NER将确定该单词是人,组织还是位置

示例代码

docText="'Jack Frost works for Boeing Company. He manages 5 aircraft and their crew in London"

words = re.split("\W+",docText) 

stops = set(stopwords.words("english"))

#remove stop words from the list
words = [w for w in words if w not in stops and len(w) > 2]

# Stemming
pstem = PorterStemmer()

words = [pstem.stem(w) for w in words]    

nounsWeWant = set(['NN' ,'NNS', 'NNP', 'NNPS'])

finalWords = []

stn = StanfordNERTagger('english.all.3class.distsim.crf.ser.gz') 
stp = StanfordPOSTagger('english-bidirectional-distsim.tagger') 

for w in words:
    if stp.tag([w.lower()])[0][1] not in nounsWeWant:
        finalWords.append(w.lower())
    else:
        finalWords.append(w)

finalString = " ".join(finalWords)
print finalString

tagged = stn.tag(finalWords)
print tagged
Run Code Online (Sandbox Code Playgroud)

这给了我

Jack Frost work Boe Compani manag aircraft crew London
[(u'Jack', u'PERSON'), (u'Frost', u'PERSON'), (u'work', u'O'), (u'Boe', u'O'), (u'Compani', u'O'), (u'manag', u'O'), (u'aircraft', u'O'), (u'crew', u'O'), (u'London', u'LOCATION')]
Run Code Online (Sandbox Code Playgroud)

很明显,我不希望波音被阻止.也不是公司.我需要限制单词,因为我的输入可能包含类似的术语Performing.我已经看到PerformingNER作为一个专有名词会被一个词所取代,因此可以被归类为 Organization.因此,首先我阻止所有单词并转换为小写.然后我检查这个单词的POS标签是否是名词.如果是这样,我保持原样.如果没有,我将单词转换为小写并将其添加到将传递给NER的最终单词列表中.

关于如何避免扼杀专有名词的任何想法？

alv*_*vas 3

使用完整的斯坦福 CoreNLP 管道来处理您的 NLP 工具链。避免使用自己的分词器、清理器、POS 标记器等。它不能很好地与 NER 工具配合使用。

wget http://nlp.stanford.edu/software/stanford-corenlp-full-2015-12-09.zip
unzip http://nlp.stanford.edu/software/stanford-corenlp-full-2015-12-09.zip
cd stanford-corenlp-full-2015-12-09
echo "Jack Frost works for Boeing Company. He manages 5 aircraft and their crew in London" > test.txt
java -cp "*" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner,parse,dcoref -file test.txt
cat test.txt.out
Run Code Online (Sandbox Code Playgroud)

[出去]：

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet href="CoreNLP-to-HTML.xsl" type="text/xsl"?>
<root>
  <document>
    <sentences>
      <sentence id="1">
        <tokens>
          <token id="1">
            <word>Jack</word>
            <lemma>Jack</lemma>
            <CharacterOffsetBegin>0</CharacterOffsetBegin>
            <CharacterOffsetEnd>4</CharacterOffsetEnd>
            <POS>NNP</POS>
            <NER>PERSON</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="2">
            <word>Frost</word>
            <lemma>Frost</lemma>
            <CharacterOffsetBegin>5</CharacterOffsetBegin>
            <CharacterOffsetEnd>10</CharacterOffsetEnd>
            <POS>NNP</POS>
            <NER>PERSON</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="3">
            <word>works</word>
            <lemma>work</lemma>
            <CharacterOffsetBegin>11</CharacterOffsetBegin>
            <CharacterOffsetEnd>16</CharacterOffsetEnd>
            <POS>VBZ</POS>
            <NER>O</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="4">
            <word>for</word>
            <lemma>for</lemma>
            <CharacterOffsetBegin>17</CharacterOffsetBegin>
            <CharacterOffsetEnd>20</CharacterOffsetEnd>
            <POS>IN</POS>
            <NER>O</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="5">
            <word>Boeing</word>
            <lemma>Boeing</lemma>
            <CharacterOffsetBegin>21</CharacterOffsetBegin>
            <CharacterOffsetEnd>27</CharacterOffsetEnd>
            <POS>NNP</POS>
            <NER>ORGANIZATION</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="6">
            <word>Company</word>
            <lemma>Company</lemma>
            <CharacterOffsetBegin>28</CharacterOffsetBegin>
            <CharacterOffsetEnd>35</CharacterOffsetEnd>
            <POS>NNP</POS>
            <NER>ORGANIZATION</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="7">
            <word>.</word>
            <lemma>.</lemma>
            <CharacterOffsetBegin>35</CharacterOffsetBegin>
            <CharacterOffsetEnd>36</CharacterOffsetEnd>
            <POS>.</POS>
            <NER>O</NER>
            <Speaker>PER0</Speaker>
          </token>
        </tokens>
        <parse>(ROOT (S (NP (NNP Jack) (NNP Frost)) (VP (VBZ works) (PP (IN for) (NP (NNP Boeing) (NNP Company)))) (. .))) </parse>
        <dependencies type="basic-dependencies">
          <dep type="root">
            <governor idx="0">ROOT</governor>
            <dependent idx="3">works</dependent>
          </dep>
          <dep type="compound">
            <governor idx="2">Frost</governor>
            <dependent idx="1">Jack</dependent>
          </dep>
          <dep type="nsubj">
            <governor idx="3">works</governor>
            <dependent idx="2">Frost</dependent>
          </dep>
          <dep type="case">
            <governor idx="6">Company</governor>
            <dependent idx="4">for</dependent>
          </dep>
          <dep type="compound">
            <governor idx="6">Company</governor>
            <dependent idx="5">Boeing</dependent>
          </dep>
          <dep type="nmod">
            <governor idx="3">works</governor>
            <dependent idx="6">Company</dependent>
          </dep>
          <dep type="punct">
            <governor idx="3">works</governor>
            <dependent idx="7">.</dependent>
          </dep>
        </dependencies>
        <dependencies type="collapsed-dependencies">
          <dep type="root">
            <governor idx="0">ROOT</governor>
            <dependent idx="3">works</dependent>
          </dep>
          <dep type="compound">
            <governor idx="2">Frost</governor>
            <dependent idx="1">Jack</dependent>
          </dep>
          <dep type="nsubj">
            <governor idx="3">works</governor>
            <dependent idx="2">Frost</dependent>
          </dep>
          <dep type="case">
            <governor idx="6">Company</governor>
            <dependent idx="4">for</dependent>
          </dep>
          <dep type="compound">
            <governor idx="6">Company</governor>
            <dependent idx="5">Boeing</dependent>
          </dep>
          <dep type="nmod:for">
            <governor idx="3">works</governor>
            <dependent idx="6">Company</dependent>
          </dep>
          <dep type="punct">
            <governor idx="3">works</governor>
            <dependent idx="7">.</dependent>
          </dep>
        </dependencies>
        <dependencies type="collapsed-ccprocessed-dependencies">
          <dep type="root">
            <governor idx="0">ROOT</governor>
            <dependent idx="3">works</dependent>
          </dep>
          <dep type="compound">
            <governor idx="2">Frost</governor>
            <dependent idx="1">Jack</dependent>
          </dep>
          <dep type="nsubj">
            <governor idx="3">works</governor>
            <dependent idx="2">Frost</dependent>
          </dep>
          <dep type="case">
            <governor idx="6">Company</governor>
            <dependent idx="4">for</dependent>
          </dep>
          <dep type="compound">
            <governor idx="6">Company</governor>
            <dependent idx="5">Boeing</dependent>
          </dep>
          <dep type="nmod:for">
            <governor idx="3">works</governor>
            <dependent idx="6">Company</dependent>
          </dep>
          <dep type="punct">
            <governor idx="3">works</governor>
            <dependent idx="7">.</dependent>
          </dep>
        </dependencies>
      </sentence>
      <sentence id="2">
        <tokens>
          <token id="1">
            <word>He</word>
            <lemma>he</lemma>
            <CharacterOffsetBegin>37</CharacterOffsetBegin>
            <CharacterOffsetEnd>39</CharacterOffsetEnd>
            <POS>PRP</POS>
            <NER>O</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="2">
            <word>manages</word>
            <lemma>manage</lemma>
            <CharacterOffsetBegin>40</CharacterOffsetBegin>
            <CharacterOffsetEnd>47</CharacterOffsetEnd>
            <POS>VBZ</POS>
            <NER>O</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="3">
            <word>5</word>
            <lemma>5</lemma>
            <CharacterOffsetBegin>48</CharacterOffsetBegin>
            <CharacterOffsetEnd>49</CharacterOffsetEnd>
            <POS>CD</POS>
            <NER>NUMBER</NER>
            <NormalizedNER>5.0</NormalizedNER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="4">
            <word>aircraft</word>
            <lemma>aircraft</lemma>
            <CharacterOffsetBegin>50</CharacterOffsetBegin>
            <CharacterOffsetEnd>58</CharacterOffsetEnd>
            <POS>NN</POS>
            <NER>O</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="5">
            <word>and</word>
            <lemma>and</lemma>
            <CharacterOffsetBegin>59</CharacterOffsetBegin>
            <CharacterOffsetEnd>62</CharacterOffsetEnd>
            <POS>CC</POS>
            <NER>O</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="6">
            <word>their</word>
            <lemma>they</lemma>
            <CharacterOffsetBegin>63</CharacterOffsetBegin>
            <CharacterOffsetEnd>68</CharacterOffsetEnd>
            <POS>PRP$</POS>
            <NER>O</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="7">
            <word>crew</word>
            <lemma>crew</lemma>
            <CharacterOffsetBegin>69</CharacterOffsetBegin>
            <CharacterOffsetEnd>73</CharacterOffsetEnd>
            <POS>NN</POS>
            <NER>O</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="8">
            <word>in</word>
            <lemma>in</lemma>
            <CharacterOffsetBegin>74</CharacterOffsetBegin>
            <CharacterOffsetEnd>76</CharacterOffsetEnd>
            <POS>IN</POS>
            <NER>O</NER>
            <Speaker>PER0</Speaker>
          </token>
          <token id="9">
            <word>London</word>
            <lemma>London</lemma>
            <CharacterOffsetBegin>77</CharacterOffsetBegin>
            <CharacterOffsetEnd>83</CharacterOffsetEnd>
            <POS>NNP</POS>
            <NER>LOCATION</NER>
            <Speaker>PER0</Speaker>
          </token>
        </tokens>
        <parse>(ROOT (S (NP (PRP He)) (VP (VBZ manages) (NP (NP (CD 5) (NN aircraft)) (CC and) (NP (NP (PRP$ their) (NN crew)) (PP (IN in) (NP (NNP London)))))))) </parse>
        <dependencies type="basic-dependencies">
          <dep type="root">
            <governor idx="0">ROOT</governor>
            <dependent idx="2">manages</dependent>
          </dep>
          <dep type="nsubj">
            <governor idx="2">manages</governor>
            <dependent idx="1">He</dependent>
          </dep>
          <dep type="nummod">
            <governor idx="4">aircraft</governor>
            <dependent idx="3">5</dependent>
          </dep>
          <dep type="dobj">
            <governor idx="2">manages</governor>
            <dependent idx="4">aircraft</dependent>
          </dep>
          <dep type="cc">
            <governor idx="4">aircraft</governor>
            <dependent idx="5">and</dependent>
          </dep>
          <dep type="nmod:poss">
            <governor idx="7">crew</governor>
            <dependent idx="6">their</dependent>
          </dep>
          <dep type="conj">
            <governor idx="4">aircraft</governor>
            <dependent idx="7">crew</dependent>
          </dep>
          <dep type="case">
            <governor idx="9">London</governor>
            <dependent idx="8">in</dependent>
          </dep>
          <dep type="nmod">
            <governor idx="7">crew</governor>
            <dependent idx="9">London</dependent>
          </dep>
        </dependencies>
        <dependencies type="collapsed-dependencies">
          <dep type="root">
            <governor idx="0">ROOT</governor>
            <dependent idx="2">manages</dependent>
          </dep>
          <dep type="nsubj">
            <governor idx="2">manages</governor>
            <dependent idx="1">He</dependent>
          </dep>
          <dep type="nummod">
            <governor idx="4">aircraft</governor>
            <dependent idx="3">5</dependent>
          </dep>
          <dep type="dobj">
            <governor idx="2">manages</governor>
            <dependent idx="4">aircraft</dependent>
          </dep>
          <dep type="cc">
            <governor idx="4">aircraft</governor>
            <dependent idx="5">and</dependent>
          </dep>
          <dep type="nmod:poss">
            <governor idx="7">crew</governor>
            <dependent idx="6">their</dependent>
          </dep>
          <dep type="conj:and">
            <governor idx="4">aircraft</governor>
            <dependent idx="7">crew</dependent>
          </dep>
          <dep type="case">
            <governor idx="9">London</governor>
            <dependent idx="8">in</dependent>
          </dep>
          <dep type="nmod:in">
            <governor idx="7">crew</governor>
            <dependent idx="9">London</dependent>
          </dep>
        </dependencies>
        <dependencies type="collapsed-ccprocessed-dependencies">
          <dep type="root">
            <governor idx="0">ROOT</governor>
            <dependent idx="2">manages</dependent>
          </dep>
          <dep type="nsubj">
            <governor idx="2">manages</governor>
            <dependent idx="1">He</dependent>
          </dep>
          <dep type="nummod">
            <governor idx="4">aircraft</governor>
            <dependent idx="3">5</dependent>
          </dep>
          <dep type="dobj">
            <governor idx="2">manages</governor>
            <dependent idx="4">aircraft</dependent>
          </dep>
          <dep type="cc">
            <governor idx="4">aircraft</governor>
            <dependent idx="5">and</dependent>
          </dep>
          <dep type="nmod:poss">
            <governor idx="7">crew</governor>
            <dependent idx="6">their</dependent>
          </dep>
          <dep type="dobj" extra="true">
            <governor idx="2">manages</governor>
            <dependent idx="7">crew</dependent>
          </dep>
          <dep type="conj:and">
            <governor idx="4">aircraft</governor>
            <dependent idx="7">crew</dependent>
          </dep>
          <dep type="case">
            <governor idx="9">London</governor>
            <dependent idx="8">in</dependent>
          </dep>
          <dep type="nmod:in">
            <governor idx="7">crew</governor>
            <dependent idx="9">London</dependent>
          </dep>
        </dependencies>
      </sentence>
    </sentences>
    <coreference>
      <coreference>
        <mention representative="true">
          <sentence>1</sentence>
          <start>1</start>
          <end>3</end>
          <head>2</head>
          <text>Jack Frost</text>
        </mention>
        <mention>
          <sentence>2</sentence>
          <start>1</start>
          <end>2</end>
          <head>1</head>
          <text>He</text>
        </mention>
      </coreference>
    </coreference>
  </document>
</root>
Run Code Online (Sandbox Code Playgroud)

或者获取 json 输出：

java -cp "*" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner,parse,dcoref -file test.txt -outputFormat json
Run Code Online (Sandbox Code Playgroud)

如果您确实需要 python 包装器，请参阅https://github.com/smilli/py-corenlp

$ cd stanford-corenlp-full-2015-12-09
$ export CLASSPATH=protobuf.jar:joda-time.jar:jollyday.jar:xom-1.2.10.jar:stanford-corenlp-3.6.0.jar:stanford-corenlp-3.6.0-models.jar:slf4j-api.jar 
$ java -mx4g edu.stanford.nlp.pipeline.StanfordCoreNLPServer &
cd
$ git clone https://github.com/smilli/py-corenlp.git
$ cd py-corenlp
$ python
>>> from corenlp import StanfordCoreNLP
>>> nlp = StanfordCoreNLP('http://localhost:9000')
>>> text = ("Jack Frost works for Boeing Company. He manages 5 aircraft and their crew in London")
>>> output = nlp.annotate(text, properties={'annotators': 'tokenize,ssplit,pos,ner',  'outputFormat': 'json'})
>>> output
{u'sentences': [{u'parse': u'SENTENCE_SKIPPED_OR_UNPARSABLE', u'index': 0, u'tokens': [{u'index': 1, u'word': u'Jack', u'lemma': u'Jack', u'after': u' ', u'pos': u'NNP', u'characterOffsetEnd': 4, u'characterOffsetBegin': 0, u'originalText': u'Jack', u'ner': u'PERSON', u'before': u''}, {u'index': 2, u'word': u'Frost', u'lemma': u'Frost', u'after': u' ', u'pos': u'NNP', u'characterOffsetEnd': 10, u'characterOffsetBegin': 5, u'originalText': u'Frost', u'ner': u'PERSON', u'before': u' '}, {u'index': 3, u'word': u'works', u'lemma': u'work', u'after': u' ', u'pos': u'VBZ', u'characterOffsetEnd': 16, u'characterOffsetBegin': 11, u'originalText': u'works', u'ner': u'O', u'before': u' '}, {u'index': 4, u'word': u'for', u'lemma': u'for', u'after': u' ', u'pos': u'IN', u'characterOffsetEnd': 20, u'characterOffsetBegin': 17, u'originalText': u'for', u'ner': u'O', u'before': u' '}, {u'index': 5, u'word': u'Boeing', u'lemma': u'Boeing', u'after': u' ', u'pos': u'NNP', u'characterOffsetEnd': 27, u'characterOffsetBegin': 21, u'originalText': u'Boeing', u'ner': u'ORGANIZATION', u'before'

归档时间：

查看次数：

1280 次

最近记录：

10 年，2 月 前
相关归档
如何在Python中获取父目录？ 306
计算值在数据帧列中出现的频率 251
如何JSON序列化集？ 138
在Python中解析YAML文件并访问数据？ 74
Python:返回列表的第一个元素的索引,使得传递的函数为true 48
语义网络的概率生成 9
TypeError:必须是unicode,而不是NLTK中的str 9
NLTK:使用数字分数而不是标签进行文档分类 8
Spacy 优化清单？ 7
如何从 Spacy NER 模型中获得每个实体的预测概率？ 5
难疑归档
grep一个文件,但显示几个周围的行？ 3277
"git add -A"和"git add"之间的区别. 2788
Bower和npm有什么区别？ 1723
如何垂直对齐div内的图像？ 1371
如何获取所有Git分支 1371
如何使用自定义对象对NSMutableArray进行排序？ 1253
迭代集合,在循环中删除对象时避免使用ConcurrentModificationException 1158
命名类 - 如何避免将所有内容称为"<WhatEver> Manager"？ 1147
我怎样才能用Python代表'Enum'？ 1146
"静态"在C中意味着什么？ 1062