hib*_*ber 6 python nltk relation semantics knowledge-base-population
我最近正致力于使用nltk从文本中提取关系.所以我建立了一个示例文本:"汤姆是微软的联合创始人." 并使用以下程序测试并返回任何内容.我无法弄清楚为什么.
我使用的是NLTK版本:3.2.1,python版本:3.5.2.
这是我的代码:
import re
import nltk
from nltk.sem.relextract import extract_rels, rtuple
from nltk.tokenize import sent_tokenize, word_tokenize
def test():
with open('sample.txt', 'r') as f:
sample = f.read() # "Tom is the cofounder of Microsoft"
sentences = sent_tokenize(sample)
tokenized_sentences = [word_tokenize(sentence) for sentence in sentences]
tagged_sentences = [nltk.tag.pos_tag(sentence) for sentence in tokenized_sentences]
OF = re.compile(r'.*\bof\b.*')
for i, sent in enumerate(tagged_sentences):
sent = nltk.chunk.ne_chunk(sent) # ne_chunk method expects one tagged sentence
rels = extract_rels('PER', 'GPE', sent, corpus='ace', pattern=OF, window=10)
for rel in rels:
print('{0:<5}{1}'.format(i, rtuple(rel)))
if __name__ == '__main__':
test()
"盖茨于1955年10月28日出生在华盛顿州西雅图."
(S(PERSON Gates/NNS)//VBD出生/ VBN in/IN(GPE Seattle/NNP),/,(GPE Washington/NNP)/ IN 10月/ NNP 28/CD,/,1955/CD ./. )
[PER:'盖茨/ NNS']'/ VBD出生/ VBN in/IN'[GPE:'Seattle/NNP']
"盖茨于1955年10月28日出生在西雅图."
测试()没有任何回报.
输出是由函数引起的: semi_rel2reldict(pairs,window = 5,trace = False),仅当len(pairs)> 2时才返回结果,这就是为什么当一个少于三个NE的句子将返回None时.
这是一个错误还是我错误地使用了NLTK？
首先,对于大块的NE ne_chunk,这个成语看起来像这样
>>> from nltk import ne_chunk, pos_tag, word_tokenize
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> chunked
Tree('S', [Tree('PERSON', [('Tom', 'NNP')]), ('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN'), Tree('ORGANIZATION', [('Microsoft', 'NNP')])])
(另请参阅/sf/answers/2228686141/)
接下来让我们来看看这个extract_rels功能.
def extract_rels(subjclass, objclass, doc, corpus='ace', pattern=None, window=10):
"""
Filter the output of ``semi_rel2reldict`` according to specified NE classes and a filler pattern.
The parameters ``subjclass`` and ``objclass`` can be used to restrict the
Named Entities to particular types (any of 'LOCATION', 'ORGANIZATION',
'PERSON', 'DURATION', 'DATE', 'CARDINAL', 'PERCENT', 'MONEY', 'MEASURE').
"""
当你唤起这个功能时:
extract_rels('PER', 'GPE', sent, corpus='ace', pattern=OF, window=10)
它按顺序执行4个过程.
subjclass和objclass有效即https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L202:
if subjclass and subjclass not in NE_CLASSES[corpus]:
if _expand(subjclass) in NE_CLASSES[corpus]:
subjclass = _expand(subjclass)
else:
raise ValueError("your value for the subject type has not been recognized: %s" % subjclass)
if objclass and objclass not in NE_CLASSES[corpus]:
if _expand(objclass) in NE_CLASSES[corpus]:
objclass = _expand(objclass)
else:
raise ValueError("your value for the object type has not been recognized: %s" % objclass)
if corpus == 'ace' or corpus == 'conll2002':
pairs = tree2semi_rel(doc)
elif corpus == 'ieer':
pairs = tree2semi_rel(doc.text) + tree2semi_rel(doc.headline)
else:
raise ValueError("corpus type not recognized")
现在让我们看看给出你的输入句子Tom is the cofounder of Microsoft,tree2semi_rel()返回什么:
>>> from nltk.sem.relextract import tree2semi_rel, semi_rel2reldict
>>> from nltk import word_tokenize, pos_tag, ne_chunk
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
因此它返回一个包含2个列表的列表,第一个内部列表由空白列表组成,Tree其中包含"PERSON"标记.
[[], Tree('PERSON', [('Tom', 'NNP')])]
第二个列表由短语is the cofounder of和Tree包含"组织" 的短语组成.
让我们继续.
extract_rel然后尝试将对更改为某种关系字典reldicts = semi_rel2reldict(pairs)
如果我们查看semi_rel2reldict函数返回的示例句子,我们会看到这是空列表返回的位置:
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> semi_rel2reldict(tree2semi_rel(chunked))
[]
那么让我们看一下semi_rel2reldict https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L144的代码:
def semi_rel2reldict(pairs, window=5, trace=False):
"""
Converts the pairs generated by ``tree2semi_rel`` into a 'reldict': a dictionary which
stores information about the subject and object NEs plus the filler between them.
Additionally, a left and right context of length =< window are captured (within
a given input sentence).
:param pairs: a pair of list(str) and ``Tree``, as generated by
:param window: a threshold for the number of items to include in the left and right context
:type window: int
:return: 'relation' dictionaries whose keys are 'lcon', 'subjclass', 'subjtext', 'subjsym', 'filler', objclass', objtext', 'objsym' and 'rcon'
:rtype: list(defaultdict)
"""
result = []
while len(pairs) > 2:
reldict = defaultdict(str)
reldict['lcon'] = _join(pairs[0][0][-window:])
reldict['subjclass'] = pairs[0][1].label()
reldict['subjtext'] = _join(pairs[0][1].leaves())
reldict['subjsym'] = list2sym(pairs[0][1].leaves())
reldict['filler'] = _join(pairs[1][0])
reldict['untagged_filler'] = _join(pairs[1][0], untag=True)
reldict['objclass'] = pairs[1][1].label()
reldict['objtext'] = _join(pairs[1][1].leaves())
reldict['objsym'] = list2sym(pairs[1][1].leaves())
reldict['rcon'] = _join(pairs[2][0][:window])
if trace:
print("(%s(%s, %s)" % (reldict['untagged_filler'], reldict['subjclass'], reldict['objclass']))
result.append(reldict)
pairs = pairs[1:]
return result
首先要做的semi_rel2reldict()是检查输出中有多于2个元素的位置tree2semi_rel(),而你的例句不是:
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> len(tree2semi_rel(chunked))
2
>>> len(tree2semi_rel(chunked)) > 2
False
啊哈,这就是为什么extract_rel没有回来的原因.
现在问题是如何extract_rel()使用2个元素返回一些东西tree2semi_rel()？这甚至可能吗？
让我们尝试一个不同的句子:
>>> text = "Tom is the cofounder of Microsoft and now he is the founder of Marcohard"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> chunked
Tree('S', [Tree('PERSON', [('Tom', 'NNP')]), ('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN'), Tree('ORGANIZATION', [('Microsoft', 'NNP')]), ('and', 'CC'), ('now', 'RB'), ('he', 'PRP'), ('is', 'VBZ'), ('the', 'DT'), ('founder', 'NN'), ('of', 'IN'), Tree('PERSON', [('Marcohard', 'NNP')])])
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])], [[('and', 'CC'), ('now', 'RB'), ('he', 'PRP'), ('is', 'VBZ'), ('the', 'DT'), ('founder', 'NN'), ('of', 'IN')], Tree('PERSON', [('Marcohard', 'NNP')])]]
>>> len(tree2semi_rel(chunked)) > 2
True
>>> semi_rel2reldict(tree2semi_rel(chunked))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': 'and/CC now/RB he/PRP is/VBZ the/DT', 'subjtext': 'Tom/NNP'})]
但是这只能证实extract_rel当tree2semi_rel返回<2对时无法提取.如果我们删除那个条件while len(pairs) > 2会怎么样？
我们为什么不能这样做while len(pairs) > 1？
如果我们仔细研究代码,我们会看到最后一行填充reldict,https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L169:
reldict['rcon'] = _join(pairs[2][0][:window])
它试图访问第3个元素,pairs如果长度pairs为2,你会得到一个IndexError.
那么如果我们删除该rcon密钥并将其更改为什么会发生什么while len(pairs) >= 2？
要做到这一点,我们必须覆盖semi_rel2redict()函数:
>>> from nltk.sem.relextract import _join, list2sym
>>> from collections import defaultdict
>>> def semi_rel2reldict(pairs, window=5, trace=False):
... """
... Converts the pairs generated by ``tree2semi_rel`` into a 'reldict': a dictionary which
... stores information about the subject and object NEs plus the filler between them.
... Additionally, a left and right context of length =< window are captured (within
... a given input sentence).
... :param pairs: a pair of list(str) and ``Tree``, as generated by
... :param window: a threshold for the number of items to include in the left and right context
... :type window: int
... :return: 'relation' dictionaries whose keys are 'lcon', 'subjclass', 'subjtext', 'subjsym', 'filler', objclass', objtext', 'objsym' and 'rcon'
... :rtype: list(defaultdict)
... """
... result = []
... while len(pairs) >= 2:
... reldict = defaultdict(str)
... reldict['lcon'] = _join(pairs[0][0][-window:])
... reldict['subjclass'] = pairs[0][1].label()
... reldict['subjtext'] = _join(pairs[0][1].leaves())
... reldict['subjsym'] = list2sym(pairs[0][1].leaves())
... reldict['filler'] = _join(pairs[1][0])
... reldict['untagged_filler'] = _join(pairs[1][0], untag=True)
... reldict['objclass'] = pairs[1][1].label()
... reldict['objtext'] = _join(pairs[1][1].leaves())
... reldict['objsym'] = list2sym(pairs[1][1].leaves())
... reldict['rcon'] = []
... if trace:
... print("(%s(%s, %s)" % (reldict['untagged_filler'], reldict['subjclass'], reldict['objclass']))
... result.append(reldict)
... pairs = pairs[1:]
... return result
...
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> semi_rel2reldict(tree2semi_rel(chunked))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': [], 'subjtext': 'Tom/NNP'})]
啊! 它有效,但仍然是第四步extract_rels().
pattern参数提供的正则表达式执行reldict的过滤器,https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L222:relfilter = lambda x: (x['subjclass'] == subjclass and
len(x['filler'].split()) <= window and
pattern.match(x['filler']) and
x['objclass'] == objclass)
现在让我们用黑客版本来试试吧semi_rel2reldict:
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> semi_rel2reldict(tree2semi_rel(chunked))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': [], 'subjtext': 'Tom/NNP'})]
>>>
>>> pattern = re.compile(r'.*\bof\b.*')
>>> reldicts = semi_rel2reldict(tree2semi_rel(chunked))
>>> relfilter = lambda x: (x['subjclass'] == subjclass and
... len(x['filler'].split()) <= window and
... pattern.match(x['filler']) and
... x['objclass'] == objclass)
>>> relfilter
<function <lambda> at 0x112e591b8>
>>> subjclass = 'PERSON'
>>> objclass = 'ORGANIZATION'
>>> window = 5
>>> list(filter(relfilter, reldicts))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': [], 'subjtext': 'Tom/NNP'})]
有用!现在让我们以元组形式看到它:
>>> from nltk.sem.relextract import rtuple
>>> rels = list(filter(relfilter, reldicts))
>>> for rel in rels:
... print rtuple(rel)
...
[PER: 'Tom/NNP'] 'is/VBZ the/DT cofounder/NN of/IN' [ORG: 'Microsoft/NNP']