合并 NLP 中的相关词

Question

我想定义一个新词，其中包含来自两个（或更多）不同词的计数值。例如：

Words Frequency
0   mom 250
1   2020    151
2   the 124
3   19  82
4   mother  81
... ... ...
10  London  6
11  life    6
12  something   6

我想将母亲定义为mom + mother：

Words Frequency
0   mother  331
1   2020    151
2   the 124
3   19  82
... ... ...
9   London  6
10  life    6
11  something   6

这是一种替代定义具有某种含义的单词组的方法（至少对于我的目的而言）。

任何建议将不胜感激。

Answer 1

更新 10-21-2020

我决定构建一个 Python 模块来处理我在这个答案中概述的任务。该模块称为wordhoard，可以从pypi下载

我曾尝试在需要确定关键字（例如医疗保健）和关键字的同义词（例如，健康计划、预防医学）的频率的项目中使用 Word2vec 和 WordNet。我发现大多数 NLP 库都没有产生我需要的结果，因此我决定使用自定义关键字和同义词构建自己的字典。这种方法适用于多个项目中的文本分析和分类。

我确信精通 NLP 技术的人可能有更强大的解决方案，但下面的解决方案是类似的，它们一次又一次地为我工作。

我对答案进行了编码以匹配您在问题中的词频数据，但可以对其进行修改以使用任何关键字和同义词数据集。

import string

# Python Dictionary
# I manually created these word relationship - primary_word:synonyms
word_relationship = {"father": ['dad', 'daddy', 'old man', 'pa', 'pappy', 'papa', 'pop'],
          "mother": ["mamma", "momma", "mama", "mammy", "mummy", "mommy", "mom", "mum"]}

# This input text is from various poems about mothers and fathers
input_text = 'The hand that rocks the cradle also makes the house a home. It is the prayers of the mother ' \
         'that keeps the family strong. When I think about my mum, I just cannot help but smile; The beauty of ' \
         'her loving heart, the easy grace in her style. I will always need my mom, regardless of my age. She ' \
         'has made me laugh, made me cry. Her love will never fade. If I could write a story, It would be the ' \
         'greatest ever told. I would write about my daddy, For he had a heart of gold. For my father, my friend, ' \
         'This to me you have always been. Through the good times and the bad, Your understanding I have had.'

# converts the input text to lowercase and splits the words based on empty space.
wordlist = input_text.lower().split()

# remove all punctuation from the wordlist
remove_punctuation = [''.join(ch for ch in s if ch not in string.punctuation) 
for s in wordlist]

# list for word frequencies
wordfreq = []

# count the frequencies of a word
for w in remove_punctuation:
wordfreq.append(remove_punctuation.count(w))

word_frequencies = (dict(zip(remove_punctuation, wordfreq)))

word_matches = []

# loop through the dictionaries
for word, frequency in word_frequencies.items():
   for keyword, synonym in word_relationship.items():
      match = [x for x in synonym if word == x]
      if word == keyword or match:
        match = ' '.join(map(str, match))
        # append the keywords (mother), synonyms(mom) and frequencies to a list
        word_matches.append([keyword, match, frequency])

# used to hold the final keyword and frequencies
final_results = {}

# list comprehension to obtain the primary keyword and its frequencies
synonym_matches = [(keyword[0], keyword[2]) for keyword in word_matches]

# iterate synonym_matches and output total frequency count for a specific keyword
for item in synonym_matches:
  if item[0] not in final_results.keys():
    frequency_count = 0
    frequency_count = frequency_count + item[1]
    final_results[item[0]] = frequency_count
  else:
    frequency_count = frequency_count + item[1]
    final_results[item[0]] = frequency_count

 
print(final_results)
# output
{'mother': 3, 'father': 2}

其他方法

下面是一些其他方法及其开箱即用的输出。

NLTK字网

在这个例子中，我查找了“母亲”这个词的同义词。请注意，WordNet 没有与单词母亲相关联的同义词“妈妈”或“妈妈”。这两个词在我上面的示例文本中。另请注意，“父亲”一词被列为“母亲”的同义词。

from nltk.corpus import wordnet

synonyms = []
word = 'mother'
for synonym in wordnet.synsets(word):
   for item in synonym.lemmas():
      if word != synonym.name() and len(synonym.lemma_names()) > 1:
        synonyms.append(item.name())

print(synonyms)
['mother', 'female_parent', 'mother', 'fuss', 'overprotect', 'beget', 'get', 'engender', 'father', 'mother', 'sire', 'generate', 'bring_forth']

字典

在此示例中，我使用 PyDictionary 查找“母亲”一词的同义词，该 PyDictionary 查询同义词.com。此示例中的同义词包括单词“mom”和“mum”。此示例还包括 WordNet 未生成的其他同义词。

但是，PyDictionary 还生成了“妈妈”的同义词列表。这与“母亲”这个词无关。似乎 PyDictionary 从页面的形容词部分而不是名词部分中提取了此列表。计算机很难区分形容词妈妈和名词妈妈。

from PyDictionary import PyDictionary
dictionary_mother = PyDictionary('mother')

print(dictionary_mother.getSynonyms())
# output 
[{'mother': ['mother-in-law', 'female parent', 'supermom', 'mum', 'parent', 'mom', 'momma', 'para I', 'mama', 'mummy', 'quadripara', 'mommy', 'quintipara', 'ma', 'puerpera', 'surrogate mother', 'mater', 'primipara', 'mammy', 'mamma']}]

dictionary_mum = PyDictionary('mum')

print(dictionary_mum.getSynonyms())
# output 
[{'mum': ['incommunicative', 'silent', 'uncommunicative']}]

其他一些可能的方法是使用牛津词典 API 或查询 thesaurus.com。这两种方法也有缺陷。例如，牛津词典 API 需要一个 API 密钥和基于查询编号的付费订阅。而且 thesaurus.com 缺少可能对单词分组有用的潜在同义词。

https://www.thesaurus.com/browse/mother
synonyms: mom, parent, ancestor, creator, mommy, origin, predecessor, progenitor, source, child-bearer, forebearer, procreator

更新

为语料库中的每个潜在单词生成精确的同义词列表很困难，并且需要多方面的方法。下面的代码使用 WordNet 和 PyDictionary 创建同义词的超集。像所有其他答案一样，这种组合方法也会导致对词频的过度计算。我一直试图通过在我的最终同义词词典中组合键和值对来减少这种过度计数。后一个问题比我预期的要困难得多，可能需要我打开自己的问题来解决。最后，我认为根据您的用例，您需要确定哪种方法效果最好，并且可能需要结合多种方法。

感谢您发布这个问题，因为它让我可以查看解决复杂问题的其他方法。

from string import punctuation
from nltk.corpus import stopwords
from nltk.corpus import wordnet
from PyDictionary import PyDictionary

input_text = """The hand that rocks the cradle also makes the house a home. It is the prayers of the mother
         that keeps the family strong. When I think about my mum, I just cannot help but smile; The beauty of
         her loving heart, the easy grace in her style. I will always need my mom, regardless of my age. She
         has made me laugh, made me cry. Her love will never fade. If I could write a story, It would be the
         greatest ever told. I would write about my daddy, For he had a heart of gold. For my father, my friend,
         This to me you have always been. Through the good times and the bad, Your understanding I have had."""


def normalize_textual_information(text):
   # split text into tokens by white space
   token = text.split()

   # remove punctuation from each token
   table = str.maketrans('', '', punctuation)
   token = [word.translate(table) for word in token]

   # remove any tokens that are not alphabetic
   token = [word.lower() for word in token if word.isalpha()]

   # filter out English stop words
   stop_words = set(stopwords.words('english'))

   # you could add additional stops like this
   stop_words.add('cannot')
   stop_words.add('could')
   stop_words.add('would')

   token = [word for word in token if word not in stop_words]

   # filter out any short tokens
   token = [word for word in token if len(word) > 1]
   return token


def generate_word_frequencies(words):
   # list to hold word frequencies
   word_frequencies = []

   # loop through the tokens and generate a word count for each token
   for word in words:
      word_frequencies.append(words.count(word))

   # aggregates the words and word_frequencies into tuples and coverts them into a dictionary
   word_frequencies = (dict(zip(words, word_frequencies)))

   # sort the frequency of the words from low to high
   sorted_frequencies = {key: value for key, value in 
   sorted(word_frequencies.items(), key=lambda item: item[1])}

 return sorted_frequencies


def get_synonyms_internet(word):
   dictionary = PyDictionary(word)
   synonym = dictionary.getSynonyms()
   return synonym

 
words = normalize_textual_information(input_text)

all_synsets_1 = {}
for word in words:
  for synonym in wordnet.synsets(word):
    if word != synonym.name() and len(synonym.lemma_names()) > 1:
      for item in synonym.lemmas():
        if word != item.name():
          all_synsets_1.setdefault(word, []).append(str(item.name()).lower())

all_synsets_2 = {}
for word in words:
  word_synonyms = get_synonyms_internet(word)
  for synonym in word_synonyms:
    if word != synonym and synonym is not None:
      all_synsets_2.update(synonym)

 word_relationship = {**all_synsets_1, **all_synsets_2}

 frequencies = generate_word_frequencies(words)
 word_matches = []
 word_set = {}
 duplication_check = set()

 for word, frequency in frequencies.items():
    for keyword, synonym in word_relationship.items():
       match = [x for x in synonym if word == x]
       if word == keyword or match:
         match = ' '.join(map(str, match))
         if match not in word_set or match not in duplication_check or word not in duplication_check:
            duplication_check.add(word)
            duplication_check.add(match)
            word_matches.append([keyword, match, frequency])

 # used to hold the final keyword and frequencies
 final_results = {}

 # list comprehension to obtain the primary keyword and its frequencies
 synonym_matches = [(keyword[0], keyword[2]) for keyword in word_matches]

 # iterate synonym_matches and output total frequency count for a specific keyword
 for item in synonym_matches:
    if item[0] not in final_results.keys():
      frequency_count = 0
      frequency_count = frequency_count + item[1]
      final_results[item[0]] = frequency_count
 else:
    frequency_count = frequency_count + item[1]
    final_results[item[0]] = frequency_count

# do something with the final results