重复消除相似的公司名称

Question

我有一张包含公司名称的表格。由于人为输入错误，存在许多重复项。对于是否应包含细分、拼写错误等存在不同的看法。我希望所有这些重复项都被标记为一个公司“1c”：

+------------------+
|     company      |
+------------------+
| 1c               |
| 1c company       |
| 1c game studios  |
| 1c wireless      |
| 1c-avalon        |
| 1c-softclub      |
| 1c: maddox games |
| 1c:inoco         |
| 1cc games        |
+------------------+

我认为编辑距离是消除拼写错误的好方法。然而，当添加细分时，编辑距离会急剧增加，并且不再是一个好的算法。它是否正确？

总的来说，我在计算语言学方面几乎没有任何经验，所以我不知道应该选择什么方法。

对于这个问题你会推荐什么算法？我想用java实现它。纯 SQL 也可以。来源链接将不胜感激。谢谢。

Answer 1

这是一个难题。一个可能对您有帮助的神奇搜索关键字是“标准化” - 虽然有时它意味着非常不同的事情（例如“数据库标准化”是不相关的），但您实际上是在尝试标准化您的输入。

A simple solution is to use Levenshtein distance with token awareness. The Python library Fuzzy Wuzzy does this and this blog post introduces how it works with motivating examples. The basic idea is simple enough you should be able to implement it in Java without much difficulty.

At a high level, the idea is to split the input into tokens on whitespace and maybe punctuation, then sort the tokens and treat them as a set, then use the set intersection size - allowing for fuzzy matching - as a metric.

Some related links:

• Are there any good libraries available for doing normalization of company names? - Open Data Stack Exchange
• NEMO: Extraction and normalization of organization names from PubMed affiliation strings
• Automatic gazetteer enrichment with user-geocoded data - For place names, this basically creates a list of "true" names and then uses fuzzy lookup.
• Normalizing company names with SPARQL and DBpedia - bobdc.blog - Uses Wikipedia redirect information.