如何测量句子之间的字符串相似度？

Question

我有以下任务.

给出的字符串列表如下:

        var strings = [
            'Steve jobs created the iPod when he was at Apple',
            'I really like the new Macbook by Apple',
            'Jony Ive was concerned being fired by Steve Jobs after his return to Apple',
            'The new Macbook has just one USB-C type connector',
            'I like bananas',
            'The brezels I can buy in my local store are much better than the ones in the supermarket',
            'the',
            'foo',
            'Steve'
        ];

我现在想要将每个字符串相互比较,并且对于每个比较,我想知道它们在0-1(或0%-100%)的范围内彼此之间的相似程度.

所以,我google了一下,发现了这个:Java中的相似性字符串比较

所以,我按照那里的指令,将方法移植similarity(String s1, String s2)到JavaScript:

        function similarity(s1, s2) {
            var longer = s1;
            var shorter = s2;
            if (s1.length < s2.length) {
                longer = s2;
                shorter = s1;
            }
            var longerLength = longer.length;
            if (longerLength == 0) {
                return 1.0;
            }
            return (longerLength - longer.LevenshteinDistance(shorter)) / longerLength;
        }

作为比较算法,我使用了Levenshtein:

        String.prototype.LevenshteinDistance = function (s2) {
            var array = new Array(this.length + 1);
            for (var i = 0; i < this.length + 1; i++)
                array[i] = new Array(s2.length + 1);

            for (var i = 0; i < this.length + 1; i++)
                array[i][0] = i;
            for (var j = 0; j < s2.length + 1; j++)
                array[0][j] = j;

            for (var i = 1; i < this.length + 1; i++) {
                for (var j = 1; j < s2.length + 1; j++) {
                    if (this[i - 1] == s2[j - 1]) array[i][j] = array[i - 1][j - 1];
                    else {
                        array[i][j] = Math.min(array[i][j - 1] + 1, array[i - 1][j] + 1);
                        array[i][j] = Math.min(array[i][j], array[i - 1][j - 1] + 1);
                    }
                }
            }
            return array[this.length][s2.length];
        };

因此,作为测试,我运行了一个完整的循环,将每个字符串相互比较并打印结果如下:

            for (var i in strings){
                var s = strings[i];
                print('Checking string: "' + s + '"');
                for (var j in strings){
                    print('-----');
                    var s2 = strings[j];
                    print('vs "' + s2 + '"');
                    var sim = similarity(s, s2);
                    print('Similarity: ' + Math.round(sim*100) + '%');
                }
                print('<br>////// NEXT /////////////////////////////////////////////////<br>');
            }

好的,现在结果如下:https://jsfiddle.net/wxksfa4w/

现在,看看结果我得到了一些很好的匹配,但也有一些完全无关,例如:

"史蒂夫·乔布斯在他上苹果时创造了iPod"和"我喜欢香蕉"的比赛为13%？

"史蒂夫·乔布斯在他上苹果时创造了iPod",只是"史蒂夫"匹配只有10%,尽管在第一句话中使用的是"史蒂夫"一词吗？

如何获得更好的语义结果？Levenshtein是错误的算法吗？根据我的理解,Levenshtein计算了如何将句子1改为句子2的步骤数.因此,即使存在语义相似性,字符串的长度似乎也会对结果产生重大影响.

有什么建议？

Answer 1

您可能应该使用两个句子中单词的存在作为高度相似性的暗示。一种简单的方法是将每个句子用作一个词袋并使用tf-idf