MongoDB如何处理文本索引和文本分数中的文档长度？

Question

我有一个集合,其中包含大量不同文本的文档,并且看起来具有更多文本的文档会获得更高的textScores.当然,文档中的文本越多,关键字显示的次数就越多.然而,这并不一定意味着它与文本较少的文档相关或多或少相关.

有没有人知道MongoDB在计算相关性时如何考虑文档中文本的长度或数量？

我用Google搜索并搜索了MongoDB文档,但无法找到描述性答案.

Answer 1

评分基于词干匹配的数量,但也有一个内置系数,可调整匹配相对于总字段长度的分数(删除了停用词).如果较长的文本包含查询的更多相关字词,则会添加到分数中.与查询不匹配的较长文本会降低分数.

来自GitHub上的MongoDB 3.2源代码片段(src/mongo/db/fts/fts_spec.cpp):

   for (ScoreHelperMap::const_iterator i = terms.begin(); i != terms.end(); ++i) {
        const string& term = i->first;
        const ScoreHelperStruct& data = i->second;

        // in order to adjust weights as a function of term count as it
        // relates to total field length. ie. is this the only word or
        // a frequently occuring term? or does it only show up once in
        // a long block of text?

        double coeff = (0.5 * data.count / numTokens) + 0.5;

        // if term is identical to the raw form of the
        // field (untokenized) give it a small boost.
        double adjustment = 1;
        if (raw.size() == term.length() && raw.equalCaseInsensitive(term))
            adjustment += 0.1;

        double& score = (*docScores)[term];
        score += (weight * data.freq * coeff * adjustment);
        verify(score <= MAX_WEIGHT);
    }
}

设置一些测试数据以查看长度系数对一个非常简单的例子的影响:

db.articles.insert([
    { headline: "Rock" },
    { headline: "Rocks" },
    { headline: "Rock paper" },
    { headline: "Rock paper scissors" },
])

db.articles.createIndex({ "headline": "text"})

db.articles.find(
    { $text: { $search: "rock" }},
    { _id:0, headline:1, score: { $meta: "textScore" }}
).sort({ score: { $meta: "textScore" }})

注释结果:

// Exact match of raw term to indexed field
// Coefficent is 1, plus 0.1 bonus for identical match of raw term
{
  "headline": "Rock",
  "score": 1.1
}

// Match of stemmed term to indexed field ("rocks" stems to "rock")
// Coefficent is 1
{
  "headline": "Rocks",
  "score": 1
}

// Two terms, one matching
// Coefficient is 0.75: (0.5 * 1 match / 2 terms) + 0.5
{
  "headline": "Rock paper",
  "score": 0.75
}

// Three terms, one matching
// Coefficient is 0.66: (0.5 * 1 match / 3 terms) + 0.5
{
  "headline": "Rock paper scissors",
  "score": 0.6666666666666666
}