换行符或标点符号作为弹性搜索中的位置间隙

Question

在弹性搜索中,有没有办法设置一个分析器,当遇到换行符或标点符号时会在标记之间产生位置间隙？

假设我使用以下无意义字符串(带换行符)作为其字段之一来索引对象:

The quick brown fox runs after the rabbit.
Then comes the jumpy frog.

标准分析仪将生成以下具有相应位置的标记:

0 the
1 quick
2 brown
3 fox
4 runs
5 after
6 the
7 rabbit
8 then
9 comes
10 the
11 jumpy
12 frog

这意味着match_phrase查询the rabbit then comes将匹配此文档作为匹配.有没有办法引进之间的位置差距rabbit和then使,除非它不匹配slop的出台？

当然,一种解决方法可能是将单个字符串转换为数组(每个条目一行)并position_offset_gap在字段映射中使用,但我真的宁愿用换行符保留一个字符串(并且最终的解决方案将涉及换行符的更大位置间隙比如标点符号).

Answer 1

我最终想出了一个解决方案,使用a char_filter在换行符和标点符号上引入额外的令牌:

PUT /index
{                                              
  "settings": {
    "analysis": {
      "char_filter": {
        "my_mapping": {
          "type": "mapping",
          "mappings": [ ".=>\\n_PERIOD_\\n", "\\n=>\\n_NEWLINE_\\n" ]
        }
      },
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "standard",
          "char_filter": ["my_mapping"],
          "filter": ["lowercase"]
        }
      }
    }
  }
}

使用示例字符串进行测试

POST /index/_analyze?analyzer=my_analyzer&pretty
The quick brown fox runs after the rabbit.
Then comes the jumpy frog.

得到以下结果:

{
  "tokens" : [ {
    "token" : "the",
    "start_offset" : 0,
    "end_offset" : 3,
    "type" : "<ALPHANUM>",
    "position" : 1
  }, {
... snip ...
    "token" : "rabbit",
    "start_offset" : 35,
    "end_offset" : 41,
    "type" : "<ALPHANUM>",
    "position" : 8
  }, {
    "token" : "_period_",
    "start_offset" : 41,
    "end_offset" : 41,
    "type" : "<ALPHANUM>",
    "position" : 9
  }, {
    "token" : "_newline_",
    "start_offset" : 42,
    "end_offset" : 42,
    "type" : "<ALPHANUM>",
    "position" : 10
  }, {
    "token" : "then",
    "start_offset" : 43,
    "end_offset" : 47,
    "type" : "<ALPHANUM>",
    "position" : 11
... snip ...
  } ]
}