我正在尝试创建一个以这种方式工作的分词器:
POST dev_threats/_analyze
{
"tokenizer": "my_tokenizer",
"text": "some.test.domain.com"
}
Run Code Online (Sandbox Code Playgroud)
并获得如下令牌:
[some, some.test, some.test.domain, some.test.domain.com, test, test.domain, test.domain.com, domain, domain.com]
Run Code Online (Sandbox Code Playgroud)
我尝试了 ngram 分词器:
"ngram_domain_tokenizer": {
"type": "ngram",
"min_gram": 1,
"max_gram": 63,
"token_chars": [
"letter",
"digit",
"punctuation"
]
},
Run Code Online (Sandbox Code Playgroud)
但对于长值,它会生成太多标记......
知道如何得到这样的结果吗?