在 python Elasticsearch 中滚动不起作用

Question

当我查询 Elasticsearch 时，我尝试使用 python 滚动所有文档，以便获得超过 10K 个结果：

from elasticsearch import Elasticsearch
es = Elasticsearch(ADDRESS, port=PORT)


result = es.search(
    index="INDEX",
    body=es_query,
    size=10000,
    scroll="3m")


scroll_id = result['_scroll_id']
scroll_size = result["hits"]["total"]
counter = 0
print('total items= ' + scroll_size)

while(scroll_size > 0):
    counter +=len(result['hits']['hits'])
   

    result = es.scroll(scroll_id=scroll_id, scroll="1s")
    scroll_id = result['_scroll_id']

    
print('found = ' +counter)

问题是有时counter（程序结束时的结果总和）小于result["hits"]["total"]。这是为什么？为什么不scroll迭代所有结果？

ElasticSearch version : 5.6
lucence version :6.6

Answer 1

如果我没记错的话，您将在循环的第一次迭代中将首字母添加result["hits"]["total"]到您的中 - 但您应该只添加检索到的命中的长度：counterwhile

scroll_id = result['_scroll_id']
total = result["hits"]["total"]

print('total = %d' % total)

scroll_size = len(result["hits"]["hits"])  # this is the current 'page' size
counter = 0

while(scroll_size > 0):
    counter += scroll_size

    result = es.scroll(scroll_id=scroll_id, scroll="1s")
    scroll_id = result['_scroll_id']
    scroll_size = len(result['hits']['hits'])

print('counter = %d' % counter)
assert counter == total

事实上，您不需要单独存储滚动大小——更简洁的while循环是：

while len(result['hits']['hits']):
    counter += len(result['hits']['hits'])

    result = es.scroll(scroll_id=scroll_id, scroll="1s")
    scroll_id = result['_scroll_id']