Control number of buckets created in an aggregation

Question

In Elasticsearch there's a limit on how many buckets you can create in an aggregation. If it creates more buckets than the specified limit, you will get a warning message In ES 6.x and an error will be thrown in future versions.

Here's the warning message:

This aggregation creates too many buckets (10001) and will throw an error in future versions. You should update the [search.max_buckets] cluster setting or use the [composite] aggregation to paginate all buckets in multiple requests.

Since ES 7.x, that limit is set to 10000 which can be adjusted, though.

The problem is, I can't actually calculate (or estimate) how many buckets an aggregation is going to create.

Consider the following request:

GET /zone_stats_hourly/_search
{
   "aggs":{
      "apps":{
         "terms":{
            "field":"appId",
            "size":<NUM_TERM_BUCKETS>,
            "min_doc_count":1,
            "shard_min_doc_count":0,
            "show_term_doc_count_error":false,
            "order":[
               {
                  "_count":"desc"
               },
               {
                  "_key":"asc"
               }
            ]
         },
         "aggregations":{
            "histogram":{
               "days":{
                  "field":"processTime",
                  "time_zone":"UTC",
                  "interval":"1d",
                  "offset":0,
                  "order":{
                     "_key":"asc"
                  },
                  "keyed":false,
                  "min_doc_count":0
               },
               "aggregations":{
                  "requests":{
                     "sum":{
                        "field":"requests"
                     }
                  },
                  "filled":{
                     "sum":{
                        "field":"filledRequests"
                     }
                  },
                  "matched":{
                     "sum":{
                        "field":"matchedRequests"
                     }
                  },
                  "imp":{
                     "sum":{
                        "field":"impressions"
                     }
                  },
                  "cv":{
                     "sum":{
                        "field":"completeViews"
                     }
                  },
                  "clicks":{
                     "sum":{
                        "field":"clicks"
                     }
                  },
                  "installs":{
                     "sum":{
                        "field":"installs"
                     }
                  },
                  "actions":{
                     "sum":{
                        "field":"actions"
                     }
                  },
                  "earningsIRT":{
                     "sum":{
                        "field":"earnings.inIRT"
                     }
                  },
                  "earningsUSD":{
                     "sum":{
                        "field":"earnings.inUSD"
                     }
                  },
                  "earningsEUR":{
                     "sum":{
                        "field":"earnings.inEUR"
                     }
                  },
                  "dealBasedEarnings":{
                     "nested":{
                        "path":"dealBasedEarnings"
                     },
                     "aggregations":{
                        "types":{
                           "terms":{
                              "field":"dealBasedEarnings.type",
                              "size":4,
                              "min_doc_count":1,
                              "shard_min_doc_count":0,
                              "show_term_doc_count_error":false,
                              "order":[
                                 {
                                    "_count":"desc"
                                 },
                                 {
                                    "_key":"asc"
                                 }
                              ]
                           },
                           "aggregations":{
                              "dealBasedEarningsIRT":{
                                 "sum":{
                                    "field":"dealBasedEarnings.amount.inIRT"
                                 }
                              },
                              "dealBasedEarningsUSD":{
                                 "sum":{
                                    "field":"dealBasedEarnings.amount.inUSD"
                                 }
                              },
                              "dealBasedEarningsEUR":{
                                 "sum":{
                                    "field":"dealBasedEarnings.amount.inEUR"
                                 }
                              }
                           }
                        }
                     }
                  }
               }
            }
         }
      }
   },
   "size":0,
   "_source":{
      "excludes":[]
   },
   "stored_fields":["*"],
   "docvalue_fields":[
      {
         "field":"eventTime",
         "format":"date_time"
      },
      {
         "field":"processTime",
         "format":"date_time"
      },
      {
         "field":"postBack.time",
         "format":"date_time"
      }
   ],
   "query":{
      "bool":{
         "must":[
            {
               "range":{
                  "processTime":{
                     "from":1565049600000,
                     "to":1565136000000,
                     "include_lower":true,
                     "include_upper":false,
                     "boost":1.0
                  }
               }
            }
         ],
         "adjust_pure_negative":true,
         "boost":1.0
      }
   }
}

If I set <NUM_TERM_BUCKETS> to 2200 and perform the request, I get the warning message that says I'm creating more than 10000 buckets (how?!).

A sample response from ES:

#! Deprecation: 299 Elasticsearch-6.7.1-2f32220 "This aggregation creates too many buckets (10001) and will throw an error in future versions. You should update the [search.max_buckets] cluster setting or use the [composite] aggregation to paginate all buckets in multiple requests."
{
  "took": 6533,
  "timed_out": false,
  "_shards": {
    "total": 3,
    "successful": 3,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 103456,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "apps": {
      "doc_count_error_upper_bound": 9,
      "sum_other_doc_count": 37395,
      "buckets":[...]
    }
  }
}

More interestingly, after decreasing <NUM_TERM_BUCKETS> to 2100, I get no warning messages, which means the number of buckets created is below 10000.

I've had a hard time to find the reason behind that and found NOTHING.

Is there any formula or something to calculate or estimate the number of buckets that an aggregation is going to create before actually performing the request?

I want to know if an aggregation throws error in ES 7.x or later regarding to a specified search.max_buckets, so that I can decide whether to use the composite aggregation or not.

UPDATE

I tried a much simpler aggregation containing no nested or sub aggregations on an index having roughly 80000 documents.

Here is the request:

GET /my_index/_search
{
   "size":0,
   "query":{
      "match_all":{}
   },
   "aggregations":{
      "unique":{
         "terms":{
            "field":"_id",
            "size":<NUM_TERM_BUCKETS>
         }
      }
   }
}

If I set the <NUM_TERM_BUCKETS> to 7000, I get this error response in ES 7.3:

{
   "error":{
      "root_cause":[
         {
            "type":"too_many_buckets_exception",
            "reason":"Trying to create too many buckets. Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max_buckets] cluster level setting.",
            "max_buckets":10000
         }
      ],
      "type":"search_phase_execution_exception",
      "reason":"all shards failed",
      "phase":"query",
      "grouped":true,
      "failed_shards":[
         {
            "shard":0,
            "index":"my_index",
            "node":"XYZ",
            "reason":{
               "type":"too_many_buckets_exception",
               "reason":"Trying to create too many buckets. Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max_buckets] cluster level setting.",
               "max_buckets":10000
            }
         }
      ]
   },
   "status":503
}

And it runs successfully if I decrease the <NUM_TERM_BUCKETS> to 6000.

Seriously, I'm confused. how on earth this aggregation creates more than 10000 buckets? Can anyone answer this?

Answer 1

根据条款聚合的文档：

该shard_size参数可用于最小化更大的 requests 带来的额外工作size。定义后，它将确定协调节点将从每个分片请求多少项。一旦所有的分片都响应了，协调节点就会将它们归结为基于size参数的最终结果——这样，可以提高返回术语的准确性并避免将大量存储桶流回的开销客户端。

默认shard_size值为(size * 1.5 + 10).

为了解决分布式系统中的准确性问题，Elasticsearch要求一个高于size每个分片的数字。

因此，NUM_TERM_BUCKETS可以使用以下公式计算简单术语聚合的最大值：

maxNumTermBuckets = (search.maxBuckets - 10) / 1.5

这是6660为了search.maxBuckets = 10000.