Python Mongo"跳过高时,排序操作使用的数量超过最大值"

Question

我有一个代码,它使一个相当简单的query-skip-limit-sort.我遇到了一个我很难解释的现象.

关于"小"跳跃值 - 一切都很好.在"高"跳过值(> 18000) - 我没有得到超过20的限制的结果而没有得到以下错误:

OperationFailure: Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.

问题是 - 为什么只有大跳数才会发生这种情况？我怎么解决这个问题？

在mongoShell上运行它(即使使用DBQuery.shellBatchSize = 300)也可以.它似乎使用索引db.my_collection.find({'foo':false}).skip(19000).limit(100).sort({'meta_data.created_at': - 1}).explain()

"queryPlanner" : {
    "plannerVersion" : 1,
    "namespace" : "bla.my_collection",
    "indexFilterSet" : false,
    "parsedQuery" : {
        "foo" : {
            "$eq" : false
        }
    },
    "winningPlan" : {
        "stage" : "LIMIT",
        "limitAmount" : 100,
        "inputStage" : {
            "stage" : "SKIP",
            "skipAmount" : 9000,
            "inputStage" : {
                "stage" : "FETCH",
                "inputStage" : {
                    "stage" : "IXSCAN",
                    "keyPattern" : {
                        "foo" : 1,
                        "meta_data.created_at" : -1
                    },
                    "indexName" : "foo_1_meta_data.created_at_-1",
                    "isMultiKey" : false,
                    "multiKeyPaths" : {
                        "foo" : [ ],
                        "meta_data.created_at" : [ ]
                    },
                    "isUnique" : false,
                    "isSparse" : false,
                    "isPartial" : false,
                    "indexVersion" : 1,
                    "direction" : "forward",
                    "indexBounds" : {
                        "foo" : [
                            "[false, false]"
                        ],
                        "meta_data.created_at" : [
                            "[MaxKey, MinKey]"
                        ]
                    }
                }
            }
        }
    },

}

更多信息: 似乎确实排序是在内存中完成的 - 它存在于被拒绝的计划中.那可以做些什么呢？

"rejectedPlans" : [
            {
                "stage" : "SKIP",
                "skipAmount" : 19000,
                "inputStage" : {
                    "stage" : "SORT",
                    "sortPattern" : {
                        "meta_data.created_at" : -1
                    },
                    "limitAmount" : 19100,
                    "inputStage" : {
                        "stage" : "SORT_KEY_GENERATOR",
                        "inputStage" : {
                            "stage" : "FETCH",
                            "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                    "foo" : 1,
                                    "_id" : 1
                                },
                                "indexName" : "foo_1__id_1",
                                "isMultiKey" : false,
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 1,
                                "direction" : "forward",
                                "indexBounds" : {
                                    "foo" : [
                                        "[false, false]"
                                    ],
                                    "_id" : [
                                        "[MinKey, MaxKey]"
                                    ]
                                }
                            }
                        }
                    }
                }
            }

另一个问题.为什么只在大跳数时发生？为什么这有关系？