那些遇到过的问题

使用LIMIT的慢速Postgres查询

Zev*_*Zev 5 postgresql database-performance sql-limit

我遇到了一个与PostgreSQL查询类似的问题，但限制为1的速度非常慢，而带ORDER和LIMIT子句的PostgreSQL 速度非常慢，尽管在我的情况下，是否LIMIT为1、5或500 都没有关系。

基本上，当我无限制地运行由Django的ORM生成的查询时，该查询花费了半秒钟，但有限制（为分页而增加）则需要7秒钟。

耗时7秒的查询是：

SELECT "buildout_itemdescription"."product_code_id",
         MIN("buildout_lineitem"."unit_price") AS "min_price"
FROM "buildout_lineitem"
INNER JOIN "buildout_itemdescription"
    ON ("buildout_lineitem"."item_description_id" = "buildout_itemdescription"."id")
WHERE (("buildout_lineitem"."report_file_id" IN (154, 172, 155, 181, 174, 156, 157, 182, 175, 176, 183, 158, 177, 159, 179, 178, 164, 180, 367, 165, 173, 166, 167, 168, 368, 422, 370, 169, 1335, 1323, 161, 160, 162, 170, 171, 676, 151, 163, 980, 152, 369, 153, 963, 1718, 881, 617, 1759, 1780, 636, 1199, 1243, 947, 1163, 1422, 1009, 1407, 1035, 1241, 1077, 1271, 1111, 1130, 1489, 1507, 1555, 1600, 1619, 1663)
        AND "buildout_lineitem"."unit_price" > 0
        AND NOT ("buildout_itemdescription"."product_code_id" IS NULL)
        AND "buildout_lineitem"."date" >= '2014-04-20'::date
        AND "buildout_lineitem"."date" <= '2019-03-25'::date))
GROUP BY  "buildout_itemdescription"."product_code_id"
ORDER BY  "buildout_itemdescription"."product_code_id" LIMIT 5
Run Code Online (Sandbox Code Playgroud)

而另一个查询是相同的，没有LIMIT 5。

Postgres对两个查询使用的计划非常不同（HASH JOIN与NESTED LOOP）。

[
  {
    "Plan": {
      "Node Type": "Limit",
      "Parallel Aware": false,
      "Startup Cost": 0.86,
      "Total Cost": 1362.36,
      "Plan Rows": 5,
      "Plan Width": 36,
      "Actual Startup Time": 7035.543,
      "Actual Total Time": 7063.808,
      "Actual Rows": 5,
      "Actual Loops": 1,
      "Output": ["buildout_itemdescription.product_code_id", "(min(buildout_lineitem.unit_price))"],
      "Shared Hit Blocks": 1995053,
      "Shared Read Blocks": 1158777,
      "Shared Dirtied Blocks": 0,
      "Shared Written Blocks": 0,
      "Local Hit Blocks": 0,
      "Local Read Blocks": 0,
      "Local Dirtied Blocks": 0,
      "Local Written Blocks": 0,
      "Temp Read Blocks": 0,
      "Temp Written Blocks": 0,
      "Plans": [
        {
          "Node Type": "Aggregate",
          "Strategy": "Sorted",
          "Partial Mode": "Simple",
          "Parent Relationship": "Outer",
          "Parallel Aware": false,
          "Startup Cost": 0.86,
          "Total Cost": 35365525.74,
          "Plan Rows": 129877,
          "Plan Width": 36,
          "Actual Startup Time": 7035.541,
          "Actual Total Time": 7063.804,
          "Actual Rows": 5,
          "Actual Loops": 1,
          "Output": ["buildout_itemdescription.product_code_id", "min(buildout_lineitem.unit_price)"],
          "Group Key": ["buildout_itemdescription.product_code_id"],
          "Shared Hit Blocks": 1995053,
          "Shared Read Blocks": 1158777,
          "Shared Dirtied Blocks": 0,
          "Shared Written Blocks": 0,
          "Local Hit Blocks": 0,
          "Local Read Blocks": 0,
          "Local Dirtied Blocks": 0,
          "Local Written Blocks": 0,
          "Temp Read Blocks": 0,
          "Temp Written Blocks": 0,
          "Plans": [
            {
              "Node Type": "Nested Loop",
              "Parent Relationship": "Outer",
              "Parallel Aware": false,
              "Join Type": "Inner",
              "Startup Cost": 0.86,
              "Total Cost": 35362951.28,
              "Plan Rows": 255139,
              "Plan Width": 10,
              "Actual Startup Time": 7035.220,
              "Actual Total Time": 7062.420,
              "Actual Rows": 10660,
              "Actual Loops": 1,
              "Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
              "Inner Unique": false,
              "Shared Hit Blocks": 1995053,
              "Shared Read Blocks": 1158777,
              "Shared Dirtied Blocks": 0,
              "Shared Written Blocks": 0,
              "Local Hit Blocks": 0,
              "Local Read Blocks": 0,
              "Local Dirtied Blocks": 0,
              "Local Written Blocks": 0,
              "Temp Read Blocks": 0,
              "Temp Written Blocks": 0,
              "Plans": [
                {
                  "Node Type": "Index Scan",
                  "Parent Relationship": "Outer",
                  "Parallel Aware": false,
                  "Scan Direction": "Forward",
                  "Index Name": "buildout_itemdescription_product_code_id_084c51de",
                  "Relation Name": "buildout_itemdescription",
                  "Schema": "public",
                  "Alias": "buildout_itemdescription",
                  "Startup Cost": 0.43,
                  "Total Cost": 143970.67,
                  "Plan Rows": 1003733,
                  "Plan Width": 8,
                  "Actual Startup Time": 0.074,
                  "Actual Total Time": 627.419,
                  "Actual Rows": 351127,
                  "Actual Loops": 1,
                  "Output": ["buildout_itemdescription.id", "buildout_itemdescription.deleted", "buildout_itemdescription.created_on", "buildout_itemdescription.modified_on", "buildout_itemdescription.description", "buildout_itemdescription.category_1", "buildout_itemdescription.category_2", "buildout_itemdescription.category_3", "buildout_itemdescription.manufacturer_sku", "buildout_itemdescription.manufacturer_name", "buildout_itemdescription.distributor_sku", "buildout_itemdescription.supplier_id", "buildout_itemdescription.unit_of_measure", "buildout_itemdescription.quantity_in_unit_of_measure", "buildout_itemdescription.created_by_id", "buildout_itemdescription.modified_by_id", "buildout_itemdescription.product_code_id", "buildout_itemdescription.region_id"],
                  "Index Cond": "(buildout_itemdescription.product_code_id IS NOT NULL)",
                  "Rows Removed by Index Recheck": 0,
                  "Shared Hit Blocks": 195375,
                  "Shared Read Blocks": 144994,
                  "Shared Dirtied Blocks": 0,
                  "Shared Written Blocks": 0,
                  "Local Hit Blocks": 0,
                  "Local Read Blocks": 0,
                  "Local Dirtied Blocks": 0,
                  "Local Written Blocks": 0,
                  "Temp Read Blocks": 0,
                  "Temp Written Blocks": 0
                },
                {
                  "Node Type": "Index Scan",
                  "Parent Relationship": "Inner",
                  "Parallel Aware": false,
                  "Scan Direction": "Forward",
                  "Index Name": "buildout_lineitem_item_description_id_88254e09",
                  "Relation Name": "buildout_lineitem",
                  "Schema": "public",
                  "Alias": "buildout_lineitem",
                  "Startup Cost": 0.43,
                  "Total Cost": 35.00,
                  "Plan Rows": 9,
                  "Plan Width": 10,
                  "Actual Startup Time": 0.018,
                  "Actual Total Time": 0.018,
                  "Actual Rows": 0,
                  "Actual Loops": 351127,
                  "Output": ["buildout_lineitem.id", "buildout_lineitem.deleted", "buildout_lineitem.created_on", "buildout_lineitem.modified_on", "buildout_lineitem.date", "buildout_lineitem.month", "buildout_lineitem.quantity", "buildout_lineitem.unit_price", "buildout_lineitem.extended_sell", "buildout_lineitem.list_price", "buildout_lineitem.list_price_ext", "buildout_lineitem.agreed_price", "buildout_lineitem.agreed_price_ext", "buildout_lineitem.baseline_price", "buildout_lineitem.baseline_price_ext", "buildout_lineitem.item_class", "buildout_lineitem.user_created_by", "buildout_lineitem.sub_region", "buildout_lineitem.business_unit", "buildout_lineitem.created_by_id", "buildout_lineitem.item_description_id", "buildout_lineitem.modified_by_id", "buildout_lineitem.report_file_id"],
                  "Index Cond": "(buildout_lineitem.item_description_id = buildout_itemdescription.id)",
                  "Rows Removed by Index Recheck": 0,
                  "Filter": "((buildout_lineitem.unit_price > '0'::numeric) AND (buildout_lineitem.date >= '2014-04-20'::date) AND (buildout_lineitem.date <= '2019-03-25'::date) AND (buildout_lineitem.report_file_id = ANY ('{154,172,155,181,174,156,157,182,175,176,183,158,177,159,179,178,164,180,367,165,173,166,167,168,368,422,370,169,1335,1323,161,160,162,170,171,676,151,163,980,152,369,153,963,1718,881,617,1759,1780,636,1199,1243,947,1163,1422,1009,1407,1035,1241,1077,1271,1111,1130,1489,1507,1555,1600,1619,1663}'::integer[])))",
                  "Rows Removed by Filter": 10,
                  "Shared Hit Blocks": 1799678,
                  "Shared Read Blocks": 1013783,
                  "Shared Dirtied Blocks": 0,
                  "Shared Written Blocks": 0,
                  "Local Hit Blocks": 0,
                  "Local Read Blocks": 0,
                  "Local Dirtied Blocks": 0,
                  "Local Written Blocks": 0,
                  "Temp Read Blocks": 0,
                  "Temp Written Blocks": 0
                }
              ]
            }
          ]
        }
      ]
    },
    "Planning Time": 2.796,
    "Triggers": [
    ],
    "Execution Time": 7063.932
  }
]
Run Code Online (Sandbox Code Playgroud)

与

[
  {
    "Plan": {
      "Node Type": "Aggregate",
      "Strategy": "Sorted",
      "Partial Mode": "Simple",
      "Parallel Aware": false,
      "Startup Cost": 246569.04,
      "Total Cost": 249781.35,
      "Plan Rows": 129877,
      "Plan Width": 36,
      "Actual Startup Time": 561.755,
      "Actual Total Time": 580.878,
      "Actual Rows": 3771,
      "Actual Loops": 1,
      "Output": ["buildout_itemdescription.product_code_id", "min(buildout_lineitem.unit_price)"],
      "Group Key": ["buildout_itemdescription.product_code_id"],
      "Shared Hit Blocks": 5544,
      "Shared Read Blocks": 27043,
      "Shared Dirtied Blocks": 0,
      "Shared Written Blocks": 0,
      "Local Hit Blocks": 0,
      "Local Read Blocks": 0,
      "Local Dirtied Blocks": 0,
      "Local Written Blocks": 0,
      "Temp Read Blocks": 3729,
      "Temp Written Blocks": 3730,
      "Plans": [
        {
          "Node Type": "Sort",
          "Parent Relationship": "Outer",
          "Parallel Aware": false,
          "Startup Cost": 246569.04,
          "Total Cost": 247206.89,
          "Plan Rows": 255139,
          "Plan Width": 10,
          "Actual Startup Time": 561.741,
          "Actual Total Time": 568.372,
          "Actual Rows": 79253,
          "Actual Loops": 1,
          "Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
          "Sort Key": ["buildout_itemdescription.product_code_id"],
          "Sort Method": "external merge",
          "Sort Space Used": 1624,
          "Sort Space Type": "Disk",
          "Shared Hit Blocks": 5544,
          "Shared Read Blocks": 27043,
          "Shared Dirtied Blocks": 0,
          "Shared Written Blocks": 0,
          "Local Hit Blocks": 0,
          "Local Read Blocks": 0,
          "Local Dirtied Blocks": 0,
          "Local Written Blocks": 0,
          "Temp Read Blocks": 3729,
          "Temp Written Blocks": 3730,
          "Plans": [
            {
              "Node Type": "Hash Join",
              "Parent Relationship": "Outer",
              "Parallel Aware": false,
              "Join Type": "Inner",
              "Startup Cost": 59518.27,
              "Total Cost": 219295.38,
              "Plan Rows": 255139,
              "Plan Width": 10,
              "Actual Startup Time": 429.997,
              "Actual Total Time": 546.546,
              "Actual Rows": 79253,
              "Actual Loops": 1,
              "Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
              "Inner Unique": true,
              "Hash Cond": "(buildout_lineitem.item_description_id = buildout_itemdescription.id)",
              "Shared Hit Blocks": 5541,
              "Shared Read Blocks": 27043,
              "Shared Dirtied Blocks": 0,
              "Shared Written Blocks": 0,
              "Local Hit Blocks": 0,
              "Local Read Blocks": 0,
              "Local Dirtied Blocks": 0,
              "Local Written Blocks": 0,
              "Temp Read Blocks": 3526,
              "Temp Written Blocks": 3526,
              "Plans": [
                {
                  "Node Type": "Index Scan",
                  "Parent Relationship": "Outer",
                  "Parallel Aware": false,
                  "Scan Direction": "Forward",
                  "Index Name": "buildout_lineitem_report_file_id_b56906e1",
                  "Relation Name": "buildout_lineitem",
                  "Schema": "public",
                  "Alias": "buildout_lineitem",
                  "Startup Cost": 0.43,
                  "Total Cost": 151740.64,
                  "Plan Rows": 332154,
                  "Plan Width": 10,
                  "Actual Startup Time": 0.063,
                  "Actual Total Time": 30.922,
                  "Actual Rows": 79253,
                  "Actual Loops": 1,
                  "Output": ["buildout_lineitem.unit_price", "buildout_lineitem.item_description_id"],
                  "Index Cond": "(buildout_lineitem.report_file_id = ANY ('{154,172,155,181,174,156,157,182,175,176,183,158,177,159,179,178,164,180,367,165,173,166,167,168,368,422,370,169,1335,1323,161,160,162,170,171,676,151,163,980,152,369,153,963,1718,881,617,1759,1780,636,1199,1243,947,1163,1422,1009,1407,1035,1241,1077,1271,1111,1130,1489,1507,1555,1600,1619,1663}'::integer[]))",
                  "Rows Removed by Index Recheck": 0,
                  "Filter": "((buildout_lineitem.unit_price > '0'::numeric) AND (buildout_lineitem.date >= '2014-04-20'::date) AND (buildout_lineitem.date <= '2019-03-25'::date))",
                  "Rows Removed by Filter": 6,
                  "Shared Hit Blocks": 1610,
                  "Shared Read Blocks": 991,
                  "Shared Dirtied Blocks": 0,
                  "Shared Written Blocks": 0,
                  "Local Hit Blocks": 0,
                  "Local Read Blocks": 0,
                  "Local Dirtied Blocks": 0,
                  "Local Written Blocks": 0,
                  "Temp Read Blocks": 0,
                  "Temp Written Blocks": 0
                },
                {
                  "Node Type": "Hash",
                  "Parent Relationship": "Inner",
                  "Parallel Aware": false,
                  "Startup Cost": 43050.17,
                  "Total Cost": 43050.17,
                  "Plan Rows": 1003733,
                  "Plan Width": 8,
                  "Actual Startup Time": 429.373,
                  "Actual Total Time": 429.373,
                  "Actual Rows": 1005163,
                  "Actual Loops": 1,
                  "Output": ["buildout_itemdescription.product_code_id", "buildout_itemdescription.id"],
                  "Hash Buckets": 131072,
                  "Original Hash Buckets": 131072,
                  "Hash Batches": 16,
                  "Original Hash Batches": 16,
                  "Peak Memory Usage": 3495,
                  "Shared Hit Blocks": 3931,
                  "Shared Read Blocks": 26052,
                  "Shared Dirtied Blocks": 0,
                  "Shared Written Blocks": 0,
                  "Local Hit Blocks": 0,
                  "Local Read Blocks": 0,
                  "Local Dirtied Blocks": 0,
                  "Local Written Blocks": 0,
                  "Temp Read Blocks": 0,
                  "Temp Written Blocks": 3213,
                  "Plans": [
                    {
                      "Node Type": "Seq Scan",
                      "Parent Relationship": "Outer",
                      "Parallel Aware": false,
                      "Relation Name": "buildout_itemdescription",
                      "Schema": "public",
                      "Alias": "buildout_itemdescription",
                      "Startup Cost": 0.00,
                      "Total Cost": 43050.17,
                      "Plan Rows": 1003733,
                      "Plan Width": 8,
                      "Actual Startup Time": 0.021,
                      "Actual Total Time": 287.632,
                      "Actual Rows": 1005163,
                      "Actual Loops": 1,
                      "Output": ["buildout_itemdescription.product_code_id", "buildout_itemdescription.id"],
                      "Filter": "(buildout_itemdescription.product_code_id IS NOT NULL)",
                      "Rows Removed by Filter": 301554,
                      "Shared Hit Blocks": 3931,
                      "Shared Read Blocks": 26052,
                      "Shared Dirtied Blocks": 0,
                      "Shared Written Blocks": 0,
                      "Local Hit Blocks": 0,
                      "Local Read Blocks": 0,
                      "Local Dirtied Blocks": 0,
                      "Local Written Blocks": 0,
                      "Temp Read Blocks": 0,
                      "Temp Written Blocks": 0
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    "Planning Time": 2.965,
    "Triggers": [
    ],
    "Execution Time": 581.324
  }
]
Run Code Online (Sandbox Code Playgroud)

在Django中，我的代码是：

        from rest_framework.pagination import LimitOffsetPagination


        line_items = LineItem.objects.filter(
            report_file__report=report,
            unit_price__gt=0
        ).exclude(
            item_description__product_code__isnull=True
        )
Run Code Online (Sandbox Code Playgroud) 
        item_refs_aggregated = line_items.values(
            'item_description__product_code_id'
        ).annotate(
            min_price=Min('unit_price'),
        ).values(
            "item_description__product_code",
            "min_price",
        ).order_by(*ordering)


        paginator = LimitOffsetPagination()
        paginator.page_size = 10
        result_page = paginator.paginate_queryset(
            item_refs_aggregated,
            request
        )
Run Code Online (Sandbox Code Playgroud)

这些表上的索引是： buildout_lineitem： 

Indexes:
    "buildout_lineitem_pkey" PRIMARY KEY, btree (id)
    "buildout_lineitem_created_by_id_a61c52b7" btree (created_by_id)
    "buildout_lineitem_item_description_id_88254e09" btree (item_description_id)
    "buildout_lineitem_modified_by_id_8668530a" btree (modified_by_id)
    "buildout_lineitem_report_file_id_b56906e1" btree (report_file_id)
Foreign-key constraints:
    "buildout_lineitem_created_by_id_a61c52b7_fk_auth_user_id" FOREIGN KEY (created_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
    "buildout_lineitem_item_description_id_88254e09_fk_buildout_" FOREIGN KEY (item_description_id) REFERENCES buildout_itemdescription(id) DEFERRABLE INITIALLY DEFERRED
    "buildout_lineitem_modified_by_id_8668530a_fk_auth_user_id" FOREIGN KEY (modified_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
    "buildout_lineitem_report_file_id_b56906e1_fk_buildout_" FOREIGN KEY (report_file_id) REFERENCES buildout_reportfile(id) DEFERRABLE INITIALLY DEFERRED
Run Code Online (Sandbox Code Playgroud)

buildout_item说明： 

Indexes:
    "buildout_itemdescription

Lau*_*lbe 11

product_code_idPostgreSQL错误地认为通过按顺序扫描行直到找到满足所有条件的第一个行可以很快，但它不知道第一个匹配行并不接近开头。因此，它最终执行的嵌套循环连接比预期的要多。

删除该索引会加快查询速度，但如果这不是一个选项，请尝试使用

...
ORDER BY buildout_itemdescription.product_code_id + 0
LIMIT 5
Run Code Online (Sandbox Code Playgroud)

那么PostgreSQL就不能使用该索引了。

归档时间：

查看次数：

131 次

最近记录：

6 年，9 月 前
PostgreSQL查询非常慢,限制为1 38
使用ORDER和LIMIT子句进行极慢的PostgreSQL查询 33
更多相关链接
相关归档
Postgresql SQL GROUP BY时间间隔,任意精度(低至毫秒) 39
sql将字符串按空格分割成postgresql中的表 17
python将"E"添加到字符串 15
PostgreSQL无法创建任何TCP/IP套接字的小牛 14
pgadmin3服务器连接和身份验证错误 13
如何在 AWS Amazon Linux AMI 2 上安装 PostgeSQL 11？ 12
如何在查询中生成序列号？ 9
Postgres列不存在,但它有别名 8
PostGIS从POLYGON中提取坐标 8
如果函数不存在，如何创建函数？ 7
难疑归档
关闭/隐藏Android软键盘 3641
适用于Android UserManager.isUserAGoat()的用例？ 3506
从Git存储库中删除文件而不从本地文件系统中删除它 2892
如何从列表中随机选择一个项目？ 1656
JavaScript发布请求,如表单提交 1465
在Bash中循环遍历一串字符串？ 1371
jQuery document.createElement等价？ 1226
如何使用AngularJS在浏览器控制台中访问$ scope变量？ 1220
如何在不使用存储库的情况下将Docker镜像从一个主机复制到另一个主机 1181
为PHP密码保护哈希和盐 1142