当涉及 OR 时，Postgres 选择过滤器而不是索引 cond

Question

我有一个表，每天都会添加大约 2000 万条记录，我试图对其进行分页，以便人们可以访问其中的所有数据，但查询时间必须“合适”（在我的例子中）定义为少于 30 秒/查询）。

为此，我过去一直使用键集分页，但对于这个特定的查询和表，我的查询时间非常慢，这似乎是因为查询规划器决定过滤掉一天的数据，然后对其运行过滤器而不是索引条件扫描。

该表如下所示：

create table mmsi_positions_archive
(
    id bigserial not null
        constraint mmsi_positions_archive_pkey
            primary key,
    position_id uuid,
    previous_id uuid,
    mmsi bigint not null,
    collection_type varchar not null,
    accuracy numeric,
    maneuver numeric,
    rate_of_turn numeric,
    status integer,
    speed numeric,
    course numeric,
    heading numeric,
    position geometry(Point,4326),
    timestamp timestamp with time zone not null,
    updated_at timestamp with time zone default now(),
    created_at timestamp with time zone default now()
);

create index ix_mmsi_positions_archive_mmsi
    on mmsi_positions_archive (mmsi);

create index ix_mmsi_positions_archive_position_id
    on mmsi_positions_archive (position_id);

create index ix_mmsi_positions_archive_timestamp_mmsi_id_asc
    on mmsi_positions_archive (timestamp, id);

我尝试分页的列是timestampand id，为了提供帮助，我还更新了表统计目标timestamp并将其设置为最大值 10 000 并分析了该表。

该表也按季度分区，但目前我只对单个分区的数据进行操作。

快速查询

SELECT id
FROM mmsi_positions_archive
WHERE timestamp > '2019-03-10 00:00:00.000000+00:00'
  AND timestamp <= '2019-03-11 00:00:00+00:00'
ORDER BY timestamp, id
LIMIT 100

其中给出了以下查询计划（注意mmsi_positions_archive表本身是空的，所有数据都在*_p2019_q1表中）：

Limit  (cost=0.60..5.39 rows=100 width=16) (actual time=0.053..0.089 rows=100 loops=1)
  ->  Merge Append  (cost=0.60..773572.19 rows=16149157 width=16) (actual time=0.053..0.082 rows=100 loops=1)
"        Sort Key: mmsi_positions_archive.""timestamp"", mmsi_positions_archive.id"
        ->  Sort  (cost=0.01..0.02 rows=1 width=16) (actual time=0.009..0.009 rows=0 loops=1)
"              Sort Key: mmsi_positions_archive.""timestamp"", mmsi_positions_archive.id"
              Sort Method: quicksort  Memory: 25kB
              ->  Seq Scan on mmsi_positions_archive  (cost=0.00..0.00 rows=1 width=16) (actual time=0.001..0.001 rows=0 loops=1)
                    Filter: (("timestamp" > '2019-03-10 00:00:00+00'::timestamp with time zone) AND ("timestamp" <= '2019-03-11 00:00:00+00'::timestamp with time zone))
        ->  Index Only Scan using mmsi_positions_archive_p2019q1_timestamp_id_index on mmsi_positions_archive_p2019q1  (cost=0.58..571707.70 rows=16149156 width=16) (actual time=0.043..0.067 rows=100 loops=1)
              Index Cond: (("timestamp" > '2019-03-10 00:00:00+00'::timestamp with time zone) AND ("timestamp" <= '2019-03-11 00:00:00+00'::timestamp with time zone))
              Heap Fetches: 0
Planning time: 67.023 ms
Execution time: 0.128 ms

键集分页查询（慢）

SELECT id
FROM mmsi_positions_archive
WHERE (timestamp > '2019-03-10 00:00:00.000000+00:00'
           OR (timestamp = '2019-03-10 00:00:00.000000+00:00' AND id >  1032749689))
  AND timestamp <= '2019-03-11 00:00:00+00:00'
ORDER BY timestamp, id
LIMIT 100

这给出了这个解释，最终运行速度要慢得多：

Limit  (cost=0.60..25.08 rows=100 width=16) (actual time=332918.152..332918.192 rows=100 loops=1)
  ->  Merge Append  (cost=0.60..41278140.09 rows=168591751 width=16) (actual time=332918.152..332918.189 rows=100 loops=1)
"        Sort Key: mmsi_positions_archive.""timestamp"", mmsi_positions_archive.id"
        ->  Sort  (cost=0.01..0.02 rows=1 width=16) (actual time=0.004..0.004 rows=0 loops=1)
"              Sort Key: mmsi_positions_archive.""timestamp"", mmsi_positions_archive.id"
              Sort Method: quicksort  Memory: 25kB
              ->  Seq Scan on mmsi_positions_archive  (cost=0.00..0.00 rows=1 width=16) (actual time=0.001..0.001 rows=0 loops=1)
                    Filter: (("timestamp" <= '2019-03-11 00:00:00+00'::timestamp with time zone) AND (("timestamp" > '2019-03-10 00:00:00+00'::timestamp with time zone) OR (("timestamp" = '2019-03-10 00:00:00+00'::timestamp with time zone) AND (id > 1032749689))))
        ->  Index Only Scan using mmsi_positions_archive_p2019q1_timestamp_id_index on mmsi_positions_archive_p2019q1  (cost=0.58..39170743.18 rows=168591750 width=16) (actual time=332918.147..332918.181 rows=100 loops=1)
              Index Cond: ("timestamp" <= '2019-03-11 00:00:00+00'::timestamp with time zone)
              Filter: (("timestamp" > '2019-03-10 00:00:00+00'::timestamp with time zone) OR (("timestamp" = '2019-03-10 00:00:00+00'::timestamp with time zone) AND (id > 1032749689)))
              Rows Removed by Filter: 953622052
              Heap Fetches: 0
Planning time: 0.778 ms
Execution time: 332918.226 ms

根据我的理解，这最终会变慢，因为索引条件Index Cond: ("timestamp" <= '2019-03-11 00:00:00+00'::timestamp with time zone最终会对大约 2000 万*70 行索引数据进行 seq 扫描，然后将它们过滤掉。

解决方法

我做了一些测试，发现问题出OR在声明中；如果我不这样做的话，他们俩都会给我一个快速的计划OR。所以我切换它并进行查询UNION以获取我想要的数据：

SELECT id
FROM (
         SELECT *
         FROM (
                  SELECT id        AS id,
                         timestamp AS timestamp
                  FROM mmsi_positions_archive
                  WHERE timestamp = '2019-03-10 00:00:00.000000+00:00'
                    AND id > 1032749689
                  ORDER BY timestamp, id
                  LIMIT 100
              ) keyset
         UNION
         SELECT *
         FROM (
                  SELECT id        AS id,
                         timestamp AS timestamp
                  FROM mmsi_positions_archive
                  WHERE timestamp > '2019-03-10 00:00:00.000000+00:00'
                    AND timestamp <= '2019-03-11 00:00:00+00:00'
                  ORDER BY timestamp, id
                  LIMIT 100
              ) all_after
     ) archive_ids
ORDER BY timestamp, id
LIMIT 100

产生快速查询和以下查询计划：

Limit  (cost=34.27..34.52 rows=100 width=16) (actual time=0.232..0.242 rows=100 loops=1)
  ->  Sort  (cost=34.27..34.77 rows=200 width=16) (actual time=0.231..0.238 rows=100 loops=1)
"        Sort Key: mmsi_positions_archive.""timestamp"", mmsi_positions_archive.id"
        Sort Method: quicksort  Memory: 34kB
        ->  HashAggregate  (cost=22.63..24.63 rows=200 width=16) (actual time=0.151..0.167 rows=200 loops=1)
"              Group Key: mmsi_positions_archive.id, mmsi_positions_archive.""timestamp"""
              ->  Append  (cost=0.71..21.63 rows=200 width=16) (actual time=0.028..0.111 rows=200 loops=1)
                    ->  Limit  (cost=0.71..12.24 rows=100 width=16) (actual time=0.028..0.049 rows=100 loops=1)
                          ->  Merge Append  (cost=0.71..17.43 rows=145 width=16) (actual time=0.027..0.046 rows=100 loops=1)
                                Sort Key: mmsi_positions_archive.id
                                ->  Index Scan using mmsi_positions_archive_pkey on mmsi_positions_archive  (cost=0.12..8.14 rows=1 width=16) (actual time=0.010..0.010 rows=0 loops=1)
                                      Index Cond: (id > 1032749689)
                                      Filter: ("timestamp" = '2019-03-10 00:00:00+00'::timestamp with time zone)
                                ->  Index Only Scan using mmsi_positions_archive_p2019q1_timestamp_id_index on mmsi_positions_archive_p2019q1  (cost=0.58..7.46 rows=144 width=16) (actual time=0.017..0.028 rows=100 loops=1)
                                      Index Cond: (("timestamp" = '2019-03-10 00:00:00+00'::timestamp with time zone) AND (id > 1032749689))
                                      Heap Fetches: 0
                    ->  Limit  (cost=0.60..5.39 rows=100 width=16) (actual time=0.012..0.049 rows=100 loops=1)
                          ->  Merge Append  (cost=0.60..773572.19 rows=16149157 width=16) (actual time=0.011..0.044 rows=100 loops=1)
"                                Sort Key: mmsi_positions_archive_1.""timestamp"", mmsi_positions_archive_1.id"
                                ->  Sort  (cost=0.01..0.02 rows=1 width=16) (actual time=0.005..0.005 rows=0 loops=1)
"                                      Sort Key: mmsi_positions_archive_1.""timestamp"", mmsi_positions_archive_1.id"
                                      Sort Method: quicksort  Memory: 25kB
                                      ->  Seq Scan on mmsi_positions_archive mmsi_positions_archive_1  (cost=0.00..0.00 rows=1 width=16) (actual time=0.001..0.001 rows=0 loops=1)
                                            Filter: (("timestamp" > '2019-03-10 00:00:00+00'::timestamp with time zone) AND ("timestamp" <= '2019-03-11 00:00:00+00'::timestamp with time zone))
                                ->  Index Only Scan using mmsi_positions_archive_p2019q1_timestamp_id_index on mmsi_positions_archive_p2019q1 mmsi_positions_archive_p2019q1_1  (cost=0.58..571707.70 rows=16149156 width=16) (actual time=0.006..0.031 rows=100 loops=1)
                                      Index Cond: (("timestamp" > '2019-03-10 00:00:00+00'::timestamp with time zone) AND ("timestamp" <= '2019-03-11 00:00:00+00'::timestamp with time zone))
                                      Heap Fetches: 0
Planning time: 1.059 ms
Execution time: 0.312 ms

虽然我可以重写我的查询以使用UNIONOR方法，但我确实想知道是否有某种方法可以更好地帮助 Postgres 通过使用?的“一目了然”更容易理解的查询获得相同的快速结果。

我也在 AWS Aurora Postgres 9.6 上运行它。我知道我们已经落后了几个主要版本，我计划尽快升级，但目前我只需要让这个东西正常工作。:)

Answer 1

幸运的是，这在 PostgreSQL 中非常简单，因为它支持可以利用索引的“行值”（或复合值）之间的比较。

所以你可以写：

WHERE (timestamp, id) > ('2019-03-10 00:00:00+00:00', 1032749689)
  AND timestamp <= '2019-03-11 00:00:00+00:00'
ORDER BY timestamp, id
LIMIT 100

此类行值的比较是按字典顺序进行的，这正是您想要的。

这是该功能的文档链接。