Why does this LEFT JOIN perform so much worse than LEFT JOIN LATERAL?

Question

Why does this LEFT JOIN perform so much worse than LEFT JOIN LATERAL?

Jel*_*rns 14 postgresql performance join execution-plan postgresql-10 postgresql-performance

I have the following tables (taken from the Sakila database):

film: film_id is pkey
actor: actor_id is pkey
film_actor: film_id and actor_id are fkeys to film/actor

I am selecting a particular film. For this film, I also want all actors participating in that film. I have two queries for this: one with a LEFT JOIN and one with a LEFT JOIN LATERAL.

select film.film_id, film.title, a.actors
from   film
left join
  (         
       select     film_actor.film_id, array_agg(first_name) as actors
       from       actor
       inner join film_actor using(actor_id)
       group by   film_actor.film_id
  ) as a
on       a.film_id = film.film_id
where    film.title = 'ACADEMY DINOSAUR'
order by film.title;

select film.film_id, film.title, a.actors
from   film
left join lateral
  (
       select     array_agg(first_name) as actors
       from       actor
       inner join film_actor using(actor_id)
       where      film_actor.film_id = film.film_id
  ) as a
on       true
where    film.title = 'ACADEMY DINOSAUR'
order by film.title;

Run Code Online (Sandbox Code Playgroud)

When comparing the query plan, the first query performs much worse (20x) than the second:

 Merge Left Join  (cost=507.20..573.11 rows=1 width=51) (actual time=15.087..15.089 rows=1 loops=1)
   Merge Cond: (film.film_id = film_actor.film_id)
   ->  Sort  (cost=8.30..8.31 rows=1 width=19) (actual time=0.075..0.075 rows=1 loops=1)
     Sort Key: film.film_id
     Sort Method: quicksort  Memory: 25kB
     ->  Index Scan using idx_title on film  (cost=0.28..8.29 rows=1 width=19) (actual time=0.044..0.058 rows=1 loops=1)
           Index Cond: ((title)::text = 'ACADEMY DINOSAUR'::text)
   ->  GroupAggregate  (cost=498.90..552.33 rows=997 width=34) (actual time=15.004..15.004 rows=1 loops=1)
     Group Key: film_actor.film_id
     ->  Sort  (cost=498.90..512.55 rows=5462 width=8) (actual time=14.934..14.937 rows=11 loops=1)
           Sort Key: film_actor.film_id
           Sort Method: quicksort  Memory: 449kB
           ->  Hash Join  (cost=6.50..159.84 rows=5462 width=8) (actual time=0.355..8.359 rows=5462 loops=1)
             Hash Cond: (film_actor.actor_id = actor.actor_id)
             ->  Seq Scan on film_actor  (cost=0.00..84.62 rows=5462 width=4) (actual time=0.035..2.205 rows=5462 loops=1)
             ->  Hash  (cost=4.00..4.00 rows=200 width=10) (actual time=0.303..0.303 rows=200 loops=1)
               Buckets: 1024  Batches: 1  Memory Usage: 17kB
               ->  Seq Scan on actor  (cost=0.00..4.00 rows=200 width=10) (actual time=0.027..0.143 rows=200 loops=1)
 Planning time: 1.495 ms
 Execution time: 15.426 ms

 Nested Loop Left Join  (cost=25.11..33.16 rows=1 width=51) (actual time=0.849..0.854 rows=1 loops=1)
   ->  Index Scan using idx_title on film  (cost=0.28..8.29 rows=1 width=19) (actual time=0.045..0.048 rows=1 loops=1)
     Index Cond: ((title)::text = 'ACADEMY DINOSAUR'::text)
   ->  Aggregate  (cost=24.84..24.85 rows=1 width=32) (actual time=0.797..0.797 rows=1 loops=1)
     ->  Hash Join  (cost=10.82..24.82 rows=5 width=6) (actual time=0.672..0.764 rows=10 loops=1)
           Hash Cond: (film_actor.actor_id = actor.actor_id)
           ->  Bitmap Heap Scan on film_actor  (cost=4.32..18.26 rows=5 width=2) (actual time=0.072..0.150 rows=10 loops=1)
             Recheck Cond: (film_id = film.film_id)
             Heap Blocks: exact=10
             ->  Bitmap Index Scan on idx_fk_film_id  (cost=0.00..4.32 rows=5 width=0) (actual time=0.041..0.041 rows=10 loops=1)
               Index Cond: (film_id = film.film_id)
           ->  Hash  (cost=4.00..4.00 rows=200 width=10) (actual time=0.561..0.561 rows=200 loops=1)
             Buckets: 1024  Batches: 1  Memory Usage: 17kB
             ->  Seq Scan on actor  (cost=0.00..4.00 rows=200 width=10) (actual time=0.039..0.275 rows=200 loops=1)
 Planning time: 1.722 ms
 Execution time: 1.087 ms

Run Code Online (Sandbox Code Playgroud)

Why is this? I want to learn to reason about this, so I can understand what is going on and can predict how the query will behave when data size increases and which decisions the planner will make under certain conditions.

My thoughts: in the first LEFT JOIN query, it looks like the subquery is executed for all films in the database, without taking into account the filtering in the outer query that we are only interested in one particular film. Why is the planner not able to have that knowledge in the subquery?

In the LEFT JOIN LATERAL query, we are more or less 'pushing' that filtering downwards. So the issue we had in the first query is not present here, hence the better performance.

I guess I am mainly looking for rule of thumbs, general wisdoms, ... so this planner magic becomes second nature - if that makes sense.

update (1)

Rewriting the LEFT JOIN as following also gives better performance (slightly better than the LEFT JOIN LATERAL):

select film.film_id, film.title, array_agg(a.first_name) as actors
from   film
left join
  (         
       select     film_actor.film_id, actor.first_name
       from       actor
       inner join film_actor using(actor_id)
  ) as a
on       a.film_id = film.film_id
where    film.title = 'ACADEMY DINOSAUR'
group by film.film_id
order by film.title;

 GroupAggregate  (cost=29.44..29.49 rows=1 width=51) (actual time=0.470..0.471 rows=1 loops=1)
   Group Key: film.film_id
   ->  Sort  (cost=29.44..29.45 rows=5 width=25) (actual time=0.428..0.430 rows=10 loops=1)
     Sort Key: film.film_id
     Sort Method: quicksort  Memory: 25kB
     ->  Nested Loop Left Join  (cost=4.74..29.38 rows=5 width=25) (actual time=0.149..0.386 rows=10 loops=1)
           ->  Index Scan using idx_title on film  (cost=0.28..8.29 rows=1 width=19) (actual time=0.056..0.057 rows=1 loops=1)
             Index Cond: ((title)::text = 'ACADEMY DINOSAUR'::text)
           ->  Nested Loop  (cost=4.47..19.09 rows=200 width=8) (actual time=0.087..0.316 rows=10 loops=1)
             ->  Bitmap Heap Scan on film_actor  (cost=4.32..18.26 rows=5 width=4) (actual time=0.052..0.089 rows=10 loops=1)
               Recheck Cond: (film_id = film.film_id)
               Heap Blocks: exact=10
               ->  Bitmap Index Scan on idx_fk_film_id  (cost=0.00..4.32 rows=5 width=0) (actual time=0.035..0.035 rows=10 loops=1)
                 Index Cond: (film_id = film.film_id)
             ->  Index Scan using actor_pkey on actor  (cost=0.14..0.17 rows=1 width=10) (actual time=0.011..0.011 rows=1 loops=10)
               Index Cond: (actor_id = film_actor.actor_id)
 Planning time: 1.833 ms
 Execution time: 0.706 ms

Run Code Online (Sandbox Code Playgroud)

How can we reason about this?

update (2)

I continued with some experiments and I think an interesting rule of thumb is: apply the aggregate function as high/late as possible. The query in update (1) probably performs better because we are aggregating in the outer query, no longer in the inner query.

The same seems to apply if we rewrite the LEFT JOIN LATERAL above as following:

select film.film_id, film.title, array_agg(a.first_name) as actors
from   film
left join lateral
  (
       select     actor.first_name
       from       actor
       inner join film_actor using(actor_id)
       where      film_actor.film_id = film.film_id
  ) as a
on       true
where    film.title = 'ACADEMY DINOSAUR'
group by film.film_id
order by film.title;

 GroupAggregate  (cost=29.44..29.49 rows=1 width=51) (actual time=0.088..0.088 rows=1 loops=1)
   Group Key: film.film_id
   ->  Sort  (cost=29.44..29.45 rows=5 width=25) (actual time=0.076..0.077 rows=10 loops=1)
     Sort Key: film.film_id
     Sort Method: quicksort  Memory: 25kB
     ->  Nested Loop Left Join  (cost=4.74..29.38 rows=5 width=25) (actual time=0.031..0.066 rows=10 loops=1)
           ->  Index Scan using idx_title on film  (cost=0.28..8.29 rows=1 width=19) (actual time=0.010..0.010 rows=1 loops=1)
             Index Cond: ((title)::text = 'ACADEMY DINOSAUR'::text)
           ->  Nested Loop  (cost=4.47..19.09 rows=200 width=8) (actual time=0.019..0.052 rows=10 loops=1)
             ->  Bitmap Heap Scan on film_actor  (cost=4.32..18.26 rows=5 width=4) (actual time=0.013..0.024 rows=10 loops=1)
               Recheck Cond: (film_id = film.film_id)
               Heap Blocks: exact=10
               ->  Bitmap Index Scan on idx_fk_film_id  (cost=0.00..4.32 rows=5 width=0) (actual time=0.007..0.007 rows=10 loops=1)
                 Index Cond: (film_id = film.film_id)
             ->  Index Scan using actor_pkey on actor  (cost=0.14..0.17 rows=1 width=10) (actual time=0.002..0.002 rows=1 loops=10)
               Index Cond: (actor_id = film_actor.actor_id)
 Planning time: 0.440 ms
 Execution time: 0.136 ms

Run Code Online (Sandbox Code Playgroud)

Here, we moved array_agg() upwards. As you can see, this plan is also better than the original LEFT JOIN LATERAL.

That said, I am not sure if this self-invented rule of thumb (apply the aggregate function as high/late as possible) is true in other cases.

additional information

Fiddle: https://dbfiddle.uk/?rdbms=postgres_10&fiddle=4ec4f2fffd969d9e4b949bb2ca765ffb

Version: PostgreSQL 10.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 6.4.0) 6.4.0, 64-bit

Environment: Docker: docker run -e POSTGRES_PASSWORD=sakila -p 5432:5432 -d frantiseks/postgres-sakila. Please note that the image on Docker hub is outdated, so I did a build locally first: build -t frantiseks/postgres-sakila after cloning the git repository.

Table definitions:

film

 film_id              | integer                     | not null default nextval('film_film_id_seq'::regclass)
 title                | character varying(255)      | not null

 Indexes:
    "film_pkey" PRIMARY KEY, btree (film_id)
    "idx_title" btree (title)

 Referenced by:
    TABLE "film_actor" CONSTRAINT "film_actor_film_id_fkey" FOREIGN KEY (film_id) REFERENCES film(film_id) ON UPDATE CASCADE ON DELETE RESTRICT

Run Code Online (Sandbox Code Playgroud)

actor

 actor_id    | integer                     | not null default nextval('actor_actor_id_seq'::regclass)
 first_name  | character varying(45)       | not null

 Indexes:
    "actor_pkey" PRIMARY KEY, btree (actor_id)

 Referenced by:
    TABLE "film_actor" CONSTRAINT "film_actor_actor_id_fkey" FOREIGN KEY (actor_id) REFERENCES actor(actor_id) ON UPDATE CASCADE ON DELETE RESTRICT

Run Code Online (Sandbox Code Playgroud)

film_actor

 actor_id    | smallint                    | not null
 film_id     | smallint                    | not null

 Indexes:
    "film_actor_pkey" PRIMARY KEY, btree (actor_id, film_id)
    "idx_fk_film_id" btree (film_id)
 Foreign-key constraints:
    "film_actor_actor_id_fkey" FOREIGN KEY (actor_id) REFERENCES actor(actor_id) ON UPDATE CASCADE ON DELETE RESTRICT
    "film_actor_film_id_fkey" FOREIGN KEY (film_id) REFERENCES film(film_id) ON UPDATE CASCADE ON DELETE RESTRICT

Run Code Online (Sandbox Code Playgroud)

Data: this is from the Sakila sample database. This question is not a real-life case, I am using this database mostly as a learning sample database. I have been introduced to SQL some months ago and I am trying to expand my knowledge. It has the following distributions:

select count(*) from film: 1000
select count(*) from actor: 200
select avg(a) from (select film_id, count(actor_id) a from film_actor group by film_id) a: 5.47

Run Code Online (Sandbox Code Playgroud)

Answer 1

Erw*_*ter 9

测试设置

您在小提琴中的原始设置留下了改进的空间。我一直要求你的设置是有原因的。

你有这些索引film_actor：
```
"film_actor_pkey" PRIMARY KEY, btree (actor_id, film_id)  
"idx_fk_film_id" btree (film_id)
```
Run Code Online (Sandbox Code Playgroud)
这已经很有帮助了。但为了最好地支持您的特定查询，您将按此顺序在, 列上设置多(film_id, actor_id)列索引。一个实用的解决方案：替换idx_fk_film_id为索引(film_id, actor_id)- 或(film_id, actor_id)为此测试的目的创建 PK ，就像我在下面做的那样。看：
- 复合索引是否也适用于第一个字段的查询？
在只读（或大多数情况下，或通常在 VACUUM 可以跟上写入活动时）中，它也有助于启用索引(title, film_id)以允许仅索引扫描。我的测试用例现在针对读取性能进行了高度优化。
film.film_id( integer) 和film_actor.film_id( smallint)之间的类型不匹配。虽然这有效，但它会使查询变慢，并可能导致各种复杂情况。也使 FK 约束更加昂贵。如果可以避免，请不要这样做。如果你不知道，挑integer了smallint。虽然每个字段smallint 可以节省 2 个字节（通常由对齐填充消耗），但比使用integer.

要优化测试本身的性能，请在批量插入大量行后创建索引和约束。将元组增量添加到现有索引比在所有行都存在的情况下从头开始创建它们要慢得多。

与本次测试无关：

独立序列加上列默认值，而不是更简单和更可靠的serial（或IDENTITY）列。别。

自增表列

timestamp without timestamp对于像last_update. 使用timestamptz来代替。请注意，严格来说，列默认值不包括“上次更新”。

长度修饰符 incharacter varying(255)表示测试用例不适合 Postgres，因为这里的奇数长度毫无意义。（或者作者无知。）

考虑小提琴中的审计测试用例：

db<>fiddle here - 在您的小提琴上构建，优化并添加查询。

有关的：

如何在 PostgreSQL 中实现多对多关系？

包含 1000 部电影和 200 位演员的测试设置的有效性有限。最有效的查询需要 < 0.2 毫秒。计划时间多于执行时间。具有 100k 或更多行的测试将更具启发性。

为什么只检索作者的名字？检索多个列后，情况就略有不同。

ORDER BY title过滤单个标题时没有意义WHERE title = 'ACADEMY DINOSAUR'。也许ORDER BY film_id？

对于总运行时间，而是使用EXPLAIN (ANALYZE, TIMING OFF)子定时开销来减少（可能误导）噪声。

回答

很难形成一个简单的经验法则，因为总体性能取决于许多因素。非常基本的准则：

聚合子表中的所有行会带来较少的开销，但仅在您实际需要所有行（或非常大的部分）时才需要付费。

对于选择几行（您的测试！），不同的查询技术会产生更好的结果。这就是LATERAL进来的地方。它带来了更多的开销，但只从子表中读取所需的行。如果只需要（非常）一小部分，则是大赢。

对于您的特定测试用例，我还将在LATERAL子查询中测试ARRAY 构造函数：

SELECT f.film_id, f.title, a.actors FROM film LEFT JOIN LATERAL ( SELECT ARRAY ( SELECT a.first_name FROM film_actor fa JOIN actor a USING (actor_id) WHERE fa.film_id = f.film_id ) AS actors ) a ON true WHERE f.title = 'ACADEMY DINOSAUR'; -- ORDER BY f.title; -- redundant while we filter for a single title
Run Code Online (Sandbox Code Playgroud)
虽然只在横向子查询中聚合单个数组，但简单的 ARRAY 构造函数比聚合函数执行得更好array_agg()。看：

为什么 array_agg() 比非聚合 ARRAY() 构造函数慢？

或者对于简单情况使用低相关子查询：

SELECT f.film_id, f.title , ARRAY (SELECT a.first_name FROM film_actor fa JOIN actor a USING (actor_id) WHERE fa.film_id = f.film_id) AS actors FROM film f WHERE f.title = 'ACADEMY DINOSAUR';
Run Code Online (Sandbox Code Playgroud)
或者，基本上，只是2xLEFT JOIN然后聚合：

SELECT f.film_id, f.title, array_agg(a.first_name) AS actors FROM film f LEFT JOIN film_actor fa USING (film_id) LEFT JOIN actor a USING (actor_id) WHERE f.title = 'ACADEMY DINOSAUR' GROUP BY f.film_id;
Run Code Online (Sandbox Code Playgroud)
这三个在我更新的小提琴中似乎最快（计划 + 执行时间）。

您的第一次尝试（仅稍作修改）通常可以最快地检索所有或大多数电影，但不适用于一小部分：

SELECT f.film_id, f.title, a.actors FROM film f LEFT JOIN ( SELECT fa.film_id, array_agg(first_name) AS actors FROM actor JOIN film_actor fa USING (actor_id) GROUP by fa.film_id ) a USING (film_id) WHERE f.title = 'ACADEMY DINOSAUR'; -- not good for a single (or few) films!
Run Code Online (Sandbox Code Playgroud)
具有更大基数的测试将更具启发性。并且不要轻易概括结果，总体性能有很多因素。

归档时间：	8 年，1 月前
查看次数：	4050 次
最近记录：	7 年，4 月前