PostgreSQL 错误:内存不足

Cer*_*rin 11 postgresql

我正在尝试运行一个应该返回大约 2000 行的查询,但是我的 RDS 托管的 PostgreSQL 9.3 数据库给了我错误“内存不足详细信息:请求大小为 2048 时失败。”。

这意味着什么?我的实例有 3GB 的内存,那么什么会限制它足以用这么小的查询耗尽内存?

编辑:

SHOW work_mem;
"1024GB"
Run Code Online (Sandbox Code Playgroud)

我无法显示完整的 SQL,但它正在尝试执行数据透视。我有两个主表librarybook,它们指向一条library记录。我的查询尝试为每个图书馆记录查找过去 12 个月中每个月最受欢迎的书,并将它们连接到结果查询集中的一个单独列中,例如:

library_id, month_1_book_id, month_2_book_id, month_3_book_id, ...
Run Code Online (Sandbox Code Playgroud)

Explain 显示了很多循环的结果:

explain
select * from myapp_library_get_monthly_popular
where id in (5495060, 5495059, 5495048)

Nested Loop Left Join  (cost=3645798.54..3750412.91 rows=3 width=2980)
  ->  Nested Loop Left Join  (cost=3645798.10..3750388.98 rows=3 width=2994)
        ->  Nested Loop Left Join  (cost=3645797.66..3750365.05 rows=3 width=2976)
              ->  Nested Loop Left Join  (cost=3645797.23..3750341.13 rows=3 width=2958)
                    ->  Nested Loop Left Join  (cost=3645796.79..3750317.20 rows=3 width=2940)
                          ->  Nested Loop Left Join  (cost=3645796.35..3750293.27 rows=3 width=2922)
                                ->  Nested Loop Left Join  (cost=3645795.91..3750269.35 rows=3 width=2904)
                                      ->  Nested Loop Left Join  (cost=3645795.48..3750245.42 rows=3 width=2886)
                                            ->  Nested Loop Left Join  (cost=3645795.04..3750221.49 rows=3 width=2868)
                                                  ->  Nested Loop Left Join  (cost=3645794.60..3750197.57 rows=3 width=2850)
                                                        ->  Nested Loop Left Join  (cost=3645794.16..3750173.64 rows=3 width=2832)
                                                              ->  Nested Loop Left Join  (cost=3645793.73..3750149.71 rows=3 width=2814)
                                                                    ->  Hash Join  (cost=3645793.29..3750125.79 rows=3 width=2796)
                                                                          Hash Cond: (c.category_id = ct.id)
                                                                          ->  Hash Join  (cost=3645792.24..3750124.70 rows=3 width=2578)
                                                                                Hash Cond: (c.company_id = cp.id)
                                                                                ->  Hash Join  (cost=3645791.18..3750123.59 rows=3 width=2360)
                                                                                      Hash Cond: (c_1.id = c.id)
                                                                                      ->  HashAggregate  (cost=3645772.11..3695454.20 rows=3974567 width=8)
                                                                                            ->  Hash Right Join  (cost=2178696.20..3595882.33 rows=9977957 width=8)
                                                                                                  Hash Cond: (u.library_id = c_1.id)
                                                                                                  Join Filter: (u.period_start_date >= ((max(u_1.period_start_date)) - '1 year'::interval))
                                                                                                  ->  Seq Scan on myapp_book u  (cost=0.00..594004.70 rows=29933870 width=12)
                                                                                                  ->  Hash  (cost=2129014.12..2129014.12 rows=3974567 width=8)
                                                                                                        ->  Hash Join  (cost=1960095.02..2129014.12 rows=3974567 width=8)
                                                                                                              Hash Cond: (c_2.id = c_1.id)
                                                                                                              ->  HashAggregate  (cost=1780709.26..1820454.93 rows=3974567 width=8)
                                                                                                                    ->  Hash Right Join  (cost=288688.46..1556205.24 rows=29933870 width=8)
                                                                                                                          Hash Cond: (u_1.library_id = c_2.id)
                                                                                                                          ->  Seq Scan on myapp_book u_1  (cost=0.00..594004.70 rows=29933870 width=8)
                                                                                                                          ->  Hash  (cost=239006.38..239006.38 rows=3974567 width=4)
                                                                                                                                ->  Hash Join  (cost=2.11..239006.38 rows=3974567 width=4)
                                                                                                                                      Hash Cond: (c_2.category_id = ct_1.id)
                                                                                                                                      ->  Hash Join  (cost=1.07..184355.03 rows=3974567 width=8)
                                                                                                                                            Hash Cond: (c_2.company_id = cp_1.id)
                                                                                                                                            ->  Seq Scan on myapp_library c_2  (cost=0.00..129703.67 rows=3974567 width=12)
                                                                                                                                            ->  Hash  (cost=1.03..1.03 rows=3 width=4)
                                                                                                                                                  ->  Seq Scan on myapp_company cp_1  (cost=0.00..1.03 rows=3 width=4)
                                                                                                                                      ->  Hash  (cost=1.02..1.02 rows=2 width=4)
                                                                                                                                            ->  Seq Scan on myapp_category ct_1  (cost=0.00..1.02 rows=2 width=4)
                                                                                                              ->  Hash  (cost=129703.67..129703.67 rows=3974567 width=4)
                                                                                                                    ->  Seq Scan on myapp_library c_1  (cost=0.00..129703.67 rows=3974567 width=4)
                                                                                      ->  Hash  (cost=19.02..19.02 rows=3 width=2328)
                                                                                            ->  Index Scan using myapp_library_pkey on myapp_library c  (cost=0.43..19.02 rows=3 width=2328)
                                                                                                  Index Cond: (id = ANY ('{5495060,5495059,5495048}'::integer[]))
                                                                                ->  Hash  (cost=1.03..1.03 rows=3 width=222)
                                                                                      ->  Seq Scan on myapp_company cp  (cost=0.00..1.03 rows=3 width=222)
                                                                          ->  Hash  (cost=1.02..1.02 rows=2 width=222)
                                                                                ->  Seq Scan on myapp_category ct  (cost=0.00..1.02 rows=2 width=222)
                                                                    ->  Index Scan using myapp_book_pkey on myapp_book u_01  (cost=0.44..7.97 rows=1 width=22)
                                                                          Index Cond: (id = ((array_agg(u.id)))[1])
                                                              ->  Index Scan using myapp_book_pkey on myapp_book u_02  (cost=0.44..7.97 rows=1 width=22)
                                                                    Index Cond: (id = ((array_agg(u.id)))[2])
                                                        ->  Index Scan using myapp_book_pkey on myapp_book u_03  (cost=0.44..7.97 rows=1 width=22)
                                                              Index Cond: (id = ((array_agg(u.id)))[3])
                                                  ->  Index Scan using myapp_book_pkey on myapp_book u_04  (cost=0.44..7.97 rows=1 width=22)
                                                        Index Cond: (id = ((array_agg(u.id)))[4])
                                            ->  Index Scan using myapp_book_pkey on myapp_book u_05  (cost=0.44..7.97 rows=1 width=22)
                                                  Index Cond: (id = ((array_agg(u.id)))[5])
                                      ->  Index Scan using myapp_book_pkey on myapp_book u_06  (cost=0.44..7.97 rows=1 width=22)
                                            Index Cond: (id = ((array_agg(u.id)))[6])
                                ->  Index Scan using myapp_book_pkey on myapp_book u_07  (cost=0.44..7.97 rows=1 width=22)
                                      Index Cond: (id = ((array_agg(u.id)))[7])
                          ->  Index Scan using myapp_book_pkey on myapp_book u_08  (cost=0.44..7.97 rows=1 width=22)
                                Index Cond: (id = ((array_agg(u.id)))[8])
                    ->  Index Scan using myapp_book_pkey on myapp_book u_09  (cost=0.44..7.97 rows=1 width=22)
                          Index Cond: (id = ((array_agg(u.id)))[9])
              ->  Index Scan using myapp_book_pkey on myapp_book u_10  (cost=0.44..7.97 rows=1 width=22)
                    Index Cond: (id = ((array_agg(u.id)))[10])
        ->  Index Scan using myapp_book_pkey on myapp_book u_11  (cost=0.44..7.97 rows=1 width=22)
              Index Cond: (id = ((array_agg(u.id)))[11])
  ->  Index Scan using myapp_book_pkey on myapp_book u_12  (cost=0.44..7.97 rows=1 width=22)
        Index Cond: (id = ((array_agg(u.id)))[12])
Run Code Online (Sandbox Code Playgroud)

RDS 实例有 3.75 GB 的内存,但 RDS 似乎限制work_mem为最多 2 GB。

我的查询包含在视图中,因此如果我想定位特定的库,我会使用这些 ID 查询视图,如上所示。问题似乎是 Postgres 如何计划使用视图。如果我在没有视图的情况下运行原始查询,结果会立即返回。只有当我将查询包装在视图中并从中查询 ID 时,才会出现内存错误。

Dan*_*ité 13

首先,让我们假设work_mem1024MB,而不是1024GB报告的不可能的(不可能3GB在机器上总共有)。

总之是太高了。正如PostgreSQL 文档中的Resource Consumption中所说,添加了一些重点:

work_mem(整数)

指定在写入临时磁盘文件之前由内部排序操作和哈希表使用的内存量。该值默认为一兆字节 (1MB)。请注意,对于复杂查询,可能会并行运行多个排序或散列操作;在开始将数据写入临时文件之前,将允许每个操作使用与此值指定的内存一样多的内存。此外,多个正在运行的会话可能会同时执行此类操作。因此,使用的总内存可能是 work_mem 值的许多倍; 在选择值时必须牢记这一事实。排序操作用于 ORDER BY、DISTINCT 和合并连接。哈希表用于哈希连接、基于哈希的聚合和基于哈希的 IN 子查询处理。

显示计划的查询很复杂,需要多个级别的散列,因此您很清楚文档警告的情况。

默认1Mb是保守的,但我也不会提高work_mem上述128MB3GB实例。shared_buffers另一方面可以设置为1024MB:这个只分配一次并保留整个实例的生命周期。