PostgreSQL 无法分叉 autovacuum 工作进程:无法分配内存

pha*_*awk 8 postgresql memory

每隔几天就会遇到一些问题,postgres 崩溃并进入恢复模式。postgres 的日志看起来像这样

... Lots of this for 5-10minutes

2015-09-24 10:07:27 GMT LOG:  could not fork autovacuum worker process: Cannot allocate memory
2015-09-24 10:07:28 GMT LOG:  could not fork autovacuum worker process: Cannot allocate memory
2015-09-24 10:07:29 GMT LOG:  could not fork autovacuum worker process: Cannot allocate memory
2015-09-24 10:07:30 GMT LOG:  could not fork autovacuum worker process: Cannot allocate memory
2015-09-24 10:07:32 GMT LOG:  server process (PID 16244) was terminated by signal 9: Killed
2015-09-24 10:07:32 GMT DETAIL:  Failed process was running: SELECT 1
2015-09-24 10:07:32 GMT LOG:  terminating any other active server processes
2015-09-24 10:07:32 GMT WARNING:  terminating connection because of crash of another server process
2015-09-24 10:07:32 GMT DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-09-24 10:07:32 GMT HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2015-09-24 10:07:32 GMT WARNING:  terminating connection because of crash of another server process
2015-09-24 10:07:32 GMT DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.


.... for some time repeats this log:

2015-09-24 10:07:33 GMT HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2015-09-24 10:07:33 GMT WARNING:  terminating connection because of crash of another server process
2015-09-24 10:07:33 GMT DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-09-24 10:07:33 GMT HINT:  In a moment you should be able to reconnect to the database and repeat your command.


.... then

2015-09-24 10:07:33 GMT FATAL:  the database system is in recovery mode
2015-09-24 10:07:33 GMT FATAL:  the database system is in recovery mode
2015-09-24 10:07:33 GMT FATAL:  the database system is in recovery mode
2015-09-24 10:07:33 GMT FATAL:  the database system is in recovery mode
2015-09-24 10:07:33 GMT FATAL:  the database system is in recovery mode
2015-09-24 10:07:33 GMT FATAL:  the database system is in recovery mode
2015-09-24 10:07:33 GMT FATAL:  the database system is in recovery mode
Run Code Online (Sandbox Code Playgroud)

该数据库在 2GB ram 数字海洋水滴上运行。我的 postgresql.conf 如下所示(已注释掉设置,以防您需要查看它将使用的默认设置)

max_connections = 250
shared_buffers = 768mb
temp_buffers = 8MB

#work_mem = 1MB             # min 64kB
#maintenance_work_mem = 16MB        # min 1MB
#max_stack_depth = 2MB          # min 100kB

# - Cost-Based Vacuum Delay -

#vacuum_cost_delay = 0ms        # 0-100 milliseconds
#vacuum_cost_page_hit = 1       # 0-10000 credits
#vacuum_cost_page_miss = 10     # 0-10000 credits
#vacuum_cost_page_dirty = 20        # 0-10000 credits
#vacuum_cost_limit = 200        # 1-10000 credits

# - Background Writer -

#bgwriter_delay = 200ms         # 10-10000ms between rounds
#bgwriter_lru_maxpages = 100        # 0-1000 max buffers written/round
#bgwriter_lru_multiplier = 2.0      # 0-10.0 multipler on buffers scanned/round

# - Asynchronous Behavior -

#effective_io_concurrency = 1       # 1-1000. 0 disables prefetching


#------------------------------------------------------------------------------
# WRITE AHEAD LOG
#------------------------------------------------------------------------------

# - Settings -

wal_level = 'hot_standby'            # minimal, archive, or hot_standby
                    # (change requires restart)
#fsync = on             # turns forced synchronization on or off
#synchronous_commit = on        # synchronization level; on, off, or local
#wal_sync_method = fsync        # the default is the first option
                    # supported by the operating system:
                    #   open_datasync
                    #   fdatasync (default on Linux)
                    #   fsync
                    #   fsync_writethrough
                    #   open_sync
#full_page_writes = on          # recover from partial page writes
#wal_buffers = -1           # min 32kB, -1 sets based on shared_buffers
                    # (change requires restart)
#wal_writer_delay = 200ms       # 1-10000 milliseconds

#commit_delay = 0           # range 0-100000, in microseconds
#commit_siblings = 5            # range 1-1000

# - Checkpoints -

#checkpoint_segments = 3        # in logfile segments, min 1, 16MB each
#checkpoint_timeout = 5min      # range 30s-1h
#checkpoint_completion_target = 0.5 # checkpoint target duration, 0.0 - 1.0
#checkpoint_warning = 30s       # 0 disables

# - Archiving -

archive_mode = on     # allows archiving to be done
                # (change requires restart)
archive_command = 'cd .'       # command to use to archive a logfile segment
#archive_timeout = 0        # force a logfile segment switch after this
                # number of seconds; 0 disables


#------------------------------------------------------------------------------
# REPLICATION
#------------------------------------------------------------------------------

# - Master Server -

# These settings are ignored on a standby server

max_wal_senders = 1        # max number of walsender processes
                # (change requires restart)
#wal_sender_delay = 1s      # walsender cycle time, 1-10000 milliseconds
wal_keep_segments = 100      # in logfile segments, 16MB each; 0 disables
#vacuum_defer_cleanup_age = 0   # number of xacts by which cleanup is delayed
#replication_timeout = 60s  # in milliseconds; 0 disables
#synchronous_standby_names = '' # standby servers that provide sync rep
                # comma-separated list of application_name
                # from standby(s); '*' = all

# - Standby Servers -

# These settings are ignored on a master server

hot_standby = on          # "on" allows queries during recovery
                    # (change requires restart)
#max_standby_archive_delay = 30s    # max delay before canceling queries
                    # when reading WAL from archive;
                    # -1 allows indefinite delay
#max_standby_streaming_delay = 30s  # max delay before canceling queries
                    # when reading streaming WAL;
                    # -1 allows indefinite delay
#wal_receiver_status_interval = 10s # send replies at least this often
                    # 0 disables
#hot_standby_feedback = off     # send info from standby to prevent
                    # query conflicts
Run Code Online (Sandbox Code Playgroud)

非常感谢任何帮助!

dez*_*zso 12

硬件问题...

此日志条目:

postmaster 命令这个服务器进程回滚当前事务并退出,因为另一个服务器进程异常退出并且可能损坏了共享内存。

可能是至少两个不同的潜在问题的结果。第一个是错误的可执行文件或故障硬件- 这就是为什么我建议将您的数据库移动到更好的地方(无论是什么)。

您目前在 Digital Ocean 小滴上,它(正如我刚刚检查过的)是一个虚拟专用服务器。至少对我来说,这并不一定意味着它是一个单独的硬件 - 如果不是,其他用户也有可能受到影响,并且问题由提供商迅速处理。希望那里的配置排除了外国系统对您的系统产生不利影响的可能性。

关于云和共享主机 :) 正如您从上面的评论中看到的,您的问题的更可能根源是您可以解决的问题。

...或内存处理问题?

错误的第二个(我认为更常见)原因是内存压力。如果您的内存不足(可能是这种情况,请参阅下面的计算),操作系统可能会决定终止某个进程以为其他一些进程分配内存。如果操作系统允许内存过量使用,则发生这种情况的可能性比没有时高得多。

查看 PostgreSQL 文档对此有何评论:

在 Linux 2.4 及更高版本中,默认的虚拟内存行为对于 PostgreSQL 不是最佳的。由于内核实现内存过量使用的方式,如果 PostgreSQL 或其他进程的内存需求导致系统耗尽虚拟内存,内核可能会终止 PostgreSQL postmaster(主服务器进程)。

如果发生这种情况,您将看到如下所示的内核消息(请参阅您的系统文档和配置以了解在哪里查找此类消息):

Out of Memory: Killed process 12345 (postgres).
Run Code Online (Sandbox Code Playgroud)

这表明 postgres 进程已因内存压力而终止。尽管现有的数据库连接将继续正常运行,但不会接受新的连接。要恢复,需要重新启动 PostgreSQL。

再往下,它描述了如何改变这一点。有趣且重要的是,您不能完全禁止 OOM 杀手——这对于保持操作系统尽可能长时间运行很重要。因此,您将过度使用行为设置为严格

sysctl -w vm.overcommit_memory=2
Run Code Online (Sandbox Code Playgroud)

(或sysctl.conf通过编辑和重新加载它sysctl)。

或者,您可以将postmaster进程的目标分数设置为尽可能低的值,从而在 OOM 杀手寻找受害者时极不可能选择它。这应该在 root 拥有的启动脚本中完成 - 编辑已经使用的脚本似乎是合适的。这是你需要的:

echo -1000 > /proc/self/oom_score_adj
Run Code Online (Sandbox Code Playgroud)

检查链接的文档页面以获取更多详细信息,每个解决方案都有较小的细节需要观察。

很高兴知道 OOM 杀手只有在物理内存和交换空间都耗尽时才会唤醒。一种廉价的出路是增加交换空间——但是,依赖它对于正常的数据库操作来说通常太慢了。但是,根据您的用例,它可能是解决方案。

请注意,对于这两种方法,您都需要对操作系统进行 root 访问。

一种无需 root 访问即可工作的方法

如果硬件问题可以排除为根本原因,并且您没有 root 访问权限,您仍然可以解决问题。这不会提供万无一失的解决方案,但可以减少问题再次发生的可能性。

让我们快速检查一下原始设置使用了多少内存:

max_connections = 250
shared_buffers = 768MB
temp_buffers = 8MB
# work_mem = 1MB        # a commented-out value means it is at the default - 
                        # in 9.4 it is 4MB
Run Code Online (Sandbox Code Playgroud)

您有 2 GB 的物理内存。

让我们计算一下使用了多少(计算最坏的情况):

  • shared_buffers 总是被占用:768MB
  • work_memtemp_buffers按会话(即连接)分配,并且max_connections是 250:(4MB + 8MB) * 250 = 3000MB
    当然,所有连接都用完所有这些空间的可能性很小。正如您在评论中所述,您一次使用的连接数不超过 70 个,这将数字降低到 840MB
  • maintenance_work_mem并且(可选)autovacuum_work_mem可以消耗更多。您似乎将它们设为默认值,即 64MB。

所有这些加起来为 1672MB。剩下的空间是 2048MB - 1672MB = 376MB。检查linux服务器安装需要多少,我以Ubuntu为例。文档说 192MiB 应该足以满足最低限度的设置——这样,你的设置就可以生存。显然,还有其他进程(都消耗内存)在那里运行,偶尔会耗尽 RAM。

为避免这种情况,您可以降低上述设置。根据数据库大小和典型查询,您可以降低其中的任何一个。在更改配置之前检查哪个用于什么。