Ubuntu 服务器“关闭”~每日

Question

tl;dr：我有一台装有 Ubuntu Server 的机器，我想它 24/7 运行，但它每天都会关闭。我可以调查什么？我已经检查了一些事情（见下文），但我无法修复它并且我已经没有想法了:)

语境

问题

我有一台新机器（Lenovo P340），我想不间断地运行它。这是一台永久连接到电源的台式机。更准确地说，对于 UPS，虽然停电在我住的地方不是问题。

该机器运行 Ubuntu Server 并运行 Docker 之类的东西（如 Web 服务），但几个小时后，它往往会自行“关闭”。我很想知道如何解决这个问题。

我所说的关机是什么意思

当我说关闭时，我的意思是我无法与机器交互：没有任何 docker 应用程序运行，并且我无法 ssh 进入机器。如果我连接键盘并按 Enter 键，或者尝试使用 cntrl+alt+f1/f3 更改“环境”，则不会发生任何情况。现在，我不认为它完全关闭，也许它进入电源模式，因为我可以看到桌面的灯亮着。

这种情况发生在 docker 应用程序运行时，有时甚至在我通过 ssh 或通过 samba 连接时（没有用户的实际输入，只是连接）。

当我在机器上实际活动时（例如，当我在 ssh 上运行命令或执行操作时），我从未见过这种情况发生。这让我觉得这可能与电源管理有关。然而，也可能只是它不是同时发生的。关闭大约每天一次，并且在不同的时间发生。它可以发生在早上、下午或晚上。

我发现“摆脱”它的唯一方法是按住关闭按钮，然后再次打开它。轻轻一按（例如“解锁”）似乎不起作用，但如果没有与机器交互的正确方法，则不确定。

故障排除

我一直在阅读、测试和收集数据。下面列出了我尝试过的一些事情。

假设：系统不是最新的

我目前已经Ubuntu 20.04.1 LTS (GNU/Linux 5.6.0-1042-oem x86_64)定期在机器上运行更新。我已将其作为故障排除的一部分完成。

假设：内存过载并且系统关闭

我发现的最常见原因之一是内存或 CPU 可能过载。机器是新的。CPU 是 Intel Core i9-10900 2.8G 10C vPro，具有 64GB RAM，而我只运行几个（约 10）个容器。

我还讨论了每 15 分钟的内存使用情况快照并存储它们。top -b -o %MEM -n 1 > top.txt这是“停止”之前的示例。

top - 06:30:01 up 1 day, 11:10,  0 users,  load average: 0.05, 0.03, 0.02
Tasks: 511 total,   1 running, 508 sleeping,   0 stopped,   2 zombie
%Cpu(s):  0.6 us,  0.3 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  64033.5 total,  43098.9 free,   7544.9 used,  13389.7 buff/cache
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.  56011.2 avail Mem 

内存使用量约为 64GB 中的 8GB。唯一引起我注意的是 2 个僵尸任务，但除此之外没有一个进程被大量使用。

假设：UPS 监视器告诉机器关闭

服务器连接到 UPS 服务器。我认为这可能是原因，因为有时无法连接（但通常不会关闭）。但是，完全禁用upsmon配置后，UPS 未连接，也未看到其上的日志，并且这种情况仍然发生。

旁注：在电源方面，机器仍然连接到 UPS，只是不监控其状态，而且我住在一个很少出现断电的地区。

假设：如果不使用机器就会休眠

我每 30 分钟拍摄一次快照。运行快照是为了保持机器处于唤醒状态。它们使用 Jenkins 运行，Jenkins 通过 ssh 连接到服务器并运行捕获日志和内存使用情况的两个命令。我希望这算作互动。一整天，我尝试每 5 分钟运行一次，不知怎的，那天没有发生“关机”。不确定是巧合还是由于过程所致，但我正在再次测试以查看结果。

假设：GUI 有电源管理，我应该删除它

我安装了一个 GUI，但我已经按照评论中 @guiverc 的建议卸载了它。

这曾经是我参加的会议：

nito-server:~$ ls /usr/share/xsessions/
gnome-xorg.desktop  gnome.desktop  ubuntu.desktop

我已经按照本教程、这个答案和另一个答案删除了 gnome 和 ubuntu 桌面。

运行后，现在不再显示 GUI：

nito-server:~$ ls /usr/share/xsessions/

尽管如此，系统仍然会定期关闭。

假设：通过 GRUB 禁用电源接口可以解决这个问题

我在这个论坛上做了一些研究，经常看到 GRUB 的更改来配置 Power Interface。我尝试了不同的变体并逐渐增加它：

• GRUB_CMDLINE_LINUX_DEFAULT="text"
• GRUB_CMDLINE_LINUX_DEFAULT="text acpi=force"
• GRUB_CMDLINE_LINUX_DEFAULT="text nomodeset acpi=force"
• GRUB_CMDLINE_LINUX_DEFAULT="text nomodeset pci=noaer"
• GRUB_CMDLINE_LINUX_DEFAULT="text nomodeset acpi=force pci=noaer"

我检查了正在运行的关闭日志sudo journalctl -b -1 -e

Jan 18 05:33:55 nito-server kernel: nvme 0000:04:00.0: AER:    [ 0] RxErr                 
Jan 18 05:36:26 nito-server kernel: pcieport 0000:00:1b.4: AER: Corrected error received: 0000:04:00.0
Jan 18 05:36:26 nito-server kernel: nvme 0000:04:00.0: AER: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan 18 05:36:26 nito-server kernel: nvme 0000:04:00.0: AER:   device [15b7:5006] error status/mask=00000001/0000e000
Jan 18 05:36:26 nito-server kernel: nvme 0000:04:00.0: AER:    [ 0] RxErr                 
Jan 18 05:36:39 nito-server kernel: pcieport 0000:00:1b.4: AER: Corrected error received: 0000:04:00.0
Jan 18 05:36:39 nito-server kernel: nvme 0000:04:00.0: AER: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan 18 05:36:39 nito-server kernel: nvme 0000:04:00.0: AER:   device [15b7:5006] error status/mask=00000001/0000e000
Jan 18 05:36:39 nito-server kernel: nvme 0000:04:00.0: AER:    [ 0] RxErr                 
Jan 18 05:37:39 nito-server kernel: pcieport 0000:00:1b.4: AER: Corrected error received: 0000:04:00.0
Jan 18 05:37:39 nito-server kernel: nvme 0000:04:00.0: AER: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan 18 05:37:39 nito-server kernel: nvme 0000:04:00.0: AER:   device [15b7:5006] error status/mask=00000001/0000e000
Jan 18 05:37:39 nito-server kernel: nvme 0000:04:00.0: AER:    [ 0] RxErr                 
Jan 18 05:39:05 nito-server kernel: pcieport 0000:00:1b.4: AER: Corrected error received: 0000:04:00.0
Jan 18 05:39:05 nito-server kernel: nvme 0000:04:00.0: AER: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan 18 05:39:05 nito-server kernel: nvme 0000:04:00.0: AER:   device [15b7:5006] error status/mask=00000001/0000e000
Jan 18 05:39:05 nito-server kernel: nvme 0000:04:00.0: AER:    [ 0] RxErr                 
Jan 18 05:43:31 nito-server kernel: pcieport 0000:00:1b.4: AER: Corrected error received: 0000:04:00.0
Jan 18 05:43:31 nito-server kernel: nvme 0000:04:00.0: AER: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan 18 05:43:31 nito-server kernel: nvme 0000:04:00.0: AER:   device [15b7:5006] error status/mask=00000001/0000e000
Jan 18 05:43:31 nito-server kernel: nvme 0000:04:00.0: AER:    [ 0] RxErr                 
Jan 18 05:48:59 nito-server kernel: pcieport 0000:00:1b.4: AER: Corrected error received: 0000:04:00.0
Jan 18 05:48:59 nito-server kernel: nvme 0000:04:00.0: AER: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan 18 05:48:59 nito-server kernel: nvme 0000:04:00.0: AER:   device [15b7:5006] error status/mask=00000001/0000e000
Jan 18 05:48:59 nito-server kernel: nvme 0000:04:00.0: AER:    [ 0] RxErr                 
Jan 18 05:49:07 nito-server kernel: pcieport 0000:00:1b.4: AER: Corrected error received: 0000:04:00.0
Jan 18 05:49:07 nito-server kernel: nvme 0000:04:00.0: AER: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan 18 05:49:07 nito-server kernel: nvme 0000:04:00.0: AER:   device [15b7:5006] error status/mask=00000001/0000e000
Jan 18 05:49:07 nito-server kernel: nvme 0000:04:00.0: AER:    [ 0] RxErr                 
Jan 18 05:50:02 nito-server kernel: pcieport 0000:00:1b.4: AER: Corrected error received: 0000:04:00.0
Jan 18 05:50:02 nito-server kernel: nvme 0000:04:00.0: AER: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan 18 05:50:02 nito-server kernel: nvme 0000:04:00.0: AER:   device [15b7:5006] error status/mask=00000001/0000e000
Jan 18 05:50:02 nito-server kernel: nvme 0000:04:00.0: AER:    [ 0] RxErr                 
Jan 18 05:50:32 nito-server kernel: pcieport 0000:00:1b.4: AER: Corrected error received: 0000:04:00.0
Jan 18 05:50:32 nito-server kernel: nvme 0000:04:00.0: AER: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Jan 18 05:50:32 nito-server kernel: nvme 0000:04:00.0: AER:   device [15b7:5006] error status/mask=00000001/0000e000
Jan 18 05:50:32 nito-server kernel: nvme 0000:04:00.0: AER:    [ 0] RxErr     

检查后，最后的配置当前设置为：GRUB_CMDLINE_LINUX_DEFAULT="text pcie_aspm=off"。

这似乎使问题发生的频率降低，但它仍然会发生。

假设：系统（BIOS）具有省电模式

我检查过 BIOS。有增强型省电模式。但是，（1）这是在已经关闭的情况下进入省电模式，而不是关闭；(2) 已禁用。在 Power 中，大多数功能都是关于自动或受控开机。没有其他相关的。

假设： cron 问题导致机器关闭

机器上没有直接配置 cron 作业。唯一的 cron 来自 Jenkins，它配置在 docker 容器内。

crontab -l节目no crontab for nito

至于ll /etc/cron.hourly/节目：

total 20
drwxr-xr-x   2 root root  4096 Aug  1 00:28 ./
drwxr-xr-x 134 root root 12288 Jan 29 16:37 ../
-rw-r--r--   1 root root   102 Feb 14  2020 .placeholder

当前状态和日志

经过上述操作，机器稳定了一段时间，但仍然每隔 48-72 小时就会停机一次。这些是最后的日志日志 ( sudo journalctl -b -1 -e)：

Jan 22 07:03:00 nito-server sshd[74442]: pam_unix(sshd:session): session opened for user nito by (uid=0)
Jan 22 07:03:00 nito-server systemd[1]: Created slice User Slice of UID 1000.
Jan 22 07:03:00 nito-server systemd[1]: Starting User Runtime Directory /run/user/1000...
Jan 22 07:03:00 nito-server systemd-logind[1131]: New session 61 of user nito.
Jan 22 07:03:00 nito-server systemd[1]: Finished User Runtime Directory /run/user/1000.
Jan 22 07:03:00 nito-server systemd[1]: Starting User Manager for UID 1000...
Jan 22 07:03:00 nito-server systemd[74473]: pam_unix(systemd-user:session): session opened for user nito by (uid=0)
Jan 22 07:03:00 nito-server systemd[74473]: Reached target Paths.
Jan 22 07:03:00 nito-server systemd[74473]: Reached target Timers.
Jan 22 07:03:00 nito-server systemd[74473]: Starting D-Bus User Message Bus Socket.
Jan 22 07:03:00 nito-server systemd[74473]: Listening on GnuPG network certificate management daemon.
Jan 22 07:03:00 nito-server systemd[74473]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jan 22 07:03:00 nito-server systemd[74473]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Jan 22 07:03:00 nito-server systemd[74473]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Jan 22 07:03:00 nito-server systemd[74473]: Listening on GnuPG cryptographic agent and passphrase cache.
Jan 22 07:03:00 nito-server systemd[74473]: Listening on debconf communication socket.
Jan 22 07:03:00 nito-server systemd[74473]: Listening on Sound System.
Jan 22 07:03:00 nito-server systemd[74473]: Listening on REST API socket for snapd user session agent.
Jan 22 07:03:00 nito-server systemd[74473]: Listening on D-Bus User Message Bus Socket.
Jan 22 07:03:00 nito-server systemd[74473]: Reached target Sockets.
Jan 22 07:03:00 nito-server systemd[74473]: Reached target Basic System.
Jan 22 07:03:00 nito-server systemd[1]: Started User Manager for UID 1000.
Jan 22 07:03:00 nito-server systemd[74473]: Starting Sound Service...
Jan 22 07:03:00 nito-server systemd[1]: Started Session 61 of user nito.
Jan 22 07:03:00 nito-server rtkit-daemon[8477]: Supervising 0 threads of 0 processes of 1 users.
Jan 22 07:03:00 nito-server rtkit-daemon[8477]: Supervising 0 threads of 0 processes of 1 users.
Jan 22 07:03:00 nito-server rtkit-daemon[8477]: Supervising 0 threads of 0 processes of 1 users.
Jan 22 07:03:00 nito-server rtkit-daemon[8477]: Supervising 0 threads of 0 processes of 1 users.
Jan 22 07:03:00 nito-server rtkit-daemon[8477]: Supervising 0 threads of 0 processes of 1 users.
Jan 22 07:03:00 nito-server systemd[74473]: Started D-Bus User Message Bus.
Jan 22 07:03:00 nito-server dbus-daemon[74579]: [session uid=1000 pid=74579] AppArmor D-Bus mediation is enabled
Jan 22 07:03:00 nito-server systemd[74473]: Started Sound Service.
Jan 22 07:03:00 nito-server systemd[74473]: Reached target Main User Target.
Jan 22 07:03:00 nito-server systemd[74473]: Startup finished in 122ms.
Jan 22 07:03:00 nito-server bluetoothd[1105]: Endpoint registered: sender=:1.477 path=/MediaEndpoint/A2DPSink/sbc
Jan 22 07:03:00 nito-server bluetoothd[1105]: Endpoint registered: sender=:1.477 path=/MediaEndpoint/A2DPSource/sbc
Jan 22 07:03:07 nito-server sshd[74442]: pam_unix(sshd:session): session closed for user nito
Jan 22 07:03:07 nito-server systemd[1]: session-61.scope: Succeeded.
Jan 22 07:03:07 nito-server systemd-logind[1131]: Session 61 logged out. Waiting for processes to exit.
Jan 22 07:03:07 nito-server systemd-logind[1131]: Removed session 61.
Jan 22 07:03:07 nito-server bluetoothd[1105]: Endpoint unregistered: sender=:1.477 path=/MediaEndpoint/A2DPSink/sbc
Jan 22 07:03:07 nito-server bluetoothd[1105]: Endpoint unregistered: sender=:1.477 path=/MediaEndpoint/A2DPSource/sbc
Jan 22 07:03:07 nito-server systemd[74473]: pulseaudio.service: Succeeded.
Jan 22 07:03:17 nito-server systemd[1]: Stopping User Manager for UID 1000...
Jan 22 07:03:17 nito-server systemd[74473]: Stopped target Main User Target.
Jan 22 07:03:17 nito-server systemd[74473]: Stopping D-Bus User Message Bus...
Jan 22 07:03:17 nito-server systemd[74473]: dbus.service: Succeeded.
Jan 22 07:03:17 nito-server systemd[74473]: Stopped D-Bus User Message Bus.
Jan 22 07:03:17 nito-server systemd[74473]: Stopped target Basic System.
Jan 22 07:03:17 nito-server systemd[74473]: Stopped target Paths.
Jan 22 07:03:17 nito-server systemd[74473]: Stopped target Sockets.
Jan 22 07:03:17 nito-server systemd[74473]: Stopped target Timers.
Jan 22 07:03:17 nito-server systemd[74473]: dbus.socket: Succeeded.
Jan 22 07:03:17 nito-server systemd[74473]: Closed D-Bus User Message Bus Socket.
Jan 22 07:03:17 nito-server systemd[74473]: dirmngr.socket: Succeeded.
Jan 22 07:03:17 nito-server systemd[74473]: Closed GnuPG network certificate management daemon.
Jan 22 07:03:17 nito-server systemd[74473]: gpg-agent-browser.socket: Succeeded.
Jan 22 07:03:17 nito-server systemd[74473]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jan 22 07:03:17 nito-server systemd[74473]: gpg-agent-extra.socket: Succeeded.
Jan 22 07:03:17 nito-server systemd[74473]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Jan 22 07:03:17 nito-server systemd[74473]: gpg-agent-ssh.socket: Succeeded.
Jan 22 07:03:17 nito-server systemd[74473]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Jan 22 07:03:17 nito-server systemd[74473]: gpg-agent.socket: Succeeded.
Jan 22 07:03:17 nito-server systemd[74473]: Closed GnuPG cryptographic agent and passphrase cache.
Jan 22 07:03:17 nito-server systemd[74473]: pk-debconf-helper.socket: Succeeded.
Jan 22 07:03:17 nito-server systemd[74473]: Closed debconf communication socket.
Jan 22 07:03:17 nito-server systemd[74473]: pulseaudio.socket: Succeeded.
Jan 22 07:03:17 nito-server systemd[74473]: Closed Sound System.
Jan 22 07:03:17 nito-server systemd[74473]: snapd.session-agent.socket: Succeeded.
Jan 22 07:03:17 nito-server systemd[74473]: Closed REST API socket for snapd user session agent.
Jan 22 07:03:17 nito-server systemd[74473]: Reached target Shutdown.
Jan 22 07:03:17 nito-server systemd[74473]: systemd-exit.service: Succeeded.
Jan 22 07:03:17 nito-server systemd[74473]: Finished Exit the Session.
Jan 22 07:03:17 nito-server systemd[74473]: Reached target Exit the Session.
Jan 22 07:03:17 nito-server systemd[1]: user@1000.service: Succeeded.
Jan 22 07:03:17 nito-server systemd[1]: Stopped User Manager for UID 1000.
Jan 22 07:03:17 nito-server systemd[1]: Stopping User Runtime Directory /run/user/1000...
Jan 22 07:03:17 nito-server systemd[1]: run-user-1000.mount: Succeeded.
Jan 22 07:03:17 nito-server systemd[1]: user-runtime-dir@1000.service: Succeeded.
Jan 22 07:03:17 nito-server systemd[1]: Stopped User Runtime Directory /run/user/1000.
Jan 22 07:03:17 nito-server systemd[1]: Removed slice User Slice of UID 1000.
Jan 22 07:09:52 nito-server wpa_supplicant[1135]: wlo1: WPA: Group rekeying completed with 76:ac:b9:30:c7:b5 [GTK=CCMP]
Jan 22 07:17:01 nito-server CRON[75952]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 22 07:17:01 nito-server CRON[75953]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jan 22 07:17:01 nito-server CRON[75952]: pam_unix(cron:session): session closed for user root

跑步sudo cat /var/log/syslog | grep -i "panic\|error\|hang"

Jan 29 00:00:08 nito-server systemd-resolved[1129]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Jan 29 00:00:08 nito-server systemd-resolved[1129]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Jan 29 00:00:08 nito-server systemd-resolved[1129]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Jan 29 00:00:12 nito-server systemd-resolved[1129]: message repeated 47 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Jan 29 03:01:30 nito-server networkd-dispatcher[1163]: ERROR:Unknown interface index 50 seen even after reload
Jan 29 03:01:30 nito-server networkd-dispatcher[1163]: ERROR:Unknown interface index 50 seen even after reload
Jan 29 03:01:30 nito-server kernel: [22595.674380] IPv6: ADDRCONF(NETDEV_CHANGE): vethc7eb143: link becomes ready
Jan 29 03:33:00 nito-server boltd[117319]: power: state changed: supported/on
Jan 29 03:33:05 nito-server boltd[117319]: power: state changed: supported/wait
Jan 29 03:33:05 nito-server boltd[117319]: power: state changed: supported/on
Jan 29 03:33:25 nito-server boltd[117319]: power: state changed: supported/wait
Jan 29 03:33:45 nito-server boltd[117319]: power: state changed: supported/off
Jan 29 08:31:20 nito-server boltd[117319]: power: state changed: supported/on
Jan 29 08:31:40 nito-server boltd[117319]: power: state changed: supported/wait
Jan 29 08:32:00 nito-server boltd[117319]: power: state changed: supported/off
Jan 29 08:45:13 nito-server NetworkManager[1154]: <info>  [1611881113.6500] dhcp4 (wlo1): state changed bound -> extended
Jan 29 14:12:25 nito-server systemd-resolved[1129]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Jan 29 14:12:25 nito-server systemd-resolved[1129]: message repeated 2 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Jan 29 16:36:01 nito-server systemd-resolved[1129]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Jan 29 16:36:01 nito-server systemd-resolved[1129]: message repeated 2 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Jan 29 16:36:25 nito-server NetworkManager[1154]: <info>  [1611909385.6366] manager: kernel firmware directory '/lib/firmware' changed
Jan 29 16:36:29 nito-server NetworkManager[1154]: <info>  [1611909389.8075] manager: kernel firmware directory '/lib/firmware' changed
Jan 29 16:37:04 nito-server ntpd[351772]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
Jan 29 16:37:04 nito-server ntpd[351772]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized

注意：显示的电源状态更改可能是手动更改，因为 1 月 29 日没有自动关闭。

最终决定

大家好，

感谢您的提示和假设。实施所有更改并运行所有建议的命令后，计算机仍会重新启动。好消息是它每 72 小时而不是 24 小时重新启动一次。

我决定重新安装 Ubuntu Server，看看是否能解决问题。谢谢！

Answer 1

虽然尚不清楚根本原因，但卸载 UI 和电源管理方面的更改将重新启动时间从约 24 小时更改为约 72 小时。

此外，重新安装 Ubuntu Server 后，这似乎不再是问题了。