太多打开的文件 (CentOS7) - 已经尝试设置更高的限制

Question

第一次设置 VPS——注意不要在这里问，除非我做了尽职调查并提供了上下文。

在我的远程 VPS 上，通过终端，我运行的几乎所有命令都以一条Error: Too many open files消息结束，我需要您的帮助才能继续前进。

我CentOS Linux release 7.6.1810 (Core)在一台有 1 个 CPU 内核和 2048Mb RAM 的机器上运行。它已经设置了一个Nginx 1.16.1, PHP-FPM 7.3.9, MariaDb 10.4.8用于简单 wordpress 站点的 LEMP 堆栈。

我试过了：

1. 谷歌和论坛搜索。
2. 应用这些设置（每次都通过控制面板手动重启 VPS）：

中的系统范围设置/etc/security/limits.conf：

nginx       soft    nofile      1024
nginx       hard    nofile      65536
root        hard    nofile      65536
root        soft    nofile      1024

调整内存限制和上传/etc/php.ini：

memory_limit = 256M
file_uploads = On
upload_max_filesize = 128M
max_execution_time = 600
max_input_time = 600
max_input_vars = 3000

PHP rlimit 设置/etc/php-fpm.d/www.conf：

rlimit_files = 65535

在 中设置 NGINX 限制（和其他设置）nginx.conf：

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  10000;
}

worker_rlimit_nofile 100000;


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;
    client_body_buffer_size 128k;
    client_header_buffer_size 10k;
    client_max_body_size 100m;
    large_client_header_buffers 4 256k;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*.conf;
    server_names_hash_bucket_size 64;
}

这是输出cat /proc/sys/fs/file-nr：

45216   0   6520154

这是输出ps aux|grep nginx|grep -v grep：

root       928  0.0  0.0  46440  1192 ?        Ss   00:25   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx      929  0.0  0.2  50880  6028 ?        S    00:25   0:00 nginx: worker process
nginx     9973  0.0  0.1 171576  4048 ?        S    04:28   0:00 php-fpm: pool www
nginx     9974  0.0  0.1 171576  4048 ?        S    04:28   0:00 php-fpm: pool www
nginx     9975  0.0  0.1 171576  4048 ?        S    04:28   0:00 php-fpm: pool www
nginx     9976  0.0  0.1 171576  4048 ?        S    04:28   0:00 php-fpm: pool www
nginx     9977  0.0  0.1 171576  4052 ?        S    04:28   0:00 php-fpm: pool www

将用户切换到nginxwithsu - nginx并检查限制 with： ulimit -Snreturns1024 ulimit -Hn返回65536

该lsof | wc -l命令返回：4776

希望您能帮助引导我朝着正确的方向解决文件过多问题！

编辑 - 以下命令显示更多信息：

service nginx restart

Redirecting to /bin/systemctl restart nginx.service
Error: Too many open files
Job for nginx.service failed because a configured resource limit was exceeded. See "systemctl status nginx.service" and "journalctl -xe" for details.
[root@pars ~]# systemctl status nginx.service
? nginx.service - nginx - high performance web server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/nginx.service.d
           ??worker_files_limit.conf
   Active: failed (Result: resources) since Fri 2019-09-13 05:32:23 CEST; 14s ago
     Docs: http://nginx.org/en/docs/
  Process: 1113 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=0/SUCCESS)
  Process: 1125 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
 Main PID: 870 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/virtualizor.service/system.slice/nginx.service

Sep 13 05:32:22 pars.work systemd[1]: Starting nginx - high performance web server...
Sep 13 05:32:22 pars.work systemd[1]: PID file /var/run/nginx.pid not readable (yet?) after start.
Sep 13 05:32:22 pars.work systemd[1]: Failed to set a watch for nginx.service's PID file /var/run/nginx.pid: Too many open files
Sep 13 05:32:23 pars.work systemd[1]: Failed to kill control group: Input/output error
Sep 13 05:32:23 pars.work systemd[1]: Failed to kill control group: Input/output error
Sep 13 05:32:23 pars.work systemd[1]: Failed to start nginx - high performance web server.
Sep 13 05:32:23 pars.work systemd[1]: Unit nginx.service entered failed state.
Sep 13 05:32:23 pars.work systemd[1]: nginx.service failed.

Answer 1

它实际上不是打开的文件句柄已经用完，而是 inotify 监视。

您可以在错误消息中看到这一点：

Sep 13 05:32:22 pars.work systemd[1]: Failed to set a watch for nginx.service's PID file /var/run/nginx.pid: Too many open files

为了解决这个问题，你需要增加系统可用的inotify手表的数量。如果您实际检查，您会发现它的值低得离谱，例如 8192。

$ sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

您可以fs.inotify.max_user_watches通过/etc/sysctl.conf在/etc/sysctl.d目录中编辑或创建文件来持久地将 sysctl设置为更高的值。例如，我的系统有：

$ cat /etc/sysctl.d/10-user-watches.conf 
fs.inotify.max_user_watches = 1048576

然后用sysctl -p.

您可能不想直接进入该数字并导致内核分配内存来跟踪一百万个用户文件监视槽；相反，只需将当前值加倍，直到问题停止发生。