php-fpm7.1 mmap/munmap(非常)在虚拟化系统上的性能很慢(hugepage)

Question

我的php-fpm进程在Ubuntu 14.04 LTS(Nginx服务器,MariaDB数据库)上面临性能问题.

strace -f $(pidof php-fpm7.1 | sed 's/\([0-9]*\)/\-p \1/g')

给我

<... epoll_wait resumed> {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, {}, 1, 103)   = 0
[pid 32533] epoll_wait(8,  <unfinished ...>
[pid 32535] mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933fdd000
[pid 32535] munmap(0x7fd933fdd000, 2097152) = 0
[pid 32535] mmap(NULL, 4190208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933dde000
[pid 32535] munmap(0x7fd933dde000, 139264) = 0
[pid 32535] munmap(0x7fd934000000, 1953792) = 0
[pid 32535] madvise(0x7fd933e00000, 2097152, MADV_HUGEPAGE) = 0
[pid 32533] <... epoll_wait resumed> {}, 1, 897) = 0
[pid 32533] epoll_wait(8, {}, 1, 1000)  = 0
[pid 32533] epoll_wait(8, {}, 1, 1000)  = 0
[pid 32533] epoll_wait(8,  <unfinished ...>
[pid 32535] mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933c00000
[pid 32535] madvise(0x7fd933c00000, 2097152, MADV_HUGEPAGE) = 0
[pid 32533] <... epoll_wait resumed> {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, {}, 1, 1000)  = 0
[pid 32533] epoll_wait(8, {}, 1, 1000)  = 0
[pid 32533] epoll_wait(8, {}, 1, 1000)  = 0
[pid 32533] epoll_wait(8,  <unfinished ...>
[pid 32535] mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933a00000
[pid 32535] madvise(0x7fd933a00000, 2097152, MADV_HUGEPAGE) = 0
[pid 32533] <... epoll_wait resumed> {}, 1, 1000) = 0
[pid 32533] epoll_wait(8,  <unfinished ...>
[pid 32535] open("/usr/share/zoneinfo/UTC", O_RDONLY) = 7
[pid 32535] fstat(7, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 32535] read(7, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
[pid 32535] lseek(7, 0, SEEK_SET)       = 0
[pid 32535] mmap(NULL, 118, PROT_READ, MAP_SHARED, 7, 0) = 0x7fd946835000
[pid 32535] close(7)                    = 0
[pid 32535] munmap(0x7fd946835000, 118) = 0
[pid 32535] pwrite(5, "_sf2_attributes|a:2:{s:14:\"_secu"..., 979, 0) = 979
[pid 32535] close(5)                    = 0
[pid 32535] mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933200000
[pid 32535] madvise(0x7fd933200000, 2097152, MADV_HUGEPAGE) = 0

我试过php-fpm7.0,PHPMod7.1但是同样的问题.

对于包含少量数据的请求,CPU最高可达100%.

配置是默认配置.

在重复的实例上,php5.6-fpm效果很好.

编辑:可能相关的PHP脚本一直在做mmap/munmap

编辑:我试图启用largepages https://wiki.debian.org/Hugepages A cat /proc/meminfo | grep Huge给了我

AnonHugePages:    108544 kB
HugePages_Total:     512
HugePages_Free:      497
HugePages_Rsvd:       50
HugePages_Surp:        0
Hugepagesize:       2048 kB

但仍然是同一个问题.

编辑:我试图启用/禁用OPCache,也设置opcache.huge_code_pages=0,没有结果.http://php.net/上没有关于largepages的文档

Answer 1

执行时括号中选择的值是多少

cat /sys/kernel/mm/transparent_hugepage/enabled 

确保它是 madvise（因为 mmap 通过 MADV_HUGEPAGE 显式请求透明大页）或始终。

下一步是查看您的内核是否不太忙于进行页面压缩（THP 必须为大页面找到 2MB 的连续内存块，如果物理内存碎片，这可能会很困难）。看看下面的数字：

cat /proc/vmstat |grep compact

跑步之前和之后。他们长大了吗？

下一步是捕获进程堆栈的内核部分：

cat /proc/YOUR_PROCESS_PID/stack

其他有用的命令是：

cat /proc/buddyinfo

查看物理内存碎片。查找最后两列。它们是大小为 2MB 和 4MB 的空闲内存块。另一个 ：

cat /proc/pagetypeinfo

并寻找不可移动的页面块。