Mic*_*eyn 96 linux multithreading kernel linux-kernel ubuntu-12.04
我们希望将服务器上的操作系统从Ubuntu 10.04 LTS升级到Ubuntu 12.04 LTS.不幸的是,似乎运行已经变为可运行的线程的延迟从2.6内核到3.2内核显着增加.事实上,我们得到的延迟数字很难相信.
让我对测试更加具体.我们有一个运行两个线程的程序.第一个线程获取当前时间(使用RDTSC以滴答为单位),然后每秒发送一次条件变量.第二个线程等待条件变量并在发出信号时唤醒.然后它获取当前时间(使用RDTSC以滴答为单位).计算第二个线程中的时间与第一个线程中的时间之间的差异,并在控制台上显示.在此之后,第二个线程再次等待条件变量.大约第二次通过后,第一个线程将再次发出信号.
因此,简而言之,我们得到一个线程,通过条件可变延迟测量一次一次地进行线程通信.
在内核2.6.32中,这种延迟大约为2.8-3.5 us,这是合理的.在内核3.2.0中,这种延迟已经增加到大约40-100 us.我已经排除了两台主机之间硬件的任何差异.它们运行在相同的硬件上(双插槽X5687 {Westmere-EP}处理器,运行频率为3.6 GHz,具有超线程,speedtep和所有C状态关闭).测试应用程序更改线程的亲和力以在同一套接字的独立物理核心上运行它们(即,第一个线程在Core 0上运行,第二个线程在Core 1上运行),因此没有线程的弹跳套接字之间的核心或弹跳/通信.
两台主机之间的唯一区别是,一台运行Ubuntu 10.04 LTS,内核为2.6.32-28(快速上下文切换盒),另一台运行最新的Ubuntu 12.04 LTS,内核为3.2.0-23(缓慢的上下文)开关盒).所有BIOS设置和硬件都相同.
内核是否有任何变化可以解释线程被安排运行多长时间的这种荒谬的减速？
更新: 如果您想在主机和Linux版本上运行测试,我已将代码发布到pastebin供您阅读.编译:
g++ -O3 -o test_latency test_latency.cpp -lpthread
运行(假设您至少有一个双核盒子):
./test_latency 0 1 # Thread 1 on Core 0 and Thread 2 on Core 1
更新2:经过大量内核参数搜索,内核更改和个人研究的帖子后,我已经找出了问题所在并已发布解决方案作为这个问题的答案.
Mic*_*eyn 93
在最近的内核中,坏线程唤醒性能问题的解决方案与intel_idle从acpi_idle旧内核中使用的驱动程序切换到cpuidle驱动程序有关.可悲的是,intel_idle驱动程序忽略用户的C状态的BIOS配置并跳转到自己的曲调.换句话说,即使您完全禁用PC(或服务器)BIOS中的所有C状态,此驱动程序仍会在短暂不活动期间强制启用它们,这几乎总是发生,除非所有核心消耗合成基准(例如,压力) ) 在跑.您可以在大多数兼容硬件上使用精彩的Google i7z工具监控C状态转换以及与处理器频率相关的其他有用信息.
要查看您的设置中当前处于活动状态的cpuidle驱动程序,请current_driver在以下cpuidle部分中捕获该文件/sys/devices/system/cpu:
cat /sys/devices/system/cpu/cpuidle/current_driver
如果您希望现代Linux操作系统具有最低的上下文切换延迟,请添加以下内核启动参数以禁用所有这些省电功能:
在Ubuntu 12.04上,您可以通过将它们添加到GRUB_CMDLINE_LINUX_DEFAULT条目/etc/default/grub然后运行来完成此操作update-grub.要添加的引导参数是:
intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll
以下是三个启动选项的详细信息:
设置intel_idle.max_cstate为零将恢复您的cpuidle驱动程序acpi_idle(至少根据选项的文档),或完全禁用它.在我的盒子上它被完全禁用(即,显示current_driver文件/sys/devices/system/cpu/cpuidle产生输出none).在这种情况下,第二个引导选项processor.max_cstate=0是不必要的.但是,文档指出将intel_idle驱动程序的max_cstate设置为零应将操作系统还原为acpi_idle驱动程序.因此,我放入第二个启动选项以防万一.
该processor.max_cstate选项将acpi_idle驱动程序的最大C状态设置为零,希望也可以禁用它.我没有可以测试它的系统,因为intel_idle.max_cstate=0在我可用的所有硬件上完全敲掉了cpuidle驱动程序.但是,如果您的安装确实将您恢复intel_idle为acpi_idle仅使用第一个引导选项,请告诉我第二个选项processor.max_cstate是否在评论中执行了记录,以便我可以更新此答案.
最后,三个参数中的最后一个idle=poll是真正的耗电量.它将禁用C1/C1E,这将消除最后剩余的延迟时间,代价是更多的功耗,所以只有在真正需要时才使用它.对于大多数人来说这将是过度杀伤,因为C1*延迟并不是那么大.使用我在原始问题中描述的硬件上运行的测试应用程序,延迟从9 us到3 us.对于高延迟敏感的应用程序(例如,金融交易,高精度遥测/跟踪,高频率数据采集等等),这无疑是一个显着的减少,但可能不值得为绝大多数桌面应用.确切知道的唯一方法是分析应用程序的性能改进与硬件功耗/热量的实际增长,并权衡权衡.
更新:
在使用各种idle=*参数进行额外测试后,我发现设置idle为mwait硬件支持是一个更好的主意.似乎使用MWAIT/MONITOR指令允许CPU进入C1E而不会在线程唤醒时间中添加任何明显的延迟.有了idle=mwait,您将获得更低的CPU温度(与之相比idle=poll),更少的功耗,并仍然保持轮询空闲循环的出色低延迟.因此,基于这些发现,针对低CPU线程唤醒延迟的更新推荐的引导参数集是:
intel_idle.max_cstate=0 processor.max_cstate=0 idle=mwait
使用idle=mwait而不是idle=poll也可能有助于启动Turbo Boost(通过帮助CPU保持低于其TDP [热设计功率])和超线程(MWAIT是同时不消耗整个物理核心的理想机制)时间避免更高的C状态).然而,这在测试中尚未得到证实,我将继续这样做.
更新2:
在mwait空闲选项已经从较新的3.x的内核去掉(感谢用户ck_进行更新).这给我们留下了两个选择:
idle=halt- 应该工作mwait,但测试,以确保您的硬件是这种情况.该HLT指令几乎等同于MWAIT带状态提示0.问题在于需要中断才能退出HLT状态,而存储器写入(或中断)可用于退出MWAIT状态.根据Linux内核在其空闲循环中使用的内容,这可以使MWAIT更有效.所以,正如我所说的测试/配置文件,看它是否满足您的延迟需求......
和
idle=poll - 性能最高的选择,牺牲功率和热量.
也许变慢的是futex,它是条件变量的构建块.这将有所启发:
strace -r ./test_latency 0 1 &> test_latency_strace & sleep 8 && killall test_latency
然后
for i in futex nanosleep rt_sig;do echo $i;grep $i test_latency_strace | sort -rn;done
这将显示有趣的系统调用所采用的微秒,按时间排序.
在内核2.6.32上
$ for i in futex nanosleep rt_sig;do echo $i;grep $i test_latency_strace | sort -rn;done
futex
1.000140 futex(0x601ac4, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601ac0, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
1.000129 futex(0x601ac4, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601ac0, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
1.000124 futex(0x601ac4, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601ac0, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
1.000119 futex(0x601ac4, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601ac0, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
1.000106 futex(0x601ac4, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601ac0, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
1.000103 futex(0x601ac4, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601ac0, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
1.000102 futex(0x601ac4, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601ac0, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
0.000125 futex(0x7f98ce4c0b88, FUTEX_WAKE_PRIVATE, 2147483647) = 0
0.000042 futex(0x601b00, FUTEX_WAKE_PRIVATE, 1) = 1
0.000038 futex(0x601b00, FUTEX_WAKE_PRIVATE, 1) = 1
0.000037 futex(0x601b00, FUTEX_WAKE_PRIVATE, 1) = 1
0.000030 futex(0x601b00, FUTEX_WAKE_PRIVATE, 1) = 1
0.000029 futex(0x601b00, FUTEX_WAKE_PRIVATE, 1) = 0
0.000028 futex(0x601b00, FUTEX_WAKE_PRIVATE, 1) = 1
0.000027 futex(0x601b00, FUTEX_WAKE_PRIVATE, 1) = 1
0.000018 futex(0x7fff82f0ec3c, FUTEX_WAKE_PRIVATE, 1) = 0
nanosleep
0.000027 nanosleep({1, 0}, {1, 0}) = 0
0.000019 nanosleep({1, 0}, {1, 0}) = 0
0.000019 nanosleep({1, 0}, {1, 0}) = 0
0.000018 nanosleep({1, 0}, {1, 0}) = 0
0.000018 nanosleep({1, 0}, {1, 0}) = 0
0.000018 nanosleep({1, 0}, {1, 0}) = 0
0.000018 nanosleep({1, 0}, 0x7fff82f0eb40) = ? ERESTART_RESTARTBLOCK (To be restarted)
0.000017 nanosleep({1, 0}, {1, 0}) = 0
rt_sig
0.000045 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000040 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000038 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000035 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000034 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000033 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000032 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000032 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000031 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000031 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000028 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000028 rt_sigaction(SIGRT_1, {0x37f8c052b0, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x37f8c0e4c0}, NULL, 8) = 0
0.000027 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000027 rt_sigaction(SIGRTMIN, {0x37f8c05370, [], SA_RESTORER|SA_SIGINFO, 0x37f8c0e4c0}, NULL, 8) = 0
0.000027 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000025 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000025 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000023 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000023 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000022 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
0.000022 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000021 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000021 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000021 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000021 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000021 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000019 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
在内核3.1.9上
$ for i in futex nanosleep rt_sig;do echo $i;grep $i test_latency_strace | sort -rn;done
futex
1.000129 futex(0x601764, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601760, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
1.000126 futex(0x601764, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601760, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
1.000122 futex(0x601764, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601760, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
1.000115 futex(0x601764, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601760, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
1.000114 futex(0x601764, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601760, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
1.000112 futex(0x601764, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601760, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
1.000109 futex(0x601764, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x601760, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
0.000139 futex(0x3f8b8f2fb0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
0.000043 futex(0x601720, FUTEX_WAKE_PRIVATE, 1) = 1
0.000041 futex(0x601720, FUTEX_WAKE_PRIVATE, 1) = 1
0.000037 futex(0x601720, FUTEX_WAKE_PRIVATE, 1) = 1
0.000036 futex(0x601720, FUTEX_WAKE_PRIVATE, 1) = 1
0.000034 futex(0x601720, FUTEX_WAKE_PRIVATE, 1) = 1
0.000034 futex(0x601720, FUTEX_WAKE_PRIVATE, 1) = 1
nanosleep
0.000025 nanosleep({1, 0}, 0x7fff70091d00) = 0
0.000022 nanosleep({1, 0}, {0, 3925413}) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
0.000021 nanosleep({1, 0}, 0x7fff70091d00) = 0
0.000017 nanosleep({1, 0}, 0x7fff70091d00) = 0
0.000017 nanosleep({1, 0}, 0x7fff70091d00) = 0
0.000017 nanosleep({1, 0}, 0x7fff70091d00) = 0
0.000017 nanosleep({1, 0}, 0x7fff70091d00) = 0
0.000017 nanosleep({1, 0}, 0x7fff70091d00) = 0
rt_sig
0.000045 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000044 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000043 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000040 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000038 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000037 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000036 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000036 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000035 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000035 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000035 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000035 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000034 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000031 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000027 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000027 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000027 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000027 rt_sigaction(SIGRT_1, {0x3f892067b0, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x3f8920f500}, NULL, 8) = 0
0.000026 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000026 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000025 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000024 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000023 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
0.000023 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
0.000022 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
0.000021 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
0.000019 rt_sigaction(SIGRTMIN, {0x3f89206720, [], SA_RESTORER|SA_SIGINFO, 0x3f8920f500}, NULL, 8) = 0
我发现这个5岁的错误报告包含一个比较的"乒乓"性能测试
我不得不补充一下
#include <stdint.h>
为了编译,我用这个命令做了
g++ -O3 -o condvar-perf condvar-perf.cpp -lpthread -lrt
在内核2.6.32上
$ ./condvar-perf 1000000
NPTL
mutex elapsed: 29085 us; per iteration: 29 ns / 9.4e-05 context switches.
c.v. ping-pong test elapsed: 4771993 us; per iteration: 4771 ns / 4.03 context switches.
signal ping-pong test elapsed: 8685423 us; per iteration: 8685 ns / 4.05 context switches.
在内核3.1.9上
$ ./condvar-perf 1000000
NPTL
mutex elapsed: 26811 us; per iteration: 26 ns / 8e-06 context switches.
c.v. ping-pong test elapsed: 10930794 us; per iteration: 10930 ns / 4.01 context switches.
signal ping-pong test elapsed: 10949670 us; per iteration: 10949 ns / 4.01 context switches.
我得出结论,在内核2.6.32和3.1.9之间,上下文切换确实放慢了速度,尽管没有你在内核3.2中观察到的那么多.我意识到这还没有回答你的问题,我会继续挖掘.
编辑:我发现更改进程的实时优先级(两个线程)可以提高3.1.9的性能以匹配2.6.32.但是,在2.6.32上设置相同的优先级会让它变慢...去图 - 我会更多地研究它.
这是我现在的结果:
在内核2.6.32上
$ ./condvar-perf 1000000
NPTL
mutex elapsed: 29629 us; per iteration: 29 ns / 0.000418 context switches.
c.v. ping-pong test elapsed: 6225637 us; per iteration: 6225 ns / 4.1 context switches.
signal ping-pong test elapsed: 5602248 us; per iteration: 5602 ns / 4.09 context switches.
$ chrt -f 1 ./condvar-perf 1000000
NPTL
mutex elapsed: 29049 us; per iteration: 29 ns / 0.000407 context switches.
c.v. ping-pong test elapsed: 16131360 us; per iteration: 16131 ns / 4.29 context switches.
signal ping-pong test elapsed: 11817819 us; per iteration: 11817 ns / 4.16 context switches.
$
在内核3.1.9上
$ ./condvar-perf 1000000
NPTL
mutex elapsed: 26830 us; per iteration: 26 ns / 5.7e-05 context switches.
c.v. ping-pong test elapsed: 12812788 us; per iteration: 12812 ns / 4.01 context switches.
signal ping-pong test elapsed: 13126865 us; per iteration: 13126 ns / 4.01 context switches.
$ chrt -f 1 ./condvar-perf 1000000
NPTL
mutex elapsed: 27025 us; per iteration: 27 ns / 3.7e-05 context switches.
c.v. ping-pong test elapsed: 5099885 us; per iteration: 5099 ns / 4 context switches.
signal ping-pong test elapsed: 5508227 us; per iteration: 5508 ns / 4 context switches.
$