使用 RHEL6 的 12G Dell 服务器上的“功率限制通知”

Question

服务器：Poweredge r620
操作系统：RHEL 6.4
内核：2.6.32-358.18.1.el6.x86_64

我在生产环境中遇到应用程序警报。严重占用 CPU 的进程正在缺乏资源并导致处理积压。该问题发生在最近部署的集群中的所有第 12 代戴尔服务器 (r620s) 上。据我所知，这种情况的发生与 CPU 利用率的峰值相匹配，并伴随着dmesg. 这些事件之一的摘录：

Nov  7 10:15:15 someserver [.crit] CPU12: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU0: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU6: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU14: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU18: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU2: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU4: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU16: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU0: Package power limit notification (total events = 11)
Nov  7 10:15:15 someserver [.crit] CPU6: Package power limit notification (total events = 13)
Nov  7 10:15:15 someserver [.crit] CPU14: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU18: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU20: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU8: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU2: Package power limit notification (total events = 12)
Nov  7 10:15:15 someserver [.crit] CPU10: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU22: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU4: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU16: Package power limit notification (total events = 13)
Nov  7 10:15:15 someserver [.crit] CPU20: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU8: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU10: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU22: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU15: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU3: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU1: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU5: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU17: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU13: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU15: Package power limit notification (total events = 375)
Nov  7 10:15:15 someserver [.crit] CPU3: Package power limit notification (total events = 374)
Nov  7 10:15:15 someserver [.crit] CPU1: Package power limit notification (total events = 376)
Nov  7 10:15:15 someserver [.crit] CPU5: Package power limit notification (total events = 376)
Nov  7 10:15:15 someserver [.crit] CPU7: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU19: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU17: Package power limit notification (total events = 377)
Nov  7 10:15:15 someserver [.crit] CPU9: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU21: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU23: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU11: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU13: Package power limit notification (total events = 376)
Nov  7 10:15:15 someserver [.crit] CPU7: Package power limit notification (total events = 375)
Nov  7 10:15:15 someserver [.crit] CPU19: Package power limit notification (total events = 375)
Nov  7 10:15:15 someserver [.crit] CPU9: Package power limit notification (total events = 374)
Nov  7 10:15:15 someserver [.crit] CPU21: Package power limit notification (total events = 375)
Nov  7 10:15:15 someserver [.crit] CPU23: Package power limit notification (total events = 374)

一点 Google Fu 显示这通常与 CPU 运行热或电压调节启动有关。不过我认为这不是正在发生的事情。群集中所有服务器的温度传感器运行良好，iDRAC 中的功率上限策略已禁用，并且我的系统配置文件在所有这些服务器上都设置为“性能”：

# omreport chassis biossetup | grep -A10 'System Profile'
System Profile Settings
------------------------------------------
System Profile                                    : Performance
CPU Power Management                              : Maximum Performance
Memory Frequency                                  : Maximum Performance
Turbo Boost                                       : Enabled
C1E                                               : Disabled
C States                                          : Disabled
Monitor/Mwait                                     : Enabled
Memory Patrol Scrub                               : Standard
Memory Refresh Rate                               : 1x
Memory Operating Voltage                          : Auto
Collaborative CPU Performance Control             : Disabled

• 戴尔邮件列表帖子几乎完美地描述了这些症状。戴尔建议作者尝试使用性能配置文件，但这没有帮助。他最终应用了戴尔指南中的一些设置，用于为低延迟环境配置服务器，其中一个设置（或其组合）似乎解决了这个问题。
• Kernel.org 错误 #36182指出默认情况下启用了功率限制中断调试，这会导致在 CPU 电压调节开始的情况下性能下降。
• 一篇 RHN KB 文章（需要 RHN 登录）提到了影响 PE r620 和 r720 服务器未运行性能配置文件的问题，并建议对两周前发布的内核进行更新。...除了我们正在运行性能配置文件...

我在网上能找到的一切都让我在这里兜兜转转。这是怎么回事？

Answer 1

导致性能问题的不是电压调节，而是由它触发的调试内核中断。

尽管 Redhat 方面存在一些错误信息，但所有链接页面都指的是相同的现象。电压调节在有或没有性能配置文件的情况下发生，可能是由于启用了Turbo Boost功能。无论出于何种原因，这些电压波动与内核 2.6.32-358.18.1.el6.x86_64 中默认启用的功率限制内核中断的交互都很差。

已确认的解决方法：

• 升级到最新发布的 Redhat 内核 (2.6.32-358.23.2.el6) 会禁用此调试并消除性能问题。
• 添加以下内核参数grub.conf将禁用 PLN：clearcpuid=229

片状解决方法：

• 设置“性能”的系统配置文件。这本身并不足以禁用我们服务器上的 PLN。你的旅费可能会改变。

糟糕的解决方法：

• 将 ACPI 相关模块列入黑名单。我已经在几个论坛主题中看到了这一点。不明智，所以不要。