Ber*_*don 5 java debugging profiling jenkins
我正在尝试解决问题,希望您可以通过该方法帮助我:)希望这也可以使其他人受益。
我有一个运行(v2.46.2)的Jenkins服务器。由于某些原因,在浏览Web UI时,某些请求非常缓慢(最多10分钟)。第一次加载UI时尤其如此。然后,它通常是非常敏感的。如果您等待10/15分钟,则第一个请求将再次变得非常缓慢。然后很快。
您将如何解决此问题?到目前为止,这是我所做的:
我从互联网上阅读了许多有趣的内容,包括:Jenkins GUI仅在等待2分钟后显示。在这种情况下,该信息有些陈旧(已过时?),并且由于我的工作仅保留有限数量的版本,因此并没有太大帮助。
这篇博客文章也非常有趣:https : //jenkins.io/blog/2016/11/21/gc-tuning/。就我而言,我不认为问题来自垃圾回收。
我剩下许多假设,例如:
目前,这些只是猜测。理想情况下,我想以某种方式分析服务器响应请求所花费的时间。然后从那里尝试找到这次的去向。
这有可能吗?我尝试使用VisualVM,但这仅显示全局数据,对吗?是否可以隔离用于回答请求的资源?您将如何处理?
注意:我正在发现Java世界(来自Python),所以请不要以为我很了解Java VM的工作方式或您使用的工具:-)
非常感谢!
我迟到了几年,但这似乎是一个常见问题,而且这似乎是搜索中出现的一个合理的帖子,所以我将在这里发布我的经验。
我使用以下方法调试了 Jenkins UI 的缓慢情况:
将 Jenkins 更新到最新版本,以解决我在问题跟踪器和论坛中发现的所有问题,以及在新版本中标记为已修复的问题。这并没有解决它。
在 Linux 主机上“top”查看使用了哪些资源。使用 GUI 时,“jenkins”几乎始终显示 100+% CPU 负载。记忆力似乎不是问题。
内存使用量约为 4GB,因此我尝试将 Jenkins JVM 最小堆大小设置为 16GB,最大堆大小设置为 22GB(主机有 24GB),以确保内存和 GC 不是问题。事实并非如此,用户界面一直很慢,内存使用量保持在最低设置。
"tcpdump" on the host to see what requests it is getting, and if Jenkins tried to poll some network resources that could be slow to respond. Some interesting findings but no real resolution.
using YourKit Java profiler to profile the Jenkins JVM, while manually continuously accessing the GUI to cause the slowness effect. Collecting dumps of CPU use, method invocations, thread scheduling, memory use. Connected the profiler to the Jenkins JVM over SSH, which worked great.
downloading Jenkins source code from Github to match the profiler results to source code and see what those places in the code were doing. Added the profiler plugin and Jenkins source code to IntelliJ IDEA to debug them.
My findings:
Difficult to pinpoint the issue as there seems to be very little visibility.
Per thread, the profiler showed numerous threads with occasional high load but nothing consistent. The overall flamegraph showed a large portion of time spent in Thread.sleep for some plugin. This seemed a bit strange as sleeping should not consumer CPU. It appears YourKit shows shows it to give an overall view of wall-clock time used. So, consider filtering Thread.sleep out to see the real issues.
The flamegraph showed a lot of logging statements interleaved with everything else, which seemed slightly suspicious. Looking at Jenkins source code for the profiler highlighted traces, I saw Jenkins supports very detailed logging. So I went looking a bit deeper into Jenkins logs and how to configure them. Also considered enabling traces (logs) for the profiler highlighted parts to see what is happening.
These logs are set up as log recorders under Manage Jenkins -> System Log -> Log Recorders. I found previously added log recorders there. This was for a Jenkins Java package that was getting accessed all the time and had detailed tracing enabled. Likely added to trace some issue but never removed. Remembering the profiler results, it looked suspicious with the amount of logs it generated, so I removed it.
After this the Jenkins performance improved to a good level. Not quite instant but close. Very much good enough. So in this case the issue was this excess logging configuration that was taking the CPU to 100% and slowing the GUI to a crawl (running in the single main thread I guess..). The main Jenkins log is in /var/log/jenkins and it was not showing any of this log, which was also a bit confusing as it was one of the first things I looked at.
This finding is of course just one potential issue to check. But the above approach worked for me, and might be useful more generally...