AEM性能问题(内存泄漏缓慢)org.slf4j.helpers.BasicMarker和org.slf4j.helpers.BasicMarkerFactory

Question

我目前正在将Adobe Experience Manager(AEM也称为CQ)用于客户端站点(Java平台).它使用OpenJDK:

java version "1.7.0_65" 
OpenJDK Runtime Environment (rhel-2.5.1.2.el6_5-x86_64 u65-b17) 
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

它在Rackspace上运行,具有以下内容:

vCPU: 4 
Memory: 16GB 
Guest OS: Red Hat Enterprise Linux 6 (64-bit)

由于它已经投入生产,我在应用程序方面遇到了非常慢的性能.它就是这样我启动应用程序,一切顺利然后7到10天后CPU使用率达到400%(每天约4000个用户点击网站).该网站变得异常缓慢,永远不会成为OOM例外.

由于我是Java内存管理的新手,我开始阅读它是如何工作的,并找到了像jstat和jmap这样的工具.当系统第二次被淹没时,我得到一个堆转储并挖到它.

这一切似乎都在指出org.slf4j.helpers.BasicMarkerFactory和org.slf4j.helpers.BasicMarker,因为当我用MAT eclipse进行分析时,我看到保留大小的最大保留对象是:

org.slf4j.helpers.BasicMarkerFactory @ 0x6021a4f00
Shallow Size: 16 B Retained Size: 6.8 GB

和

当我运行泄漏嫌疑人报告时,我得到以下结果:

Description
One instance of "org.slf4j.helpers.BasicMarkerFactory" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5 @ 0x60219a878" occupies 7,263,024,848 (96.71%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class loader>".

Keywords
java.util.concurrent.ConcurrentHashMap$Segment[]
org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5 @ 0x60219a878
org.slf4j.helpers.BasicMarkerFactory

和

Shortest Paths To the Accumulation Point

Class Name                                                                                  Shallow Heap    Retained Heap

java.util.concurrent.ConcurrentHashMap$Segment[16] @ 0x6021a4f40                                80              7,263,024,784
.
... segments java.util.concurrent.ConcurrentHashMap @ 0x6021a4f10                               48              7,263,024,832
    .
    ... markerMap org.slf4j.helpers.BasicMarkerFactory @ 0x6021a4f00                            16              7,263,024,848
        .
        ... markerFactory org.slf4j.impl.StaticMarkerBinder @ 0x6021d3970                       16                         16
            .
            ... SINGLETON class org.slf4j.impl.StaticMarkerBinder @ 0x6021d38f8                 8                          24
                .
                ... [328] java.lang.Object[640] @ 0x6021d2ee8                                   2,576                   9,592
                    .
                    ... elementData java.util.Vector @ 0x6021d0fe0                              32                      9,624
                        .
                        ... classes org.apache.felix.framework.BundleWiringImpl$
                            BundleClassLoaderJava5 @ 0x6021c32e0                                96                     26,888
                            .
                            ... <classloader> class ch.qos.logback.classic.Logger @ 
                                0x600be4310                                                     16                         16
                            .   .
                            .   ...<class> ch.qos.logback.classic.Logger @ 0x600282a78          48                          48
                            .
                            ... <Java Local> java.lang.Thread @ 0x60077b450
                                pool-9-thread-1 Thread                                          104                     3,344
                                .
                                ... <class> ch.qos.logback.classic.Logger @ 0x60025b850         48                          48
                                .
                                ... <class> ch.qos.logback.classic.Logger @ 0x604b0a708         48                          48
                                .
                                ... <class> ch.qos.logback.classic.Logger @ 0x604b0a6d8         48                          48
                                .
                                ... <class> ch.qos.logback.classic.Logger @ 0x6049debe0         48                          48
                                .
                                ... <class> ch.qos.logback.classic.Logger @ 0x604535228         48                          48
                                .
                                ... <class> ch.qos.logback.classic.Logger @ 0x604124248         48                         48

当我跑的时候:

$ sudo -u aem jmap -histo PID


 num     #instances         #bytes  class name
----------------------------------------------
   1:      11460084      950827248  [C
   2:      10740160      257763840  java.lang.String
   3:       7681495      245807840  java.util.concurrent.ConcurrentHashMap$HashEntry
   4:       7563527      181524648  org.slf4j.helpers.BasicMarker
   5:        217007      173568376  [I
   6:        177602      158721184  [B
   7:         60611       69739136  [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
   8:       1147481       69348496  [Ljava.lang.Object;
   9:       1797107       43130568  org.apache.jackrabbit.oak.plugins.segment.RecordId
  10:        208912       33824544  <constMethodKlass>
  11:        570143       31928008  org.mozilla.javascript.ast.Name
  12:         22350       27643920  <constantPoolKlass>
  13:        208912       26752544  <methodKlass>
  14:        821217       26278944  java.util.UUID
  15:        793800       25401600  java.util.HashMap$Entry
  16:        532946       21317840  org.mozilla.javascript.Node
  17:        792296       19015104  java.lang.Long
  18:        191294       18335600  [Ljava.util.HashMap$Entry;
  19:         22350       16133328  <instanceKlassKlass>
  20:        173883       15855152  [Ljava.lang.String;
  21:        635690       15256560  org.apache.sling.engine.impl.request.SlingRequestProgressTracker$TrackingEntry
  22:         18509       14662848  <constantPoolCacheKlass>
  23:        911112       14577792  java.lang.Integer
  24:        255426       14303856  org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder
  25:        519324       12463776  java.util.ArrayList
  26:        254643       12222864  org.apache.jackrabbit.oak.core.SecureNodeBuilder
  27:        137703       11016240  java.lang.reflect.Method
  28:        312116        9987712  org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState
  29:         19236        9828448  [Lorg.apache.jackrabbit.oak.plugins.segment.SegmentId;
  30:        242179        9687160  java.util.TreeMap$Entry
  31:        197121        9461808  java.util.HashMap
  32:         15041        9416328  <methodDataKlass>
  33:        387927        9310248  org.apache.jackrabbit.oak.plugins.segment.MapRecord
  34:        250049        8001568  org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$UnconnectedHead
  35:        248586        7954752  org.apache.jackrabbit.oak.core.MutableTree
  36:        107865        7948112  [S
  37:        191950        7678000  java.util.LinkedHashMap$Entry
  38:        102212        6541568  org.mozilla.javascript.ast.PropertyGet
  39:         37021        6515696  org.mozilla.javascript.ast.FunctionNode
  40:        161905        6476200  org.mozilla.javascript.ScriptableObject$Slot

.....

8210:             1             16   org.slf4j.helpers.BasicMarkerFactory

我注意到:

   4:       7563527      181524648  org.slf4j.helpers.BasicMarker

和

8210:             1             16  org.slf4j.helpers.BasicMarkerFactory

当我进入org.slf4j.helpers.BasicMarkerFacotry的文档时,我看到以下内容引起了我的注意:

detachMarker
public boolean detachMarker(String name)
Description copied from interface: IMarkerFactory
Detach an existing marker.
Note that after a marker is detached, there might still be "dangling" references to the detached marker.

Specified by:
detachMarker in interface IMarkerFactory
Parameters:
name - The name of the marker to detach
Returns:
whether the marker could be detached or not

特别是:

Note that after a marker is detached, there might still be "dangling" references to the detached marker.

1. 希望有人能够帮助查明我的问题的原因,因为我有点迷失在这里？
2. 谁看过这个吗 ？
3. 我怎么能进一步解决这个问题呢？
4. 你是否同意org.slf4j.helpers.BasicMarker和org.slf4j.helpers.BasicMarkerFactory似乎是我问题的根本原因？
5. 我的日志配置是否可疑？
6. 这是一个缓慢的内存泄漏或性能调整问题(在我看来,似乎内存泄漏一周左右)？

欢迎任何建议.

提前致谢.

Answer 1

您是否使用MarkerFactory来创建Marker 实例？

听起来很明显，ConcurrentHashMap你的标记的持有量在不断增长。例如，如果您每次创建一个具有不同名称的标记（例如使用日期），就会出现这种情况。标记用于标记您的日志，以便您可以相应地过滤它们。您可能想要发布您创建标记的代码部分。