cassandra-stress"无法通过JMX连接;无法收集这些统计信息"

Question

我今天第一次尝试使用cassandra-stress工具.虽然我能够运行该工具,但输出中会显示很多" 无法通过JMX连接;未收集这些统计信息 "消息

命令

cassandra-stress user \
    profile=./stress_write.yaml ops\(insert=1\) \
    n=1000000 \
    -log file=./stress_write.log \
    -node node1,node2,node3,node4,node5,node6

产量

WARN  19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node5) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN  19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node1) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN  19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node2) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN  19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node4) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN  19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node3) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN  19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node5) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN  19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node1) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN  19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node2) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN  19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node4) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN  19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node3) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
INFO  19:44:26 Using data-center name 'DC2' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
INFO  19:44:26 New Cassandra host /node2:9042 added
INFO  19:44:26 New Cassandra host /node5:9042 added
Connected to cluster: MyCluster
INFO  19:44:26 New Cassandra host /node4:9042 added
INFO  19:44:26 New Cassandra host /node1:9042 added
INFO  19:44:26 New Cassandra host /node6:9042 added
Datatacenter: DC2; Host: /node4; Rack: rack1
Datatacenter: DC2; Host: /node3; Rack: rack1
Datatacenter: DC2; Host: /node6; Rack: rack1
Datatacenter: DC2; Host: /node5; Rack: rack1
Datatacenter: DC2; Host: /node1; Rack: rack1
Datatacenter: DC2; Host: /node2; Rack: rack1
INFO  19:44:26 New Cassandra host /node3:9042 added
Created schema. Sleeping 6s for propagation.
Failed to connect over JMX; not collecting these stats
Generating batches with [1..1] partitions and [1..1] rows (of [1..1] total rows in the partitions)
Failed to connect over JMX; not collecting these stats
Failed to connect over JMX; not collecting these stats
Improvement over 4 threadCount: 36%
Failed to connect over JMX; not collecting these stats
Improvement over 8 threadCount: 138%
Failed to connect over JMX; not collecting these stats
Improvement over 16 threadCount: 48%
Failed to connect over JMX; not collecting these stats
Improvement over 24 threadCount: 33%
Failed to connect over JMX; not collecting these stats
Improvement over 36 threadCount: 27%
Failed to connect over JMX; not collecting these stats
Improvement over 54 threadCount: 39%
Failed to connect over JMX; not collecting these stats
Improvement over 81 threadCount: 37%
Failed to connect over JMX; not collecting these stats
Improvement over 121 threadCount: 16%
Failed to connect over JMX; not collecting these stats
Improvement over 181 threadCount: 1%
Failed to connect over JMX; not collecting these stats
Improvement over 271 threadCount: 15%
Failed to connect over JMX; not collecting these stats
Improvement over 406 threadCount: 3%
Failed to connect over JMX; not collecting these stats
Improvement over 609 threadCount: -3%

是否需要为JMX指定任何命令行或基于文件的配置参数？我已经测试并确认压力机和我的节点之间的连接不是问题,因为我能够通过jmxsh在它们之间建立连接.

输出的另一个问题是,它可能与JMX错误有关,也可能与JMX错误无关,因为它缺少一些关键部分.我引用此Datastax 文档页面中的示例输出来显示我所得到的部分:

WARNING: uncertainty mode (err<) results in uneven workload between thread runs, so should be used for high level analysis only
Running with 4 threadCount
Running WRITE with 4 threads until stderr of mean < 0.02
total ops , adj row/s,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,     max,   time,   stderr,  gc: #,  max ms,  sum ms,  sdv ms,      mb
2552      ,      2553,    2553,    2553,    2553,     1.5,     1.4,     2.5,     6.0,    12.6,    18.0,    1.0,  0.00000,      0,       0,       0,       0,       0
5173      ,      2634,    2613,    2613,    2613,     1.5,     1.5,     1.8,     2.6,     8.6,     9.2,    2.0,  0.00000,      0,       0,       0,       0,       0
...

Results:
op rate                   : 3954
partition rate            : 3954
row rate                  : 3954
latency mean              : 1.0
latency median            : 0.8
latency 95th percentile   : 1.5
latency 99th percentile   : 1.8
latency 99.9th percentile : 2.2
latency max               : 73.6
total gc count            : 25
total gc mb               : 1826
total gc time (s)         : 1
avg gc time(ms)           : 37
stdev gc time(ms)         : 10
Total operation time      : 00:00:59
Sleeping for 15s
Running with 4 threadCount

笔记

• 我的群集正在运行DSE 4.6.1(Cassandra 2.0.12)
• 我正在从另一台机器上运行压力工具
• 压力工具版本来自DS C 2.1(Cassandra 2.1)

Answer 1

我有相同的设置（Cassandra 版本是 2.0.12），压力工具来自 2.1，并且看到了类似的问题。终于我有时间去调查了。

我下载了源代码并在调试器中运行它。我看到的是这个错误消息具有误导性。该工具连接到 JMX，但其中一个 mBean ( org.apache.cassandra.service:type=GCInspector) 有问题。

当我使用选项运行压力测试时，我看到了相同的异常：-log level=verbose并看到以下异常：

java.lang.reflect.UndeclaredThrowableException
        at com.sun.proxy.$Proxy11.getAndResetStats(Unknown Source)
        at org.apache.cassandra.tools.NodeProbe.getAndResetGCStats(NodeProbe.java:385)
        at org.apache.cassandra.stress.util.JmxCollector.<init>(JmxCollector.java:86)
        at org.apache.cassandra.stress.StressMetrics.<init>(StressMetrics.java:64)
        at org.apache.cassandra.stress.StressAction.run(StressAction.java:187)
        at org.apache.cassandra.stress.StressAction.warmup(StressAction.java:97)
        at org.apache.cassandra.stress.StressAction.run(StressAction.java:61)
        at org.apache.cassandra.stress.Stress.main(Stress.java:109)
Caused by: javax.management.InstanceNotFoundException: org.apache.cassandra.service:type=GCInspector 
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown Source)
        at ....

我使用 jConsole 连接到 Cassandra，但 2.0.12 版本没有此 mBean。

但我的输出包含示例中引用的大部分数据（垃圾收集统计信息除外）。

您是否尝试过使用默认配置运行 cassandra-stress ？还可以尝试设置详细的日志记录，也许它会给你一些想法。