这些指标对于 Spark 结构化流意味着什么？

Question

spark.streams.addListener(new StreamingQueryListener() {\n    ......\n    override def onQueryProgress(queryProgress: QueryProgressEvent): Unit = {\n        println("Query made progress: " + queryProgress.progress)\n    }\n    ......\n})\n

\n\n

当 StreamingQueryListener 添加到 Spark Structured Streaming 会话并连续输出 queryProgress 时，您将获得的指标之一是urationMs：

\n\n

Query made progress: {\n  ......\n  "durationMs" : {\n    "addBatch" : 159136,\n    "getBatch" : 0,\n    "getEndOffset" : 0,\n    "queryPlanning" : 38,\n    "setOffsetRange" : 14,\n    "triggerExecution" : 159518,\n    "walCommit" : 182\n  }\n  ......\n}\xe2\x80\x8b\n

\n\n

谁能告诉我durationMs中的那些子指标在spark上下文中意味着什么？例如“addBatch 159136”是什么意思。

\n

Answer 1

https://www.waitingforcode.com/apache-spark-structured-streaming/query-metrics-apache-spark-structured-streaming/read

这是一个出色的网站，解决了这些方面以及更多问题，因此将功劳归于该网站。