小编gal*_*esh的帖子

PySpark 和 Kafka“集合消失了。某些数据可能已丢失..”

我正在本地模式下使用 Spark 集群运行 PySpark,并尝试将流式 DataFrame 写入 Kafka 主题。

当我运行查询时,我收到以下消息:

java.lang.IllegalStateException: Set(topicname-0) are gone. Some data may have been missed.. 
Some data may have been lost because they are not available in Kafka any more; either the
 data was aged out by Kafka or the topic may have been deleted before all the data in the
 topic was processed. If you don't want your streaming query to fail on such cases, set the
 source option "failOnDataLoss" to "false".
Run Code Online (Sandbox Code Playgroud)

这是我的代码:

java.lang.IllegalStateException: Set(topicname-0) …
Run Code Online (Sandbox Code Playgroud)

apache-kafka pyspark spark-structured-streaming spark-streaming-kafka

4
推荐指数
1
解决办法
8216
查看次数