相关疑难解决方法(0)

Spark - 流式数据帧/数据集不支持非基于时间的窗口;

我需要使用内部选择和分区来编写 Spark sql 查询。问题是我有 AnalysisException。我已经在这上面花了几个小时,但使用其他方法我没有成功。

例外:

Exception in thread "main" org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;;
Window [sum(cast(_w0#41 as bigint)) windowspecdefinition(deviceId#28, timestamp#30 ASC NULLS FIRST, RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS grp#34L], [deviceId#28], [timestamp#30 ASC NULLS FIRST]
+- Project [currentTemperature#27, deviceId#28, status#29, timestamp#30, wantedTemperature#31, CASE WHEN (status#29 = cast(false as boolean)) THEN 1 ELSE 0 END AS _w0#41]
Run Code Online (Sandbox Code Playgroud)

我认为这太复杂了,无法像这样实现。但我不知道要修复它。

 SparkSession spark = SparkUtils.getSparkSession("RawModel");

 Dataset<RawModel> datasetMap = readFromKafka(spark);

 datasetMap.registerTempTable("test");

 Dataset<Row> res = datasetMap.sqlContext().sql("" +
                " select …
Run Code Online (Sandbox Code Playgroud)

java apache-spark spark-streaming apache-spark-sql

5
推荐指数
1
解决办法
3453
查看次数