我已经使用 Spark 2.4 一段时间了,最近几天才开始切换到 Spark 3.0。切换到 Spark 3.0 运行后出现此错误udf((x: Int) => x, IntegerType):
Caused by: org.apache.spark.sql.AnalysisException: You're using untyped Scala UDF, which does not have the input type information. Spark may blindly pass null to the Scala closure with primitive-type argument, and the closure will see the default value of the Java type for the null argument, e.g. `udf((x: Int) => x, IntegerType)`, the result is 0 for null input. To get rid of this error, you could: …Run Code Online (Sandbox Code Playgroud)