小编Hoa*_*yen的帖子

什么是非类型化 Scala UDF 和类型化 Scala UDF?他们有什么区别?

我已经使用 Spark 2.4 一段时间了,最​​近几天才开始切换到 Spark 3.0。切换到 Spark 3.0 运行后出现此错误udf((x: Int) => x, IntegerType)

Caused by: org.apache.spark.sql.AnalysisException: You're using untyped Scala UDF, which does not have the input type information. Spark may blindly pass null to the Scala closure with primitive-type argument, and the closure will see the default value of the Java type for the null argument, e.g. `udf((x: Int) => x, IntegerType)`, the result is 0 for null input. To get rid of this error, you could: …
Run Code Online (Sandbox Code Playgroud)

scala user-defined-functions apache-spark

8
推荐指数
1
解决办法
4502
查看次数