正如问题所述,我想使用由orElse组成的部分函数作为spark中的udf.这是一个可以在spark shell中运行的示例:
val df = sc.parallelize(1 to 15).toDF("num")
df.show
//Testing out a normal udf - this works
val gt5: (Int => String) = num => (num > 5).toString
val gt5Udf = udf(gt5)
df.withColumn("gt5", gt5Udf(col("num"))).show
//Now create a udf of a partial function composed with orElse
val baseline: PartialFunction[Int, String] = { case _ => "baseline" }
val ge3: PartialFunction[Int, String] = { case x if x >= 3 => ">=3" }
val ge7: PartialFunction[Int, String] = { case x if …