Spark DataDrame中=== null和isNull之间的差异

Question

当我们使用时,我对这种差异感到困惑

 df.filter(col("c1") === null) and df.filter(col("c1").isNull) 

相同的数据帧我在=== null中得到计数,但在isNull中计数为零.请帮我理解其中的区别.谢谢

Answer 1

首先,null除非出于兼容性原因,否则不要在Scala代码中使用.

关于你的问题,这是一个简单的SQL.col("c1") === null被解释为c1 = NULL和,因为NULL标记未定义的值,结果是未定义的包括NULL其自身的任何值.

spark.sql("SELECT NULL = NULL").show

+-------------+
|(NULL = NULL)|
+-------------+
|         null|
+-------------+

spark.sql("SELECT NULL != NULL").show

+-------------------+
|(NOT (NULL = NULL))|
+-------------------+
|               null|
+-------------------+

spark.sql("SELECT TRUE != NULL").show

+------------------------------------+
|(NOT (true = CAST(NULL AS BOOLEAN)))|
+------------------------------------+
|                                null|
+------------------------------------+

spark.sql("SELECT TRUE = NULL").show

+------------------------------+
|(true = CAST(NULL AS BOOLEAN))|
+------------------------------+
|                          null|
+------------------------------+

要检查的唯一有效方法NULL是:

• IS NULL:

  spark.sql("SELECT NULL IS NULL").show
  

  +--------------+
  |(NULL IS NULL)|
  +--------------+
  |          true|
  +--------------+
  

  spark.sql("SELECT TRUE IS NULL").show
  

  +--------------+
  |(true IS NULL)|
  +--------------+
  |         false|
  +--------------+
  

• IS NOT NULL:

  spark.sql("SELECT NULL IS NOT NULL").show
  

  +------------------+
  |(NULL IS NOT NULL)|
  +------------------+
  |             false|
  +------------------+
  

  spark.sql("SELECT TRUE IS NOT NULL").show
  

  +------------------+
  |(true IS NOT NULL)|
  +------------------+
  |              true|
  +------------------+
  

中实现DataFrameDSL作为Column.isNull和Column.isNotNull分别.

注意:

对于NULL-safe比较使用IS DISTINCT/ IS NOT DISTINCT:

spark.sql("SELECT NULL IS NOT DISTINCT FROM NULL").show

+---------------+
|(NULL <=> NULL)|
+---------------+
|           true|
+---------------+

spark.sql("SELECT NULL IS NOT DISTINCT FROM TRUE").show

+--------------------------------+
|(CAST(NULL AS BOOLEAN) <=> true)|
+--------------------------------+
|                           false|
+--------------------------------+

或 not(_ <=> _)/<=>

spark.sql("SELECT NULL AS col1, NULL AS col2").select($"col1" <=> $"col2").show

+---------------+
|(col1 <=> col2)|
+---------------+
|           true|
+---------------+

spark.sql("SELECT NULL AS col1, TRUE AS col2").select($"col1" <=> $"col2").show

+---------------+
|(col1 <=> col2)|
+---------------+
|          false|
+---------------+

DataFrame分别在SQL和DSL中.

相关:

在Apache Spark Join中包含空值