如何过滤掉spark数据框中的布尔字段？

Question

我的数据框中有三列。在这第二个和第三个是布尔字段。我想过滤掉正确的值。我试过这个，nn.filter(col("col3")===true).show但它说无效的列名“true”。我的代码有什么用？请问有什么帮助吗？

我的代码：

scala> nn.printSchema
root
 |-- ID: integer (nullable = true)
 |-- col2: boolean (nullable = true)
 |-- col3: boolean (nullable = true)

scala> nn.show
+---+-----+-----+
| ID| col2| col3|
+---+-----+-----+
|  4| true|false|
|  5|false|false|
|  6|false|false|
|  7|false|false|
| 12|false|false|
| 13|false|false|
| 14|false|false|
| 15|false| true|
| 16|false|false|
| 17|false|false|
| 18|false|false|
| 22|false|false|
| 36|false|false|
| 37|false|false|
| 38|false|false|
| 39|false|false|
| 40|false|false|
| 41| true|false|
| 42|false|false|
+---+-----+-----+

scala> nn.filter(col("col3")===true).show
[Stage 14:>                                                         (0 + 1) / 1]19/05/26 22:44:16 ERROR executor.Executor: Exception in task 0.0 in stage 14.0 (TID 14)
com.microsoft.sqlserver.jdbc.SQLServerException: Invalid column name 'true'.
        at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:217)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1655)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:440)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:385)
        at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2445)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:191)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:166)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:297)
        at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD.compute(JDBCRDD.scala:301)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:109)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Answer 1

您可以直接对布尔值应用过滤器。你为什么对它应用条件 as col("col3")===true？您的列值是布尔类型，当我们应用于过滤器中的条件时，它会返回布尔值true或false。当您的列是布尔值时，您为什么再次尝试相同的内容？

scala> val someDf = Seq((1, true, false), (2, true, true)).toDF("col1", "col2", "col3")
someDf: org.apache.spark.sql.DataFrame = [col1: int, col2: boolean ... 1 more field]

我们有 DF 的值：

scala> someDf.show
+----+----+-----+
|col1|col2| col3|
+----+----+-----+
|   1|true|false|
|   2|true| true|
+----+----+-----+

现在应用过滤器：

scala> someDf.filter(col("col3")).show
+----+----+----+
|col1|col2|col3|
+----+----+----+
|   2|true|true|
+----+----+----+

谢谢。