如何收集spark sql输出到文件？

Question

下面是我的spark sql脚本,它加载文件并在其上使用SQL,我想从sql查询中收集输出并将其写入文件,不知道如何能有人帮助.

   //import classes for sql
import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkConf, SparkContext}

val sqlContext = new org.apache.spark.sql.SQLContext(sc)

// createSchemaRDD is used to implicitly convert an RDD to a SchemaRDD.
import sqlContext.createSchemaRDD


//hdfs paths
val warehouse="hdfs://quickstart.cloudera/user/hive/warehouse/"
val customers_path=warehouse+"people/people.txt"
customers_path

//create rdd file called file
val file=sc.textFile(customers_path)

val schemaString="name age"

import org.apache.spark.sql._



val schema =
  StructType(
    schemaString.split(",").map(fieldName => StructField(fieldName, StringType, true)))

val rowRDD=file.map(_.split(",")).map(p => Row(p(0),p(1).trim))

val peopleSchemRDD=sqlContext.applySchema(rowRDD, schema)

// Register the SchemaRDD as a table.
peopleSchemRDD.registerTempTable("people")

// SQL statements can be run by using the sql methods provided by sqlContext.
sqlContext.sql("select count(*) from people").collect().foreach(println)
System.exit(0)

Answer 1

如果您只想计算 HDFS 上一个大文件的行数并将其写入另一个文件：

import java.nio.file.{ Files, Paths }
val path = "hdfs://quickstart.cloudera/user/hive/warehouse/people/people.txt"
val rdd = sc.textFile(path)
val linesCount = rdd.count
Files.write(Paths.get("line_count.txt"), linesCount.toString.getBytes)