SparkSQL - 直接读取镶木地板文件

Question

我正在从Impala迁移到SparkSQL,使用以下代码读取表:

my_data = sqlContext.read.parquet('hdfs://my_hdfs_path/my_db.db/my_table')

我如何调用上面的SparkSQL,因此它可以返回如下内容:

'select col_A, col_B from my_table'

Answer 1

从镶木地板文件创建Dataframe后,您必须将其注册为临时表以sql queries在其上运行.

val sqlContext = new org.apache.spark.sql.SQLContext(sc)

val df = sqlContext.read.parquet("src/main/resources/peopleTwo.parquet")

df.printSchema

// after registering as a table you will be able to run sql queries
df.registerTempTable("people")

sqlContext.sql("select * from people").collect.foreach(println)

Answer 2

使用纯SQL

无需在Spark DataFrame上创建表,即可查询JSON,ORC,Parquet和CSV文件.

//This Spark 2.x code you can do the same on sqlContext as well
val spark: SparkSession = SparkSession.builder.master("set_the_master").getOrCreate

spark.sql("select col_A, col_B from parquet.`hdfs://my_hdfs_path/my_db.db/my_table`")
   .show()