在Spark SQL中,我可以使用
val spark = SparkSession
.builder()
.appName("SparkSessionZipsExample")
.master("local")
.config("spark.sql.warehouse.dir", "warehouseLocation-value")
.getOrCreate()
val df = spark.read.json("source/myRecords.json")
df.createOrReplaceTempView("shipment")
val sqlDF = spark.sql("SELECT * FROM shipment")
从"myRecords.json"获取数据,这个json文件的结构是:
df.printSchema()
root
|-- _id: struct (nullable = true)
| |-- $oid: string (nullable = true)
|-- container: struct (nullable = true)
| |-- barcode: string (nullable = true)
| |-- code: string (nullable = true)
我可以得到这个json的特定列,例如:
val sqlDF = spark.sql("SELECT container.barcode, container.code FROM shipment")
但是如何从这个json文件中获取id.$ oid？我曾尝试"SELECT id.$oid FROM shipment_log"或"SELECT id.\$oid FROM shipment_log" …