如何在 PySpark 中创建具有结构列的数据框而不指定架构？

Question

我正在学习 PySpark，能够快速创建示例数据帧来尝试 PySpark API 的功能很方便。

以下代码（其中spark是 Spark 会话）：

import pyspark.sql.types as T
df = [{'id': 1, 'data': {'x': 'mplah', 'y': [10,20,30]}},
      {'id': 2, 'data': {'x': 'mplah2', 'y': [100,200,300]}},
]
df = spark.createDataFrame(df)
df.printSchema()

给出一个映射（并且不能正确解释数组）：

root
 |-- data: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)
 |-- id: long (nullable = true)

我需要一个结构。如果我给出一个模式，我可以强制一个结构：

import pyspark.sql.types as T
df = [{'id': 1, 'data': {'x': 'mplah', 'y': [10,20,30]}},
      {'id': 2, 'data': {'x': 'mplah2', 'y': [100,200,300]}},
]
schema = T.StructType([
    T.StructField('id', LongType()),
    T.StructField('data', StructType([
        StructField('x', T.StringType()),
        StructField('y', T.ArrayType(T.LongType())),
    ]) )
])
df = spark.createDataFrame(df, schema=schema)
df.printSchema()

这确实给出了：

root
 |-- id: long (nullable = true)
 |-- data: struct (nullable = true)
 |    |-- x: string (nullable = true)
 |    |-- y: array (nullable = true)
 |    |    |-- element: long (containsNull = true)

但这打字太多了。

是否有其他快速方法来创建数据框，以便数据列是一个结构而不指定架构？

Answer 1

When creating an example dataframe, you can use Python's tuples which are transformed into Spark's structs. But this way you cannot specify struct field names.

df = spark.createDataFrame(
    [(1, ('mplah', [10,20,30])),
     (2, ('mplah2', [100,200,300]))],
    ['id', 'data']
)
df.printSchema()
# root
#  |-- id: long (nullable = true)
#  |-- data: struct (nullable = true)
#  |    |-- _1: string (nullable = true)
#  |    |-- _2: array (nullable = true)
#  |    |    |-- element: long (containsNull = true)

Using this approach, you may want to add the schema:

df = spark.createDataFrame(
    [(1, ('mplah', [10,20,30])),
     (2, ('mplah2', [100,200,300]))],
    'id: bigint, data: struct<x:string,y:array<bigint>>'
)
df.printSchema()
# root
#  |-- id: long (nullable = true)
#  |-- data: struct (nullable = true)
#  |    |-- x: string (nullable = true)
#  |    |-- y: array (nullable = true)
#  |    |    |-- element: long (containsNull = true)

However, I often prefer a method using struct. This way detailed schema is not provided and struct field names are taken from column names.

from pyspark.sql import functions as F
df = spark.createDataFrame(
    [(1, 'mplah', [10,20,30]),
     (2, 'mplah2', [100,200,300])],
    ['id', 'x', 'y']
)
df = df.select('id', F.struct('x', 'y').alias('data'))

df.printSchema()
# root
#  |-- id: long (nullable = true)
#  |-- data: struct (nullable = false)
#  |    |-- x: string (nullable = true)
#  |    |-- y: array (nullable = true)
#  |    |    |-- element: long (containsNull = true)