尝试使用 pyspark 和 spark sql 将 kafka 密钥(二进制/字节数组)转换为 long/bigint 导致数据类型不匹配:无法将二进制转换为 bigint
环境细节:
Python 3.6.8 |Anaconda custom (64-bit)| (default, Dec 30 2018, 01:22:34)
[GCC 7.3.0] on linux
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.3.0.cloudera2
/_/
Using Python version 3.6.8 (default, Dec 30 2018 01:22:34)
SparkSession available as 'spark'.
Run Code Online (Sandbox Code Playgroud)
测试用例:
from pyspark.sql.types import StructType, StructField, BinaryType
df1_schema = StructType([StructField("key", BinaryType())])
df1_value = [[bytearray([0, 6, 199, 95, …Run Code Online (Sandbox Code Playgroud)