小编Che*_*ter的帖子

我正在努力将JavaRDD(字符串是JSON字符串)转换为数据帧并显示它.我正在做类似下面的事情,

public void call(JavaRDD<String> rdd, Time time) throws Exception {
            if (rdd.count() > 0) {
                JavaRDD<String> filteredRDD = rdd.filter(x -> x.length()>0);                    
                sqlContext = SQLContextSingleton.getInstance(filteredRDD.context());
                DataFrame df = sqlContext.read().schema(SchemaBuilder.buildSchema()).json(filteredRDD);
                df.show();
            }
        }

架构如下所示,

public static StructType buildSchema() {
    StructType schema = new StructType(
            new StructField[] { DataTypes.createStructField("student_id", DataTypes.StringType, false),
                    DataTypes.createStructField("school_id", DataTypes.IntegerType, false),
                    DataTypes.createStructField("teacher", DataTypes.StringType, true),
                    DataTypes.createStructField("rank", DataTypes.StringType, true),
                    DataTypes.createStructField("created", DataTypes.TimestampType, true),
                    DataTypes.createStructField("created_user", DataTypes.StringType, true),
                    DataTypes.createStructField("notes", DataTypes.StringType, true),
                    DataTypes.createStructField("additional_data", DataTypes.StringType, true),
                    DataTypes.createStructField("datetime", DataTypes.TimestampType, true) });
    return (schema);
}

上面的代码回复了我,

|student_id|school_id|teacher|rank|created|created_user|notes|additional_data|datetime|
+----------+------+--------+-----+-----------+-------+------------+--------+-------------+-----+-------------------+---------+---------------+--------+----+-------+-----------+
| …