小编Fbk*_*bkk的帖子

将 Dataset<Row> 中的行映射到对象类 Spark Java

使用 HBase 和 Parquet,我编写了代码来从 HBase 获取值并将值映射到 Object 类,但是我无法使用 Dataset 用 Parquet 复制它。

HBase:

JavaPairRDD<ImmutableBytesWritable, Result> data = sc.newAPIHadoopRDD(getHbaseConf(),
            TableInputFormat.class, ImmutableBytesWritable.class, Result.class);

JavaRDD<List<Tuple3<Long, Integer, Double>>> tempData = data
                .values()
                //Uses HBaseResultToSimple... class to parse the data.
                .map(value -> {
                    SimpleObject object = oParser.call(value);
                    // Get the sample property, remove leading and ending spaces and split it by comma
                    // to get each sample individually
                    List<Tuple2<String, Integer>> samples = zipWithIndex((object.getSamples().trim().split(",")));

                    // Gets the unique identifier for that sp.
                    Long sp = object.getPos(); …
Run Code Online (Sandbox Code Playgroud)

java functional-programming java-8 apache-spark parquet

1
推荐指数
1
解决办法
1万
查看次数