小编Pra*_*wal的帖子

使用Apache Spark 2.2:Structured Streaming,我正在创建一个程序,它从Kafka读取数据并将其写入Hive.我正在寻找写入Kafka主题@100记录/秒的批量数据.

蜂巢表创建:

CREATE TABLE demo_user( timeaa BIGINT, numberbb INT, decimalcc DOUBLE, stringdd STRING, booleanee BOOLEAN ) STORED AS ORC ;

通过手动Hive查询插入:

INSERT INTO TABLE demo_user (1514133139123, 14, 26.4, 'pravin', true);

通过spark结构化流媒体代码插入:

SparkConf conf = new SparkConf();
conf.setAppName("testing");
conf.setMaster("local[2]");
conf.set("hive.metastore.uris", "thrift://localhost:9083");
SparkSession session = 
SparkSession.builder().config(conf).enableHiveSupport().getOrCreate();

// workaround START: code to insert static data into hive
String insertQuery = "INSERT INTO TABLE demo_user (1514133139123, 14, 26.4, 'pravin', true)";
session.sql(insertQuery);
// workaround END:

// Solution START
Dataset<Row> dataset = readFromKafka(sparkSession); …