使用分区将数据从一个Hive表加载到另一个Hive表

Question

我在一个Hive表中有数据,并希望将数据加载到另一个hive表中.

源表是reg_logs,它有2个分区,日期和小时.数据每小时加载到此表中.架构是:

CREATE EXTERNAL TABLE IF NOT EXISTS reg_logs (
id int,
region_code int,
count int
)
PARTITIONED BY (utc_date STRING, utc_hour STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE
LOCATION '/ad_data/raw/reg_logs';

目标表是reg_logs_org我想要做的就是从utc_hour列旁边的reg_logs复制所有数据.

我创建的架构是:(如果我错了,请更正)

CREATE EXTERNAL TABLE IF NOT EXISTS reg_logs_org (
id int,
region_code int,
count int
)
PARTITIONED BY (utc_date STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE
LOCATION '/ad_data/reg_logs_org';

从reg_logs将数据插入reg_logs_org:

insert overwrite table reg_logs_org
select id, region_code, sum(count), utc_date
from 
reg_logs
group by 
utc_date, id, region_code

错误信息:

 FAILED: SemanticException 1:23 Need to specify partition columns because the destination table is partitioned. Error encountered near token 'reg_logs_org'

==

Thank you,
Rio

Answer 1

这是因为您在插入查询中缺少分区信息

  insert overwrite table reg_logs_org PARTITION (utc_date)
  select id, region_code, sum(count), utc_date
  from 
  reg_logs
  group by 
  utc_date, id, region_code

Answer 2

创建表的副本

CREATE TABLE my_table_backup LIKE my_table;

启用动态分区

SET hive.exec.dynamic.partition = true;
SET hive.exec.dynamic.partition.mode = nonstrict;
SET hive.mapred.mode = nonstrict;

复制表格

INSERT OVERWRITE TABLE dim_data_products_cube_backup PARTITION (ds)
SELECT * FROM dim_data_products_cube 
WHERE ds = ds;

如果使用严格模式,则需要where子句.

Answer 3

在某些情况下,你可能需要设置hive.exec.dynamic.partition.mode = 不严格 ,能够将数据插入到分区表,例如,

CREATE TABLE hivePartitionedTable
(
          c1    int
        , c2    int
        , c3    string
)
PARTITIONED BY (year  int)
ROW FORMAT DELIMITED
    FIELDS TERMINATED BY ','
STORED AS SEQUENCEFILE
;

然后这个INSERT将工作:

set hive.exec.dynamic.partition.mode=nonstrict;

INSERT INTO hivePartitionedTable PARTITION (year)
VALUES (1,2,'3', 1999);