Hive: Usage of Concat in Column Name

Question

I am trying to get data from a table that has column name as: year_2016, year_2017, year_2018 etc. I am not sure how to get the data from this table. The data looks like:

| count_of_accidents | year_2016 | year_2017 |year_2018 | 
|--------------------|-----------|-----------|----------|
| 15                 | 12        | 5         | 1        |
| 5                  | 10        | 6         | 18       |

I have tried 'concat' function but this doesn't really work.

I have tried with this:

| count_of_accidents | year_2016 | year_2017 |year_2018 | 
|--------------------|-----------|-----------|----------|
| 15                 | 12        | 5         | 1        |
| 5                  | 10        | 6         | 18       |

The column name (year_2017 or year_2018 etc) will be passed as a parameter. So, I am not really able to hardcode the column name like this-

select SUM( count_of_accidents * concat('year_',year(regexp_replace('2018_1_1','_','-')))) 
from table_name;

Is there any way I can do this?

Answer 1

您可以使用正则表达式来完成。像这样：

--create test table
create table test_col(year_2018 string, year_2019 string);

set hive.support.quoted.identifiers=none;
set hive.cli.print.header=true;

--test select using hard-coded pattern
select year_2018, `(year_)2019` from test_col;
OK
year_2018       year_2019
Time taken: 0.862 seconds

--test pattern parameter
set hivevar:year_param=2019;

select year_2018, `(year_)${year_param}` from test_col;
OK
year_2018       year_2019
Time taken: 0.945 seconds

--two parameters
set hivevar:year_param1=2018;
set hivevar:year_param2=2019;

select `(year_)${year_param1}`, `(year_)${year_param2}`  from test_col t;
OK
year_2018       year_2019
Time taken: 0.159 seconds

--parameter contains full column_name and using more strict regexp pattern
set hivevar:year_param2=year_2019;

select `^${year_param2}$` from test_col t;
OK
year_2019
Time taken: 0.053 seconds

--select all columns using single pattern year_ and four digits
select `^year_[0-9]{4}$`  from test_col t;
OK
year_2018       year_2019

应计算参数并将其传递给 hive 脚本，列名中不支持 concat()、regexp_replace 等函数。

此外，列别名不适用于使用正则表达式提取的列：

 select t.number_of_incidents, `^${year_param}$` as year1 from test_t t;

抛出异常：

失败：SemanticException [错误 10004]：第 1:30 行无效的表别名或列引用“ ^year_2018$”：（可能的列名是：number_of_incidents、year_2016、year_2017、year_2018）

我找到了一种使用 union all 与空数据集对列进行别名的解决方法，请参阅此测试：

create table test_t(number_of_incidents int, year_2016 int, year_2017 int, year_2018 int);
insert into table test_t values(15, 12, 5, 1); --insert test data
insert into table test_t values(5,10,6,18);

--parameter, can be passed from outside the script from command line  
set hivevar:year_param=year_2018;

--enable regex columns and print column names
set hive.support.quoted.identifiers=none;
set hive.cli.print.header=true;

--Alias column using UNION ALL with empty dataset
select sum(number_of_incidents*year1) incidents_year1 
  from
    (--UNION ALL with empty dataset to alias columns extracted
     select 0 number_of_incidents, 0 year1 where false --returns no rows because of false condition
     union all
     select t.number_of_incidents, `^${year_param}$` from test_t t
    )s;

结果：

OK
incidents_year1
105
Time taken: 38.003 seconds, Fetched: 1 row(s)

中的第一个查询UNION ALL不影响数据，因为它不返回任何行。但是它的列名变成了整个 UNION ALL 数据集的名称，可以在上层查询中使用。这个技巧有效。如果您会找到使用正则表达式提取的别名列的更好解决方法，请同时添加您的解决方案。

更新：

如果您可以将完整的 column_name 作为参数传递，则不需要正则表达式。Hive 在查询执行之前按原样替换变量（不计算它们）。仅当由于某种原因无法传递完整的列名时才使用正则表达式，并且在原始查询中需要一些模式连接。看这个测试：

--parameter, can be passed from outside the script from command line  
set hivevar:year_param=year_2018;

select sum(number_of_incidents*${year_param}) incidents_year1 from test_t t;

结果：

OK
incidents_year1
105
Time taken: 63.339 seconds, Fetched: 1 row(s)