相关疑难解决方法(0)

我们将XML数据作为名为XML的单个字符串列加载到Hadoop中.我们试图检索到数据级别进行规范化或将其分解为单行进行处理(你知道,就像一个表!)尝试过爆炸功能,但没有得到我们想要的.

示例XML

<Reports>
<Report ID="1">
<Locations>
  <Location ID="20001">
    <LocationName>Irvine Animal Shelter</LocationName>
  </Location>   
  <Location ID="20002">
    <LocationName>Irvine City Hall</LocationName>
  </Location>   
  </Locations>
</Report>
<Report ID="2">
<Locations>
  <Location ID="10001">
    <LocationName>California Fish Grill</LocationName>
  </Location>   
  <Location ID="10002">
    <LocationName>Fukada</LocationName>
  </Location>   
  </Locations>
</Report>
</Reports>

查询1

我们正在查询更高级别的Report.Id,然后是孩子的ID和名称(位置/位置).以下基本上给出了所有可能组合的笛卡尔积(在这个例子中,8行而不是我们希望的4行).

SELECT xpath_int(xml, '/Reports/Report/@ID') AS id, location_id, location_name 
FROM   xmlreports 
LATERAL VIEW explode(xpath(xml, '/Reports/Report/Locations/Location/@ID')) myTable1 AS location_id 
LATERAL VIEW explode(xpath(xml, '/Reports/Report/Locations/Location/LocationName/text()')) myTable2 AS location_name;

查询2

试图分组成一个结构然后爆炸,但这会返回两行和两个数组.

SELECT id, loc.col1, loc.col2
FROM (
SELECT xpath_int(xml, '/Reports/Report/@ID') AS id, 
       array(struct(xpath(xml, '/Reports/Report/Locations/Location/@ID'), xpath(xml,     '/Reports/Report/Locations/Location/LocationName/text()'))) As foo …

我正在努力学习Hive.令人惊讶的是,我找不到如何编写简单的字数统计工作的示例.以下是否正确？

假设我有一个输入文件input.tsv:

hello, world
this is an example input file

我在Python中创建了一个拆分器,将每行转换为单词:

import sys

for line in sys.stdin:
 for word in line.split():
   print word

然后我的Hive脚本中有以下内容:

CREATE TABLE input (line STRING);
LOAD DATA LOCAL INPATH 'input.tsv' OVERWRITE INTO TABLE input;

-- temporary table to hold words...
CREATE TABLE words (word STRING);

add file splitter.py;

INSERT OVERWRITE TABLE words 
  SELECT TRANSFORM(text) 
    USING 'python splitter.py' 
    AS word
  FROM input;

SELECT word, count(*) AS count FROM words GROUP BY word;

我不确定我是否遗漏了某些东西,或者它是否真的很复杂.(特别是,我是否需要临时words表,是否需要编写外部分离器功能？)

可能重复:
SQL Query JOIN with Table

CREATE EXTERNAL TABLE IF NOT EXISTS TestingTable1   (This is the MAIN table through which comparisons need to be made)
(
BUYER_ID BIGINT,
ITEM_ID BIGINT,
CREATED_TIME STRING
)

这是上面第一个表格中的数据

**BUYER_ID**  |  **ITEM_ID**     |      **CREATED_TIME**   
--------------+------------------+-------------------------
 1015826235      220003038067       *2001-11-03 19:40:21*
 1015826235      300003861266        2001-11-08 18:19:59
 1015826235      140002997245        2003-08-22 09:23:17
 1015826235     *210002448035*       2001-11-11 22:21:11

这是Hive中的第二个表 - 它还包含有关我们正在购买的项目的信息.

CREATE EXTERNAL TABLE IF NOT EXISTS TestingTable2
(
USER_ID BIGINT,
PURCHASED_ITEM ARRAY<STRUCT<PRODUCT_ID: BIGINT,TIMESTAMPS:STRING>>
)

这是上面第二个表(TestingTable2)中的数据 -

**USER_ID**    **PURCHASED_ITEM**
1015826235 …