PIG将文本行转换为稀疏矢量

Question

我需要使用Apache PIG将文件组合在一起.第一个文件包含书籍标题列表,如下所示,每个标题都在行中.

Ted Dunning,  Mahout in Action
Leo Tolstoy,  War and Peace
Douglas Adams, The hitchhiker guide to the galaxy.
James Sununu,  galaxy III for Dummies
Tom McArthur,  The War we went to

第二个文件是单词及其ID的列表.像这样

ted, 12
tom, 13
douglas, 14
galaxy, 15
war, 16
leo, 17
peace, 18

我需要加入这两个文件来生成如下输出:

它应该产生"Leo Tolstoy,War and piece"系列

17:1,16:1,18:1

对于'汤姆麦克阿瑟,战争,我们去'它应该生产

13:1,16:1

换句话说,我需要使用单词作为键来执行连接.到目前为止,我已经在猪中编写了以下代码

titles = LOAD 'Titles' AS ( title : chararray );  
termIDs = LOAD  'TermIDs' AS (  term:chararray,id:int);

A = SAMPLE titles 0.01;
X = FOREACH A GENERATE STRSPLIT(title,'[ _\\[\\]\\/,\\.\\(\\)]+');

这样就可以加载两个文件,X包含每个包含相应行上出现的术语的BAGS列表.像这样:

((ted,dunning,mahout,in,action))
((leo,tolstoy,war,and,peace))

由于周六晚上迟到的原因,我无法在不编写UDF或使用流媒体的情况下找到JOIN步骤的方法.甚至可以仅使用PIG原语.

Answer 1

你可以平移TOKENIZE的结果,因此所有的包都变成行,现在你可以用termsID加入X关系.

X = foreach A generate title, flatten(TOKENIZE(title)) as term;
J = join X by (term),  termIDs by (term);
G = group J by title;
Result = foreach G generate group as title, termIDs.id;

上面的代码是在我的手机上输入的,所以没有调试过.

更新1:

对于最好使用STRSPLIT而不是TOKENIZE的情况,你可以将FLATTEN和TOBAG组合起来,以达到与TOKENIZE相同的效果,TOKENIZE从STRSPLIT返回的元组中得到一个单词.

SPLT = foreach A generate title, FLATTEN(STRSPLIT(title,'[ _\\[\\]\\/,\\.\\(\\)]+'));
X_tmp = foreach SPLT generate $0 as title, FLATTEN(TOBAG($1..$20)) as term; -- pivots the row
X = filter X_tmp by term is not null; -- this removes the extra bag rows when title was split in less than 20 terms
J = join X by (term),  termIDs by (term) using 'replicated';
G = group J by title;
Result = foreach G generate group as title, termIDs.id;

如果任何标题超过20个术语而不是增加TOBAG中的数字.