如何在Hadoop 1.0.4中链接mapper/reducer？

Question

我正在使用Hadoop 1.0.4的"新"API(包org.apache.hadoop.mapreduce中的类).当我想链映射器/缩减器时,我发现ChainMapper,ChainReducer是为"旧"API(包org.apache.hadoop.mapred中的类)编写的.我该怎么办？

Answer 1

我也在寻找同样的东西.我确实得到了答案,尽管我迟到了,我认为分享这个可能对某人有所帮助.

从Hadoop 2.0开始,您可以在包org.apache.hadoop.mapreduce.lib.chain中找到ChainMapper和ChainReducer.

ChainMapper使用模式:

...
Job job = new Job(conf, "MyJob");

Configuration map1Conf = new Configuration(false); 
... ChainMapper.addMapper(job, AMap.class, LongWritable.class, Text.class, Text.class, Text.class, true, map1Conf);

Configuration map2Conf = new Configuration(false); 
... ChainMapper.addMapper(job, BMap.class, Text.class, Text.class, LongWritable.class, Text.class, false, map2Conf);

Configuration map3Conf = new Configuration(false); 
... ChainReducer.setReducer(job, CReducer.class, Text.class, Text.class, LongWritable.class, Text.class, false, map3Conf);
...

job.waitForComplettion(true);
...