从eclipse启动mapreduce作业

Question

我用Java编写了一个mapreduce程序,我可以将其提交给以分布式运行的远程集群.目前,我使用以下步骤提交作业:

1. 将mapreuce作业导出为jar(例如myMRjob.jar)
2. 使用以下shell命令将作业提交到远程集群: hadoop jar myMRjob.jar

当我尝试运行程序时,我想直接从Eclipse提交作业.我怎样才能做到这一点？

我目前正在使用CDH3,我的conf的删节版本是:

conf.set("hbase.zookeeper.quorum", getZookeeperServers());
conf.set("fs.default.name","hdfs://namenode/");
conf.set("mapred.job.tracker", "jobtracker:jtPort");
Job job = new Job(conf, "COUNT ROWS");
job.setJarByClass(CountRows.class);

// Set up Mapper
TableMapReduceUtil.initTableMapperJob(inputTable, scan, 
    CountRows.MyMapper.class, ImmutableBytesWritable.class,  
    ImmutableBytesWritable.class, job);  

// Set up Reducer
job.setReducerClass(CountRows.MyReducer.class);
job.setNumReduceTasks(16);

// Setup Overall Output
job.setOutputFormatClass(MultiTableOutputFormat.class);

job.submit();

当我直接从Eclipse运行它时,该作业已启动,但Hadoop无法找到映射器/缩减器.我收到以下错误:

12/06/27 23:23:29 INFO mapred.JobClient:  map 0% reduce 0%  
12/06/27 23:23:37 INFO mapred.JobClient: Task Id :   attempt_201206152147_0645_m_000000_0, Status : FAILED  
java.lang.RuntimeException: java.lang.ClassNotFoundException:   com.mypkg.mapreduce.CountRows$MyMapper  
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)  
    at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:212)  
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:602)  
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)   
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)  
    at java.security.AccessController.doPrivileged(Native Method)  
    at javax.security.auth.Subject.doAs(Subject.java:396)  
    at   org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)  
    at org.apache.hadoop.mapred.Child.main(Child.java:264)  
...

有谁知道如何克服这些错误？如果我可以解决这个问题,我可以将更多MR作业集成到我的脚本中,这将是非常棒的!

Answer 1

如果您从定义作业类的Eclipse项目中提交hadoop作业,那么您很可能遇到类路径问题.

该job.setjarByClass(CountRows.class)呼叫在寻找的构建类路径类文件,而不是在CountRows.jar(这可能会或可能还没有被建立,甚至在classpath).

您应该能够通过job.getJar()在调用之后打印出结果来断言这是真的job.setjarByClass(..),如果它没有显示jar文件路径,那么它找到了构建类,而不是jar'd类