Hadoop Basics的MapReduce程序中的java.lang.NoClassDefFoundError

Question

我正在尝试Hadoop的Basic MapReduce程序,其教程在http://java.dzone.com/articles/hadoop-basics-creating

该类的完整代码是(代码存在于上面的网址上)

import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class Dictionary {
public static class WordMapper extends Mapper<Text, Text, Text, Text> {
    private Text word = new Text();

    public void map(Text key, Text value, Context context) throws IOException, InterruptedException {
        StringTokenizer itr = new StringTokenizer(value.toString(), ",");
        while (itr.hasMoreTokens()) {
            word.set(itr.nextToken());
            context.write(key, word);
        }
    }
}

public static class AllTranslationsReducer extends Reducer<Text, Text, Text, Text> {
    private Text result = new Text();

    public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
        String translations = "";
        for (Text val : values) {
            translations += "|" + val.toString();
        }
        result.set(translations);
        context.write(key, result);
    }
}

public static void main(String[] args) throws Exception {
    System.out.println("welcome to Java 1");
    Configuration conf = new Configuration();
    System.out.println("welcome to Java 2");
    Job job = new Job(conf, "dictionary");
    job.setJarByClass(Dictionary.class);
    job.setMapperClass(WordMapper.class);
    job.setReducerClass(AllTranslationsReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(Text.class);
    job.setInputFormatClass(KeyValueTextInputFormat.class);
    FileInputFormat.addInputPath(job, new Path("/tmp/hadoop-cscarioni/dfs/name/file"));
    FileOutputFormat.setOutputPath(job, new Path("output"));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}

但是在日食之后跑步; 我收到了错误,

welcome to Java 1
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:73)
at Dictionary.main(Dictionary.java:43)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 2 more

Answer 1

请注意,异常是NoClassDefFoundError而不是ClassNotFoundException.

注意:当类在运行时不可见但在编译时可见时,抛出NoClassDefFoundError.这可能是在分发或生成JAR文件时可能发生的事情,其中​​并未包含所有必需的类文件.

解决方法:请检查构建时间和运行时类路径的差异.

NoClassDefFoundError和ClassNotFoundException是不同的.一个是错误,另一个是异常.

NoClassDefFoundError:来自JVM,在找到预期找到的类时遇到问题.由于找不到类文件,在编译时工作的程序无法运行.

ClassNotFoundException:此异常表示在类路径中找不到类,即我们正在尝试加载类定义,并且包含类的类/ jar在类路径中不存在.

Answer 2

当一个类在运行时不可见但在编译时可见时,会出现NoClassDefFoundError.这可能与JAR文件有关,因为未包含所有必需的类文件.

所以尝试在类路径中添加commons-logging-1.1.1 jar,你可以从http://commons.apache.org/logging/download_logging.cgi中获取