Hadoop中的内存不足错误

Question

我尝试按照http://hadoop.apache.org/common/docs/stable/single_node_setup.html文档安装Hadoop .当我尝试执行此操作时

bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+' 

我得到以下例外

java.lang.OutOfMemoryError: Java heap space

请建议一个解决方案,以便我可以尝试这个例子.下面列出了整个例外情况.我是Hadoop的新手,我可能做了一些愚蠢的事情.任何建议将受到高度赞赏.

anuj@anuj-VPCEA13EN:~/hadoop$ bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'
11/12/11 17:38:22 INFO util.NativeCodeLoader: Loaded the native-hadoop library
11/12/11 17:38:22 INFO mapred.FileInputFormat: Total input paths to process : 7
11/12/11 17:38:22 INFO mapred.JobClient: Running job: job_local_0001
11/12/11 17:38:22 INFO util.ProcessTree: setsid exited with exit code 0
11/12/11 17:38:22 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@e49dcd
11/12/11 17:38:22 INFO mapred.MapTask: numReduceTasks: 1
11/12/11 17:38:22 INFO mapred.MapTask: io.sort.mb = 100
11/12/11 17:38:22 WARN mapred.LocalJobRunner: job_local_0001
java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:949)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:428)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
11/12/11 17:38:23 INFO mapred.JobClient:  map 0% reduce 0%
11/12/11 17:38:23 INFO mapred.JobClient: Job complete: job_local_0001
11/12/11 17:38:23 INFO mapred.JobClient: Counters: 0
11/12/11 17:38:23 INFO mapred.JobClient: Job Failed: NA
java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1257)
    at org.apache.hadoop.examples.Grep.run(Grep.java:69)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.examples.Grep.main(Grep.java:93)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

Answer 1

对于使用RPM或DEB软件包的任何人来说,文档和常见建议都会产生误导.这些包将hadoop配置文件安装到/ etc/hadoop中.这些将优先于其他设置.

/etc/hadoop/hadoop-env.sh为Hadoop设置了最大的Java堆内存,默认情况下它是:

   export HADOOP_CLIENT_OPTS="-Xmx128m $HADOOP_CLIENT_OPTS"

此Xmx设置太低,只需将其更改为此并重新运行

   export HADOOP_CLIENT_OPTS="-Xmx2048m $HADOOP_CLIENT_OPTS"

Answer 2

您可以通过编辑conf/mapred-site.xml文件并添加属性来分配更多内存:

  <property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx1024m</value>
  </property>

这将启动具有更多堆空间的hadoop JVM.

Answer 3

另一种可能性是编辑hadoop-env.sh,其中包含export HADOOP_CLIENT_OPTS="-Xmx128m $HADOOP_CLIENT_OPTS".在我的情况下改变128m到1024m(Debian上的Hadoop 1.0.0.1).

Answer 4

在尝试了这么多组合之后,最后我在我的环境(Ubuntu 12.04,Hadoop 1.0.4)上得出同样的错误是由于两个问题.

1. 与上面提到的Zach Gamer相同.
2. 不要忘记先执行"ssh localhost".信不信由你!没有ssh也会在Java堆空间上抛出错误消息.

Answer 5

你需要作出调整,mapreduce.{map|reduce}.java.opts并且也mapreduce.{map|reduce}.memory.mb.

例如:

  hadoop jar <jarName> <fqcn> \
      -Dmapreduce.map.memory.mb=4096 \
      -Dmapreduce.map.java.opts=-Xmx3686m

这是一个很好的资源,回答这个问题