Hadoop:拆分元数据大小超过10000000

Question

当我运行级联工作时,我收到一个错误:

Split metadata size exceeded 10000000

我尝试通过将以下内容传递给命令行来增加每个作业级别的限制

xxx.jar -D mapreduce.job?.split.metainfo.maxsi??ze=30000000

我也试过了

xxx.jar -D mapreduce.jobtracker?.split.metainfo.maxsi??ze=30000000

但两者都不起作用,我仍然得到相同的错误,所以没有选择参数.我正在使用hadoop 2.5.任何人都可以指出我做错了什么？

Answer 1

您可以尝试在中设置以下属性conf/mapred-site.xml：

\n\n

<!-- No limits if set to -1 -->\n<property>\n    <name>mapreduce.jobtracker.split.metainfo.maxsize</name>\n    <value>-1</value>\n</property>\n

\n\n

不确定以下是否有帮助（试一试）

\n\n

xxx.jar -D mapreduce.jobtracker\xe2\x80\x8b.split.metainfo.maxsi\xe2\x80\x8c\xe2\x80\x8bze=-1\n

\n\n

参考： https: //archive.cloudera.com/cdh/3/hadoop/mapred-default.html

\n\n

| Name                                        | Default Value | Description                                                                                                                                                                                                                   |\n|---------------------------------------------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| mapred.jobtracker.job.history.block.size    | 3145728       | The block size of the job history file. Since the job recovery uses job,history, its important to dump job history to disk as soon as possible.,Note that this is an expert level parameter. The default value is set to,3 MB |\n| mapreduce.jobtracker.split.metainfo.maxsize | 10000000      | The maximum permissible size of the split metainfo file. The JobTracker,won\'t attempt to read split metainfo files bigger than the configured,value. No limits if set to -1.                                                  |\n

\n