Apache Giraph - 无法在拆分主/工作模式下运行,因为一次只能执行1个任务

Question

我在这里使用PageRank Benchmark示例运行了带有hadoop 2.2.0的Giraph 1.0.0 .

突然间我得到了这个错误结果:

线程"main"中的异常java.lang.IllegalArgumentException:checkLocalJobRunnerConfiguration:使用LocalJobRunner时,必须只有一个worker,因为一次只能有一个任务!在org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:151)在org.apache.giraph.job.GiraphJob.run(GiraphJob.java:225)在org.apache.giraph.benchmark.GiraphBenchmark.run (GiraphBenchmark.java:90)在org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)在org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)在org.apache. giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:71)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43)在java.lang.reflect.Method.invoke(Method.java:606)在org.apache.hadoop.util.RunJar.main(RunJar.java:212)

当我将工人数量改为1时,我得到了:

线程"main"中的异常java.lang.IllegalArgumentException:checkLocalJobRunnerConfiguration:使用LocalJobRunner时,由于一次只有1个任务,因此无法在拆分主/工作模式下运行!在org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:157)在org.apache.giraph.job.GiraphJob.run(GiraphJob.java:225)在org.apache.giraph.benchmark.GiraphBenchmark.run (GiraphBenchmark.java:90)在org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)在org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)在org.apache. giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:71)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43)在java.lang.reflect.Method.invoke(Method.java:606)在org.apache.hadoop.util.RunJar.main(RunJar.java:212)

有解决方案吗

Answer 1

嗨,我假设您没有在群集上运行？如果我在我们的演示VM中运行,我会得到同样的错误.

您可以在giraph-site.xml中禁用拆分主工作线程行为

giraph.SplitMasterWorker=false

如果您只想在一次性执行期间禁用此功能,您还可以将其作为命令行参数传递给您的程序.

-ca giraph.SplitMasterWorker=false

例如,我为我的大数据讲座运行了一个演示,如下所示:

#!/bin/bash

yarn jar /root/giraph-0.0.1-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.GiraphRunner at.jku.tk.steinbauer.bigdata.giraph.MaxInDegreeComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hue/graph/tinygraph.txt -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat  -op /user/hue/graph/degree -w 1 -ca giraph.SplitMasterWorker=false