为什么spark-submit因"IllegalArgumentException:缺少应用程序资源而失败"而失败？

Question

当我创建我的Spark应用程序的jar并尝试使用它运行时spark-submit,我收到以下错误.

这是我以前运行的命令.

spark-submit --executor-memory 1g --jars s3://test-data-lab-users/spachari/test/test_2.10-1.0.jar

这是我得到的错误.这是否意味着我没有在我的spark-submit中传递正确的参数？

Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
    at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:276)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
    at org.apache.spark.launcher.Main.main(Main.java:86)
Command exiting with ret '1'

Answer 1

tl; dr删除--jars选项并重新开始.

java.lang.IllegalArgumentException:缺少应用程序资源.

你错过了......好吧......消息引用的Spark应用程序称为"应用程序资源".

当您执行spark-submit并查看不同的命令行选项及其含义时,这一点更为明显.

./bin/spark-submit
Usage: spark-submit [options] <app jar | python file | R file> [app arguments]

那部分<app jar | python file | R file>是你错过的.

要重现您的问题,你可以简单地执行spark-submit与--jars选项而没有指定星火应用程序的主要JAR或类.

$ ./bin/spark-submit --jars target/spark-parent_2.11-2.3.0-SNAPSHOT-tests.jar
Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
    at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:274)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
    at org.apache.spark.launcher.Main.main(Main.java:86)

引用spark-submit --help,--jars是......

--jars JARS逗号分隔的jar列表,​​包含在驱动程序和执行程序类路径中.

--jars当一个Spark应用程序依赖于额外的jar文件(也就是依赖项),即mysql-connect.jar你不能(或者很可能不想)"汇编"到你的超级jar 时,它会非常有用.