小编Fil*_*sus的帖子

Apache Spark 休息 API

我正在使用我拥有的 log4j 属性的 spark-submit 命令来调用 Spark-submit,如下所示:

/opt/spark-1.6.2-bin-hadoop2.6/bin/spark-submit \
--driver-java-options \
"-Dlog4j.configuration=file:/home/test_api/log4j-driver.properties\ --class Test testing.jar
Run Code Online (Sandbox Code Playgroud)

如何--driver-java-options通过 curl(Apache Spark 的 Hidden REST API)提交作业?

我试过这个:

curl -X POST http://host-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
"action" : "CreateSubmissionRequest",
"appArgs" : [ "" ],
"appResource" : "hdfs://host-ip:9000/test/testing.jar",
"clientSparkVersion" : "1.6.2",
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass" : "Test",
"spark.driver.extraJavaOptions" : "-Dlog4j.configuration=file:/home/test_api/log4j-driver.properties",
"sparkProperties" : {
"spark.jars" : "hdfs://host-ip:9000/test/testing.jar",
"spark.app.name" : "Test",
"spark.eventLog.enabled": "true",
"spark.eventLog.dir": "hdfs://host-ip:9000/test/spark-events",
"spark.submit.deployMode" : "cluster",
"spark.master" : "spark://host-ip:7077"
}
}' …
Run Code Online (Sandbox Code Playgroud)

curl apache-spark spark-jobserver

5
推荐指数
1
解决办法
2247
查看次数

标签 统计

apache-spark ×1

curl ×1

spark-jobserver ×1