运行spark作为java web应用程序

Question

我使用了Spark ML,并且能够在预测业务问题时获得合理的准确性

数据并不大,我能够使用stanford NLP转换输入(基本上是一个csv文件)并在我的本地机器上运行Naive Bayes进行预测.

我想运行这个预测服务,就像一个简单的java主程序或一个简单的MVC Web应用程序

目前我使用spark-submit命令运行我的预测？相反,我可以从我的servlet /控制器类创建spark上下文和数据框吗？

我找不到关于这种情况的任何文档.

请就上述的可行性提出建议

Answer 1

Spark有REST apis通过调用spark master hostname来提交作业.

提交一份申请:

curl -X POST http://spark-cluster-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
  "action" : "CreateSubmissionRequest",
  "appArgs" : [ "myAppArgument1" ],
  "appResource" : "file:/myfilepath/spark-job-1.0.jar",
  "clientSparkVersion" : "1.5.0",
  "environmentVariables" : {
    "SPARK_ENV_LOADED" : "1"
  },
  "mainClass" : "com.mycompany.MyJob",
  "sparkProperties" : {
    "spark.jars" : "file:/myfilepath/spark-job-1.0.jar",
    "spark.driver.supervise" : "false",
    "spark.app.name" : "MyJob",
    "spark.eventLog.enabled": "true",
    "spark.submit.deployMode" : "cluster",
    "spark.master" : "spark://spark-cluster-ip:6066"
  }
}'

提交回复:

{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20151008145126-0000",
  "serverSparkVersion" : "1.5.0",
  "submissionId" : "driver-20151008145126-0000",
  "success" : true
}

获取已提交应用程序的状态

curl http://spark-cluster-ip:6066/v1/submissions/status/driver-20151008145126-0000

状态响应

{
  "action" : "SubmissionStatusResponse",
  "driverState" : "FINISHED",
  "serverSparkVersion" : "1.5.0",
  "submissionId" : "driver-20151008145126-0000",
  "success" : true,
  "workerHostPort" : "192.168.3.153:46894",
  "workerId" : "worker-20151007093409-192.168.3.153-46894"
}

现在,您提交的spark应用程序应该执行所有操作,并且save output to any datasource and access the data via thrift server没有太多数据要传输(如果要在MVC app db和Hadoop集群之间传输数据,可以考虑sqoop).

学分:link1,link2

编辑:(根据评论中的问题) 构建具有必要依赖关系的spark应用程序jar并以本地模式运行作业.编写jar以读取CSV并使用MLib然后将预测输出存储在某些数据源中以从Web应用程序访问它.