初始化SparkContext时出错:SparkException

Question

我正在运行以下spark-java代码.

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.hive.HiveContext;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.SQLContext;
import org.apache.hadoop.fs.*;
public class Resource{
public static void main(String[] args) {
SparkConf conf = new SparkConf().setAppName("cust data");
JavaSparkContext jsc = new JavaSparkContext(conf);
HiveContext hiveSQLContext = new    org.apache.spark.sql.hive.HiveContext(jsc.sc());
DataFrame df = hiveSQLContext.sql(" select * from emprecord");
df.registerTempTable("mytempTable");//creating temptable
hiveSQLContext.sql("create table xyz as select * from mytempTable");//inserting into hie table
jsc.close(
}
}

编译代码如下:

 [harsha@hdp-poc1 SparkworkingJars]$ javac -cp $CLASSPATH Resource.java
warning: /home/harsha/SparkworkingJars/spark-core_2.11-    1.6.1.jar(org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.class):     major version 51 is newer than 50, the highest major version supported by this    compiler.
It is recommended that the compiler be upgraded.1 warning

RUN片段如下:

    [harsha@hdp-poc1 SparkworkingJars]$ java -cp $CLASSPATH Resource
16/05/24 10:50:04 INFO spark.SparkContext: Running Spark version 1.6.1
16/05/24 10:50:05 WARN util.NativeCodeLoader: Unable to load native-hadoop     library for your platform... using builtin-java classes where applicable
16/05/24 10:50:05 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your     configuration
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:401)
    at org.apache.spark.api.java.JavaSparkContext.<init>    (JavaSparkContext.scala:59)
    at Resource.main(Resource.java:15)
    16/05/24 10:50:05 INFO spark.SparkContext: Successfully stopped SparkContext
    Exception in thread "main" org.apache.spark.SparkException: A master URL     must be set in your configuration
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:401)
    at org.apache.spark.api.java.JavaSparkContext.<init>    (JavaSparkContext.scala:59)
    at Resource.main(Resource.java:15)

请帮我解决与org.apache.spark.SparkException相关的问题.

Answer 1

您需要在通过spark-submit启动spark应用程序时指定主URL.您可以添加--master local[*]为命令行参数(如果要在独立本地模式下运行(或--master在yarn或mesos上运行的其他选项).参考 - 提交Spark应用程序

否则,您也可以在程序中指定master.

SparkConf conf = new SparkConf().setAppName("cust data").setMaster("local[*]");

但首选方法是前者,因为如果将主服务器保留在源代码之外,则可以将相同的工作启动到不同类型的集群.