我正在使用Spark在Scala中执行测试,创建一个SparkContext,如下所示:
val conf = new SparkConf().setMaster("local").setAppName("test")
val sc = new SparkContext(conf)
第一次执行后没有错误.但现在我收到此消息(以及测试通知失败):
Only one SparkContext may be running in this JVM (see SPARK-2243).
看起来我需要检查是否有任何正在运行的SparkContext并在启动一个新的之前停止它(我不想允许多个上下文).我怎样才能做到这一点？
更新:
我试过这个,但是有同样的错误(我正在运行来自IntellijIdea的测试,我在执行之前编写代码):
val conf = new SparkConf().setMaster("local").setAppName("test")
// also tried: .set("spark.driver.allowMultipleContexts", "true")
更新2:
class TestApp extends SparkFunSuite with TestSuiteBase {
// use longer wait time to ensure job completion
override def maxWaitTimeMillis: Int = 20000
System.clearProperty("spark.driver.port")
System.clearProperty("spark.hostPort")
var ssc: StreamingContext = _
val config: SparkConf = new SparkConf().setMaster("local").setAppName("test")
.set("spark.driver.allowMultipleContexts", "true")
val sc: SparkContext = new SparkContext(config) …我尝试在IntellijIdea中打开Maven项目.这个项目使用spark和scala.当我跑:
mvn clean install -DskipTests
然后它给出错误:
[错误]无法执行目标org.scalastyle:scalastyle-maven-plugin:0.7.0:检查(默认)项目spark-mllib_2.10:在scalastyle执行期间失败:您有50个Scalastyle违规(s)
我认为问题出在Maven的版本中.确实在IntellijIdea设置中我有版本Bundled(Maven 3)3.0.5,而pom.xml定义<maven.version>3.3.3</maven.version>
如果我mvn -version在终端运行,它指向Maven 3.3.3
那么,我如何更新IntellijIdea使用的Maven版本？
我使用play json库来处理Scala中的json对象.为了id从json对象获取字段,我运行以下代码:
val id = json \ "id"
然后我想转换id成一个字符串.我试过id.get.toString但不是doi:10.1186-s13612-016-0045-3我得到的JsDefined("doi:10.1186-s13612-016-0045-3")
如何将其转换为字符串？
当我运行以下测试时，它抛出“无法在停止的 SparkContext 上调用方法”。可能的问题是我使用了TestSuiteBaseStreaming Spark Context。在val gridEvalsRDD = ssc.sparkContext.parallelize(gridEvals)我需要使用SparkContext我访问的线路上ssc.sparkContext，这就是我遇到问题的地方（请参阅下面的警告和错误消息）
class StreamingTest extends TestSuiteBase with BeforeAndAfter {
test("Test 1") {
//...
val gridEvals = for (initialWeights <- gridParams("initialWeights");
stepSize <- gridParams("stepSize");
numIterations <- gridParams("numIterations")) yield {
val lr = new StreamingLinearRegressionWithSGD()
.setInitialWeights(initialWeights.asInstanceOf[Vector])
.setStepSize(stepSize.asInstanceOf[Double])
.setNumIterations(numIterations.asInstanceOf[Int])
ssc = setupStreams(inputData, (inputDStream: DStream[LabeledPoint]) => {
lr.trainOn(inputDStream)
lr.predictOnValues(inputDStream.map(x => (x.label, x.features)))
})
val output: Seq[Seq[(Double, Double)]] = runStreams(ssc, numBatches, numBatches)
val cvRMSE = calculateRMSE(output, nPoints)
println(s"RMSE = $cvRMSE")
(initialWeights, …怎么传DataFrame cc进去Array[Seq[String]]？
val factors = $(ccCols).split(",")
val cc = dataset.select(factors.head, factors.tail: _*)
我试过这种方式，但它给了我Array[Row]：
cc.rdd.collect()
这是我在执行spring boot应用程序时收到的错误消息:
java.lang.IllegalStateException: Failed to load ApplicationContext
at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:124)
at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:83)
at org.springframework.boot.test.autoconfigure.AutoConfigureReportTestExecutionListener.prepareTestInstance(AutoConfigureReportTestExecutionListener.java:49)
at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:230)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:228)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:287)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:289)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:247)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from class path resource [elasticsearch-template-test.xml]; nested …