相关疑难解决方法(0)

使用这个简单的例子,我遇到了"ClassNotFound"异常的问题:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

import java.net.URLClassLoader

import scala.util.Marshal

class ClassToRoundTrip(val id: Int) extends scala.Serializable {
}

object RoundTripTester {

  def test(id : Int) : ClassToRoundTrip = {

    // Get the current classpath and output. Can we see simpleapp jar?
    val cl = ClassLoader.getSystemClassLoader
    val urls = cl.asInstanceOf[URLClassLoader].getURLs
    urls.foreach(url => println("Executor classpath is:" + url.getFile))

    // Simply instantiating an instance of object and using it works fine.
    val testObj = new ClassToRoundTrip(id)
    println("testObj.id: " + testObj.id)

    val testObjBytes …

我是Spark streaming和Scala的初学者.对于项目要求,我试图在github中运行存在的TwitterPopularTags示例.由于SBT组装不适合我,我不熟悉SBT,我正在尝试使用Maven进行构建.经过很多初步的打嗝,我能够创建jar文件.但在尝试执行它时,我收到以下错误.有人可以帮我解决这个问题吗？

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$
    at TwitterPopularTags$.main(TwitterPopularTags.scala:43)
    at TwitterPopularTags.main(TwitterPopularTags.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:331)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.twitter.TwitterUtils$
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 9 more

我添加了以下依赖项Spark-streaming_2.10:1.1.0 Spark-core_2.10:1.1.0 Spark-streaming-twitter_2.10:1.1.0

我甚至尝试过1.2.0 for Spark-streaming-twitter,但这也给了我同样的错误.

我在这里先向您的帮助表示感谢.

问候,vpv

是否可以从IntelliJ IDEA中运行sbt-assembly？

另外,我在文档中读到可以在SBT工具窗口中添加任务.但我所看到的是,它只能帮助您查看项目而非任务？我不能在那里添加任何任务.工具窗口如何正常工作？

我有最新版本的IntelliJ IDEA.

我在spark中运行了一个字数统计程序,但是我收到了以下错误 scala-xml_2.11-1.0.2.jar

    Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
    16/12/16 05:14:02 INFO SparkContext: Running Spark version 2.0.2
    16/12/16 05:14:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    16/12/16 05:14:03 WARN Utils: Your hostname, ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.59.132 instead (on interface ens33) 
    16/12/16 05:14:03 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
    16/12/16 05:14:04 INFO SecurityManager: Changing view acls to: hadoopusr
    16/12/16 05:14:04 INFO SecurityManager: Changing …

我必须建立一个jar从我scala使用的代码sbt:

sudo sbt package

它使用了我的构建文件:

name := "PSG CCD"
version := "1.0"
scalaVersion := "2.11.8"

resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven"

libraryDependencies ++= Seq(
        "org.apache.spark" %% "spark-core" % "2.2.0",
        "org.apache.spark" %% "spark-sql" % "2.2.0",
        "org.apache.spark" %% "spark-streaming" % "2.2.0",
        "neo4j-contrib" % "neo4j-spark-connector" % "2.0.0-M2"
)

我也建好了jar.然后,我通过FTP将其传输到我的spark服务器并执行以下操作spark-submit:

 spark-submit --class "PSGApp" --master local[4] psg_ccd.jar 

我收到此错误:

Exception in thread "main" java.lang.NoClassDefFoundError: org/neo4j/spark/Neo4j
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at …