我不确定mergestrategy或exclude jars是否是最好的选择.任何有关如何进一步处理此错误的帮助都会很棒!
[sameert@pzxdcc0151 approxstrmatch]$ sbt assembly
[info] Loading project definition from /apps/sameert/software/approxstrmatch/project
[info] Set current project to approxstrmatch (in build file:/apps/sameert/software/approxstrmatch/)
[info] Including from cache: scala-library.jar
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[info] Including from cache: curator-client-2.4.0.jar
[info] Including from cache: secondstring-20140729.jar
[info] Including from cache: slf4j-api-1.7.5.jar
[info] Including from cache: jsr305-1.3.9.jar
[info] Including from cache: jul-to-slf4j-1.7.5.jar
[info] Including from cache: jcl-over-slf4j-1.7.5.jar
[info] Including from cache: commons-digester-1.8.jar
[info] Including from cache: compress-lzf-1.0.0.jar
[info] Including from cache: commons-beanutils-1.7.0.jar …在sbt空目录中运行命令时,它会一直失败并出现以下异常:
$ sbt
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
Error wrapping InputStream in GZIPInputStream: java.io.EOFException
at sbt.ErrorHandling$.translate(ErrorHandling.scala:11)
at sbt.WrapUsing.open(Using.scala:36)
at sbt.Using.apply(Using.scala:24)
at sbt.IO$$anonfun$gzipFileIn$1.apply(IO.scala:778)
at sbt.IO$$anonfun$gzipFileIn$1.apply(IO.scala:777)
at sbt.Using.apply(Using.scala:25)
at sbt.IO$.gzipFileIn(IO.scala:777)
at sbt.Sync$.readUncaught(Sync.scala:88)
at sbt.Sync$.readInfo(Sync.scala:84)
at sbt.Sync$$anonfun$apply$1.apply(Sync.scala:29)
at sbt.Sync$$anonfun$apply$1.apply(Sync.scala:23)
at sbt.Defaults$$anonfun$copyResourcesTask$1.apply(Defaults.scala:770)
at sbt.Defaults$$anonfun$copyResourcesTask$1.apply(Defaults.scala:766)
at scala.Function4$$anonfun$tupled$1.apply(Function4.scala:35)
at scala.Function4$$anonfun$tupled$1.apply(Function4.scala:34)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
at sbt.std.Transform$$anon$4.work(System.scala:64)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
at sbt.Execute.work(Execute.scala:244)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at …这是一个例子build.sbt:
import AssemblyKeys._
assemblySettings
buildInfoSettings
net.virtualvoid.sbt.graph.Plugin.graphSettings
name := "scala-app-template"
version := "0.1"
scalaVersion := "2.9.3"
val FunnyRuntime = config("funnyruntime") extend(Compile)
libraryDependencies += "org.spark-project" %% "spark-core" % "0.7.3" % "provided"
sourceGenerators in Compile <+= buildInfo
buildInfoPackage := "com.psnively"
buildInfoKeys := Seq[BuildInfoKey](name, version, scalaVersion, target)
assembleArtifact in packageScala := false
val root = project.in(file(".")).
configs(FunnyRuntime).
settings(inConfig(FunnyRuntime)(Classpaths.configSettings ++ baseAssemblySettings ++ Seq(
libraryDependencies += "org.spark-project" %% "spark-core" % "0.7.3" % "funnyruntime"
)): _*)
目标是使用spark-core,"provided"因此它和它的依赖项不包含在程序集工件中,而是将它们重新包含在运行时类路径中以用于run- 和 - test相关的任务. …
我刚刚偶然发现了sbt-pack插件.开发流似乎很稳定.令我惊讶的是,我认为(引用sbt-pack的标题)"创建可分发的Scala包"的唯一插件.是sbt-assembly(在其他功能中).
插件之间的主要区别是什么？我什么时候应该使用另一个？
我正在尝试运行sbt汇编.根据https://github.com/sbt/sbt-assembly ,对于sbt 0.13.6+(我在0.13.7上),这应该自动包含在JvmPlugin的任何内容中.根据sbt plugins我在root中启用了JvmPlugin.当我跑步时,sbt assembly我得到"不是一个有效的commamdn:assembly".我尝试过使用包含sbt-assembly的旧方法和所有不同类型的sbt配置,但似乎都没有.这是我的构建文件的样子(注意sbt包工作正常)
assembly.sbt
addSbtPlugin("com.eed3si8n" % "sbt-assembly" % "0.13.0")
build.sbt
lazy val commonSettings = Seq(
organization := "com.test",
version := "1.0",
scalaVersion := "2.10.4"
)
lazy val root = (project in file(".")).
settings(commonSettings: _*).
settings(
name := "test",
resolvers ++= Seq(
...
),
libraryDependencies ++= Seq(
...
)
)
这是错误:
[error] Not a valid command: assembly
[error] Not a valid project ID: assembly
[error] Expected ':' (if selecting a configuration)
[error] Not a …我正在尝试使用sbt-assembly将scala项目转换为可部署的胖jar .当我在sbt中运行我的程序集任务时,我收到以下错误:
Merging 'org/apache/commons/logging/impl/SimpleLog.class' with strategy 'deduplicate'
:assembly: deduplicate: different file contents found in the following:
[error] /Users/home/.ivy2/cache/commons-logging/commons-logging/jars/commons-logging-1.1.1.jar:org/apache/commons/logging/impl/SimpleLog.class
[error] /Users/home/.ivy2/cache/org.slf4j/jcl-over-slf4j/jars/jcl-over-slf4j-1.6.4.jar:org/apache/commons/logging/impl/SimpleLog.class
现在来自sbt-assembly文档:
如果多个文件共享相同的相对路径(例如,多个依赖项JAR中名为application.conf的资源),则默认策略是验证所有候选项具有相同的内容,否则出错.可以使用以下内置策略之一或编写自定义策略在每个路径的基础上配置此行为:
MergeStrategy.deduplicate是上面描述的默认值MergeStrategy.first选择classpath顺序中的第一个匹配文件MergeStrategy.last挑选最后一个MergeStrategy.singleOrError在冲突时出现错误消息MergeStrategy.concat简单地连接所有匹配的文件并包含结果MergeStrategy.filterDistinctLines也可以连接,但在此过程中会留下重复的内容MergeStrategy.rename重命名源自jar文件的文件MergeStrategy.discard只是丢弃匹配的文件
通过这个我设置我的build.sbt如下:
import sbt._
import Keys._
import sbtassembly.Plugin._
import AssemblyKeys._
name := "my-project"
version := "0.1"
scalaVersion := "2.9.2"
crossScalaVersions := Seq("2.9.1","2.9.2")
//assemblySettings
seq(assemblySettings: _*)
resolvers ++= Seq(
"Typesafe Releases Repository" at "http://repo.typesafe.com/typesafe/releases/",
"Typesafe Snapshots Repository" at "http://repo.typesafe.com/typesafe/snapshots/",
"Sonatype Repository" at …我正在Scala中构建一个Apache Spark应用程序,我正在使用SBT来构建它.这是事情:
sbt test,我希望Spark依赖项包含在类路径中(与#1相同,但来自SBT)为了匹配约束#2,我将Spark依赖关系声明为provided:
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
...
)
然后,sbt-assembly的文档建议添加以下行以包含单元测试的依赖项(约束#3):
run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))
这使得约束#1没有被完全填充,即我无法在IntelliJ IDEA中运行应用程序,因为没有拾取Spark依赖项.
使用Maven,我使用特定的配置文件来构建超级JAR.这样,我将Spark依赖关系声明为主要配置文件(IDE和单元测试)的常规依赖关系,同时将它们声明provided为胖JAR打包.请参阅https://github.com/aseigneurin/kafka-sandbox/blob/master/pom.xml
使用SBT实现这一目标的最佳方法是什么？
我有一个使用build.scala文件使用sbt构建的scala项目.我可以使用sbt-assembly从这个项目生成一个可执行的jar文件没有问题.现在我想生成多个可执行jar文件,其中每个文件包含一组通用的内部和外部基本依赖项以及不同的外部依赖项.
当前版本的sbt-assembly有可能吗？在maven中这很容易,因为可以在pom中定义多个配置文件,每个配置文件生成一个单独的jar,但是在sbt-assembly中你将程序集设置传递给你的项目,我还没有想出一种方法来修改设置来生成多个jar文件.
这可能听起来像一个愚蠢的问题,但我一直在互联网上搜索如何在sbt文件中注释一行.有谁知道怎么做？
我正在尝试运行像https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKafkaWordCount.scala这样的示例.我从http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html上的Spark Structured Streaming Programming指南开始.
我的代码是
package io.boontadata.spark.job1
import org.apache.spark.sql.SparkSession
object DirectKafkaAggregateEvents {
val FIELD_MESSAGE_ID = 0
val FIELD_DEVICE_ID = 1
val FIELD_TIMESTAMP = 2
val FIELD_CATEGORY = 3
val FIELD_MEASURE1 = 4
val FIELD_MEASURE2 = 5
def main(args: Array[String]) {
if (args.length < 3) {
System.err.println(s"""
|Usage: DirectKafkaAggregateEvents <brokers> <subscribeType> <topics>
| <brokers> is a list of one or more Kafka brokers
| <subscribeType> sample value: subscribe
| <topics> is a list of one or more …scala sbt sbt-assembly apache-spark spark-structured-streaming