Spark:如何从spark shell运行spark文件

Question

我正在使用CDH 5.2.我可以使用spark-shell来运行命令.

1. 如何运行包含spark命令的文件(file.spark).
2. 有没有办法在没有sbt的情况下在CDH 5.2中运行/编译scala程序？

提前致谢

Answer 1

在命令行中,您可以使用

spark-shell -i file.scala

运行写入的代码 file.scala

Answer 2

要从spark-shell加载外部文件,请执行此操作

:load PATH_TO_FILE

这将调用文件中的所有内容.

我对你的SBT问题没有解决方案,但抱歉:-)

Answer 3

您可以使用sbt或maven来编译spark程序.只需将spark作为依赖添加到maven

<repository>
      <id>Spark repository</id>
      <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
</repository>

然后依赖:

<dependency>
      <groupId>spark</groupId>
      <artifactId>spark</artifactId>
      <version>1.2.0</version>
</dependency>

在使用spark命令运行文件方面:您可以简单地执行此操作:

echo"
   import org.apache.spark.sql.*
   ssc = new SQLContext(sc)
   ssc.sql("select * from mytable").collect
" > spark.input

现在运行命令脚本:

cat spark.input | spark-shell

Answer 4

spark-shell version 1.6.3在和上进行了测试spark2-shell version 2.3.0.2.6.5.179-4，您可以直接通过管道连接到 shell 的标准输入，例如

spark-shell <<< "1+1"

或者在您的用例中，

spark-shell < file.spark

Answer 5

只是为了给出更多的答案

Spark-shell是一个scala repl

您可以键入:help以查看scala shell中可能的操作列表

scala> :help
All commands can be abbreviated, e.g., :he instead of :help.
:edit <id>|<line>        edit history
:help [command]          print this summary or command-specific help
:history [num]           show the history (optional num is commands to show)
:h? <string>             search the history
:imports [name name ...] show import history, identifying sources of names
:implicits [-v]          show the implicits in scope
:javap <path|class>      disassemble a file or class name
:line <id>|<line>        place line(s) at the end of history
:load <path>             interpret lines in a file
:paste [-raw] [path]     enter paste mode or paste a file
:power                   enable power user mode
:quit                    exit the interpreter
:replay [options]        reset the repl and replay all previous commands
:require <path>          add a jar to the classpath
:reset [options]         reset the repl to its initial state, forgetting all session entries
:save <path>             save replayable session to a file
:sh <command line>       run a shell command (result is implicitly => List[String])
:settings <options>      update compiler options, if possible; see reset
:silent                  disable/enable automatic printing of results
:type [-v] <expr>        display the type of an expression without evaluating it
:kind [-v] <expr>        display the kind of expression's type
:warnings                show the suppressed warnings from the most recent line which had any

:加载文件中的解释行