小编Rax*_*ore的帖子

Spark-Hadoop-> org.apache.hadoop.mapred.InvalidInputException:输入路径不存在

我在尝试将文件从hdfs读入Spark时遇到错误.文件README.md存在于hdfs中

 spark@osboxes hadoop]$ hdfs dfs -ls README.md
16/02/26 00:29:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-rw-r--r--   1 spark supergroup       4811 2016-02-25 23:38 README.md
Run Code Online (Sandbox Code Playgroud)

在Spark shell中,我给了

scala> val readme = sc.textFile("hdfs://localhost:9000/README.md")
readme: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at textFile at <console>:27

scala> readme.count
16/02/26 00:25:26 DEBUG BlockManager: Getting local block broadcast_4
16/02/26 00:25:26 DEBUG BlockManager: Level for block broadcast_4 is StorageLevel(true, true, false, true, 1)
16/02/26 00:25:26 DEBUG BlockManager: Getting block broadcast_4 from memory …
Run Code Online (Sandbox Code Playgroud)

hadoop apache-spark

6
推荐指数
2
解决办法
2万
查看次数

Akka Stream:无法写入文件接收器

我试图运行一个简单的Akka流文件接收器示例但没有成功.我可以创建一个Source,运行Flow然后创建一个文件但ByteString没有写入文件.然而,如果我尝试将流输出打印到控制台,我可以这样做.我在这里错过了什么吗?

import akka.stream._ 
import akka.stream.scaladsl._
import akka.{ NotUsed, Done}
import akka.actor.ActorSystem
import akka.util.ByteString
import scala.concurrent._
import scala.concurrent.duration._
import java.nio.file.Paths

object First extends App {

  val source: Source[Int, NotUsed] = Source ( 1 to 100)

  implicit val system = ActorSystem("QuickStart")
  implicit val materializer = ActorMaterializer()

  // works: prints 1-100
  //source.runForeach(println) (materializer)

  val factorials = source.scan(BigInt(1))((acc,next) => acc * next)

  // there is no content in the Sink (file)
  /**val result =
    factorials
    .map(num => ByteString(s"${num}\n"))
    .runWith(FileIO.toPath(Paths.get("factorials.txt")))
**/

  def lineSink(fileName: String): Sink[String, …
Run Code Online (Sandbox Code Playgroud)

scala akka akka-stream

1
推荐指数
1
解决办法
3760
查看次数

Brew 命令抛出 github 相关错误

我的所有 brew 命令(更新、配置、docker 等)都在我的 Mac v10.14.2 上抛出以下错误:

brew update
fatal: could not read Username for 'https://github.com': terminal prompts disabled
fatal: could not read Username for 'https://github.com': terminal prompts disabled
Error: homebrew/homebrew-dupes does not exist! Run 'brew untap homebrew/homebrew-dupes'
homebrew/homebrew-x11 does not exist! Run 'brew untap homebrew/homebrew-x11' 
Run Code Online (Sandbox Code Playgroud)

我已经尝试了以下选项:
1. brew untap homebrew/homebrew-x11 -> 给出与报告相同的错误。
2. brew untap homebrew/homebrew-dupes -> 给出与报告相同的错误。
3. brew resintall 失败,错误与报告相同Failed during: /usr/local/bin/brew update --force

~/.bash_profile 中的 brew 相关条目:
alias brew='brew update && brew'

关于如何摆脱这个问题的任何建议?

homebrew homebrew-cask

1
推荐指数
1
解决办法
1981
查看次数