如何在单元测试中抑制Spark记录？

Question

所以感谢我试过的易于谷歌的博客:

import org.specs2.mutable.Specification

class SparkEngineSpecs extends Specification {
  sequential

  def setLogLevels(level: Level, loggers: Seq[String]): Map[String, Level] = loggers.map(loggerName => {
    val logger = Logger.getLogger(loggerName)
    val prevLevel = logger.getLevel
    logger.setLevel(level)
    loggerName -> prevLevel
  }).toMap

  setLogLevels(Level.WARN, Seq("spark", "org.eclipse.jetty", "akka"))

  val sc = new SparkContext(new SparkConf().setMaster("local").setAppName("Test Spark Engine"))

  // ... my unit tests

但不幸的是它不起作用,我仍然得到很多火花输出,例如:

14/12/02 12:01:56 INFO MemoryStore: Block broadcast_4 of size 4184 dropped from memory (free 583461216)
14/12/02 12:01:56 INFO ContextCleaner: Cleaned broadcast 4
14/12/02 12:01:56 INFO ContextCleaner: Cleaned shuffle 4
14/12/02 12:01:56 INFO ShuffleBlockManager: Deleted all files for shuffle 4

Answer 1

将以下代码添加到dir log4j.properties内src/test/resources的文件中,如果不存在则创建文件/ dir

# Change this to set Spark log level
log4j.logger.org.apache.spark=WARN

# Silence akka remoting
log4j.logger.Remoting=WARN

# Ignore messages below warning level from Jetty, because it's a bit verbose
log4j.logger.org.eclipse.jetty=WARN

当我运行我的单元测试时(我正在使用JUnit和Maven),我只接收WARN级别日志,换句话说,不再使用INFO级别日志(虽然它们在调试时很有用).

我希望这有帮助.

Answer 2

在我的情况下,我自己的一个库带来了logback-classic.这在开始时的警告中实现:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/alex/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/alex/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

我通过从依赖项中排除它来解决这个问题:

"com.mystuff" % "mylib" % "1.0.0" exclude("ch.qos.logback", "logback-classic")

现在我可以添加一个现在被Spark使用的log4j.properties文件test/resources.

Answer 3

经过一段时间与Spark日志输出的挣扎,我发现了一篇博文,其中包含我特别喜欢的解决方案.

如果使用slf4j,可以简单地交换底层日志实现.测试范围的一个好的结果是slf4j-nop,它巧妙地将日志输出放在太阳永不闪耀的地方.

使用Maven时,您可以将以下内容添加到依赖项列表的顶部:

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-api</artifactId>
  <version>1.7.12</version>
  <scope>provided</scope>
</dependency>

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-nop</artifactId>
  <version>1.7.12</version>
  <scope>test</scope>
</dependency>

请注意,将它放在依赖项列表的开头可能很重要,以确保使用给定的实现而不是其他包可能带来的实现(并且您可以考虑排除这些实现以保持类路径整洁并避免意外冲突).