相关疑难解决方法(0)

我最近发现了一种在Apache Spark中使用logback而不是log4j的方法(两者都用于本地使用spark-submit).然而,最后一块遗失了.

问题是Spark非常努力地不在logback.xml其类路径中查看设置.我已经找到了一种在本地执行期间加载它的方法:

到目前为止我有什么

基本上,检查系统属性logback.configurationFile,但加载logback.xml从我/src/main/resources/以防万一:

// the same as default: https://logback.qos.ch/manual/configuration.html
private val LogbackLocation = Option(System.getProperty("logback.configurationFile"))
// add some default logback.xml to your /src/main/resources
private lazy val defaultLogbackConf = getClass.getResource("/logback.xml").getPath

private def getLogbackConfigPath = {
   val path = LogbackLocation.map(new File(_).getPath).getOrElse(defaultLogbackConf)
   logger.info(s"Loading logging configuration from: $path")
   path
}

然后当我初始化我的SparkContext时......

val sc = SparkContext.getOrCreate(conf)
sc.addFile(getLogbackConfigPath)

我可以确认它在本地工作.

和谁玩 spark-submit

spark-submit \
  ...
  --master yarn \
  --class com.company.Main\
  /path/to/my/application-fat.jar \ …

我对使用Apache Spark设置日志记录非常困惑.Apache spark使用Log4j进行日志记录,它会生成大量的日志数据.有没有办法为spark日志设置log4j并使用logback进行应用程序日志.我非常熟悉logback,但似乎 spark只支持log4j.下面的代码工作正常,直到我介绍了apache spark.在这方面的任何帮助都会有所帮助.

import com.typesafe.scalalogging.LazyLogging

import scala.util.{Failure, Success}
import scala.xml.{Elem, XML}



object MainApp extends App with LazyLogging  {



  val currency = new YahooCurrencyLoader() with CurrencyParameters
  val ccy = currency.getXML(currency.ccyUrl) match {

    case Success(v) => XML.save("PreviousRun.xml",v); logger.info("XML has been saved for use")
    case Failure(ex) => logger.error("XML extraction failed. Look at Yahoo extraction class. ${ex.getMessage}" )

  }

  val xmllocation: String = "./PreviousRun.xml"
  val loadxml: Elem = XML.loadFile(xmllocation)
  //print(loadxml)
  //print(currency.findCurrency(loadxml,"GBP"))

  logger.info("USD CAD Cross is " + currency.findCurrency(loadxml,"CAD").head)