我想停止火花壳上的各种消息.
我试图编辑该log4j.properties文件以阻止这些消息.
这是内容 log4j.properties
# Define the root logger with appender file
log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
但是消息仍在控制台上显示.
以下是一些示例消息
15/01/05 15:11:45 INFO SparkEnv: Registering BlockManagerMaster
15/01/05 15:11:45 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150105151145-b1ba
15/01/05 15:11:45 INFO MemoryStore: MemoryStore started with capacity 0.0 B.
15/01/05 15:11:45 INFO ConnectionManager: Bound socket to port 44728 with id = ConnectionManagerId(192.168.100.85,44728)
15/01/05 …我在 cdh 5.3 集群上有一个 spark 1.2.0。
由于 jar 中捆绑了一个自定义的 log4j.properties 文件，我设法将我的 spark 应用程序日志记录到本地文件系统。这很好，直到在纱线客户端模式下启动 spark 时，但在纱线集群模式下不可行，因为无法知道驱动程序在哪台机器上运行。
我查看了纱线日志聚合器，查看了在 h 中生成的文件，dfs://nameservice1/user/spark/applicationHistory/application_1444387971657_0470/*这与普通文件系统上的文件完全不匹配，但是是这样的信息
{"Event":"SparkListenerTaskEnd","Stage ID":1314,"Stage Attempt ID":0,"Task Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task ID":3120,"Index":1,"Attempt":0,"Launch Time":1445512311024,"Executor ID":"3","Host":"usqrtpl5328.internal.unicreditgroup.eu","Locality":"RACK_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":1445512311685,"Failed":false,"Accumulables":[]},"Task Metrics":{"Host Name":"usqrtpl5328.internal.unicreditgroup.eu","Executor Deserialize Time":5,"Executor Run Time":652,"Result Size":1768,"JVM GC Time":243,"Result Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle Read Metrics":{"Remote Blocks Fetched":26,"Local Blocks Fetched":10,"Fetch Wait Time":0,"Remote Bytes Read":16224},"Output Metrics":{"Data Write Method":"Hadoop","Bytes Written":82983}}}
现在有没有办法将所有内容以及我想要的内容记录到 HDFS 中？
欢迎任何建议
编辑我在发布我的问题时看到过这个问题。它不能解决我的问题，因为我需要登录到 HDFS 而没有考虑到这一点。
我什至不知道是否可以直接使用 log4j 登录到 HDFS，如果您对如何相应地编写 log4j.properties 有任何想法，请分享