如何在Spark中抑制镶木地板日志消息？

Question

如何阻止这些消息进入我的spark-shell控制台.

5 May, 2015 5:14:30 PM INFO: parquet.hadoop.InternalParquetRecordReader: at row 0. reading next block
5 May, 2015 5:14:30 PM INFO: parquet.hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 89213 records.
5 May, 2015 5:14:30 PM INFO: parquet.hadoop.InternalParquetRecordReader: block read in memory in 2 ms. row count = 120141
5 May, 2015 5:14:30 PM INFO: parquet.hadoop.InternalParquetRecordReader: at row 0. reading next block
5 May, 2015 5:14:30 PM INFO: parquet.hadoop.InternalParquetRecordReader: block read in memory in 2 ms. row count = 89213
5 May, 2015 5:14:30 PM WARNING: parquet.hadoop.ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutp
[Stage 12:=================================================>    (184 + 4) / 200]

谢谢

Answer 1

来自SPARK-8118问题评论的解决方案似乎有效:

您可以通过创建包含以下内容的属性文件来禁用聊天输出:

org.apache.parquet.handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=SEVERE

然后在提交应用程序时将文件路径传递给Spark.假设文件位于/tmp/parquet.logging.properties中(当然,需要在所有工作节点上都可用):

spark-submit \
     --conf spark.driver.extraJavaOptions="-Djava.util.logging.config.file=/tmp/parquet.logging.properties" \`
      --conf spark.executor.extraJavaOptions="-Djava.util.logging.config.file=/tmp/parquet.logging.properties" \
      ... 

积分归Justin Bailey所有.

Answer 2

要关闭除 ERROR 之外的所有消息，您应该编辑 conf/log4j.properties 文件，更改以下行：

log4j.rootCategory=INFO, console

进入

log4j.rootCategory=ERROR, console

希望它能有所帮助！