我在Amazon Elastic MapReduce上运行Spark和Spark SQL中描述的在EMR上运行Spark:
本教程将指导您在Amazon EMR集群上安装和运行Spark,这是一种用于大规模数据处理的快速通用引擎.您还将使用Spark SQL在Amazon S3中创建和查询数据集,并了解如何使用Amazon CloudWatch监控Amazon EMR集群上的Spark.
我试图INFO通过编辑$HOME/spark/conf/log4j.properties来抑制日志无济于事.
输出如下:
$ ./spark/bin/spark-sql
Spark assembly has been built with Hive, including Datanucleus jars on classpath
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/.versions/2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/.versions/spark-1.1.1.e/lib/spark-assembly-1.1.1-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2014-12-14 20:59:01,819 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
2014-12-14 20:59:01,825 INFO …