我已经尝试升级到Apache Spark 1.6.0 RC3.我的应用程序现在几乎每个任务都会发现这些错误:
Managed memory leak detected; size = 15735058 bytes, TID = 830
我已经设置日志记录级别org.apache.spark.memory.TaskMemoryManager来DEBUG看看在日志中:
I2015-12-18 16:54:41,125 TaskSetManager: Starting task 0.0 in stage 7.0 (TID 6, localhost, partition 0,NODE_LOCAL, 3026 bytes)
I2015-12-18 16:54:41,125 Executor: Running task 0.0 in stage 7.0 (TID 6)
I2015-12-18 16:54:41,130 ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
I2015-12-18 16:54:41,130 ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
D2015-12-18 16:54:41,188 TaskMemoryManager: Task 6 acquire 5.0 MB for null
I2015-12-18 16:54:41,199 …apache-spark ×1