由于java.lang.OutOfMemoryError:Java堆空间,我的spark工作失败了.我试图玩弄配置PARAMS像executor-cores,executor-memory,num-executors,driver-cores,driver-memory,spark.yarn.driver.memoryOverhead,spark.yarn.executor.memoryOverhead根据RAMZY的答案.以下是我的配置集
--master yarn-cluster --executor-cores 4 --executor-memory 10G --num-executors 30 --driver-cores 4 --driver-memory 16G --queue team_high --conf spark.eventLog.dir=hdfs:///spark-history --conf spark.eventLog.enabled=true --conf spark.yarn.historyServer.address=xxxxxxxxx:xxxx --conf spark.sql.tungsten.enabled=true --conf spark.ui.port=5051 --conf spark.sql.shuffle.partitions=30 --conf spark.yarn.driver.memoryOverhead=1024 --conf spark.yarn.executor.memoryOverhead=1400 --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.sql.orc.filterPushdown=true --conf spark.scheduler.mode=FAIR --conf hive.exec.dynamic.partition=false --conf hive.exec.dynamic.partition.mode=nonstrict --conf mapreduce.fileoutputcommitter.algorithm.version=2 --conf orc.stripe.size=67108864 --conf hive.merge.orcfile.stripe.level=true --conf hive.merge.smallfiles.avgsize=2560000 --conf hive.merge.size.per.task=2560000 --conf spark.driver.extraJavaOptions='-XX:+UseCompressedOops -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps' --conf spark.executor.extraJavaOptions='-XX:+UseCompressedOops -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC' …