Apache Spark中是否重新提交失败的任务？

Question

失败的任务是否自动在Apache Spark中重新提交给相同或另一个执行者？

Answer 1

I believe failed tasks are resubmitted because I have seen the same failed task submitted multiple times on the Web UI. However, if the same task fails multiple times, the full job fail:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 120 in stage 91.0 failed 4 times, most recent failure: Lost task 120.3 in stage 91.0

Answer 2

是的，但是为最大失败次数设置了一个参数

spark.task.maxFailures  4   Number of individual task failures before giving up on the job. Should be greater than or equal to 1. Number of allowed retries = this value - 1.