我正在 Kubernetes 上使用 Spark-submit cli 运行 Spark 3.0.0 和 Hadoop 2.7，如下所示，
./spark-submit \
--master=k8s://https://api.k8s.my-domain.com \
--deploy-mode cluster \
--name sparkle \
--num-executors 2 \
--executor-cores 2 \
--executor-memory 2g \
--driver-memory 2g \
--class com.myorg.sparkle.Sparkle \
--conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/opt/spark/conf/log4j.properties \
--conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/opt/spark/conf/log4j.properties \
--conf spark.kubernetes.submission.waitAppCompletion=false \
--conf spark.kubernetes.allocation.batch.delay=10s \
--conf spark.kubernetes.appKillPodDeletionGracePeriod=20s \
--conf spark.kubernetes.node.selector.workloadType=spark \
--conf spark.kubernetes.driver.pod.name=sparkle-driver \
--conf spark.kubernetes.container.image=custom-registry/spark:latest \
--conf spark.kubernetes.namespace=spark \
--conf spark.eventLog.dir='s3a://my-bucket/spark-logs' \
--conf spark.history.fs.logDirectory='s3a://my-bucket/spark-logs' \
--conf spark.eventLog.enabled='true' \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.authenticate.executor.serviceAccountName=spark \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem …我正在使用 Spark 3.0.0 和 Kubernetes master。我正在使用集群模式来运行 Spark 作业。请找到如下spark提交命令
./spark-submit \
--master=k8s://https://api.k8s.my-domain.com \
--deploy-mode cluster \
--name sparkle \
--num-executors 2 \
--executor-cores 2 \
--executor-memory 2g \
--driver-memory 2g \
--class com.myorg.sparkle.Sparkle \
--conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/opt/spark/conf/log4j.properties \
--conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/opt/spark/conf/log4j.properties \
--conf spark.kubernetes.submission.waitAppCompletion=false \
--conf spark.kubernetes.allocation.batch.delay=10s \
--conf spark.kubernetes.appKillPodDeletionGracePeriod=20s \
--conf spark.kubernetes.node.selector.workloadType=spark \
--conf spark.kubernetes.driver.pod.name=sparkle-driver \
--conf spark.kubernetes.container.image=custom-registry/spark:latest \
--conf spark.kubernetes.namespace=spark \
--conf spark.eventLog.dir='s3a://my-bucket/spark-logs' \
--conf spark.history.fs.logDirectory='s3a://my-bucket/spark-logs' \
--conf spark.eventLog.enabled='true' \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.authenticate.executor.serviceAccountName=spark \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider …