小编avi*_*ikm的帖子

我正在尝试从 Spark 开始。我的库中有 Hadoop (3.3.1) 和 Spark (3.2.2)。我已将 SPARK_HOME、PATH、HADOOP_HOME 和 LD_LIBRARY_PATH 设置为各自的路径。我还运行 JDK 17（echo 和 -version 在终端中工作正常）。

然而，我仍然收到以下错误：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/10/25 17:17:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x1f508f09) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed …

I am looking to write docker compose file to locally execute airflow in production similar environent.

For older airflow v1.10.14, docker compose is working fine. But same docker compose is not working for latest stable version, airflow scheduler & webservice is failing continuously. error message looks like unable to create audit tables.

docker-compose.yaml:

 version: "2.1"
services:
  postgres:
    image: postgres:12
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow
    ports:
      - "5433:5432"

  scheduler:
    image: apache/airflow:1.10.14
    restart: always
    depends_on:
      - postgres
      - webserver …