Sad*_*had 5 docker docker-compose airflow apple-m1
我正在尝试在 docker 中设置 Apache Airflow，以进行本地开发。我让它在基于 Intel 的 Mac 上运行。
\n我正在关注他们的官方文档。当我尝试运行服务时，docker compose up我看到以下回溯。
# docker-compose.yaml\n---\nversion: \'3\'\nx-airflow-common:\n &airflow-common\n image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.0}\n environment:\n &airflow-common-env\n AIRFLOW__CORE__EXECUTOR: CeleryExecutor\n AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow\n AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow\n AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0\n AIRFLOW__CORE__FERNET_KEY: \'\'\n AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: \'true\'\n AIRFLOW__CORE__LOAD_EXAMPLES: \'true\'\n AIRFLOW__API__AUTH_BACKEND: \'airflow.api.auth.backend.basic_auth\'\n volumes:\n - ./dags:/opt/airflow/dags\n - ./logs:/opt/airflow/logs\n - ./plugins:/opt/airflow/plugins\n user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"\n depends_on:\n redis:\n condition: service_healthy\n postgres:\n condition: service_healthy\n\nservices:\n postgres:\n image: postgres:13\n# platform: linux/amd64\n environment:\n POSTGRES_USER: airflow\n POSTGRES_PASSWORD: airflow\n POSTGRES_DB: airflow\n volumes:\n - postgres-db-volume:/var/lib/postgresql/data\n healthcheck:\n test: ["CMD", "pg_isready", "-U", "airflow"]\n interval: 5s\n retries: 5\n restart: always\n\n redis:\n image: redis:latest\n ports:\n - 6379:6379\n healthcheck:\n test: ["CMD", "redis-cli", "ping"]\n interval: 5s\n timeout: 30s\n retries: 50\n restart: always\n\n airflow-webserver:\n <<: *airflow-common\n command: webserver\n ports:\n - 8080:8080\n healthcheck:\n test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]\n interval: 10s\n timeout: 10s\n retries: 5\n restart: always\n\n airflow-scheduler:\n <<: *airflow-common\n command: scheduler\n healthcheck:\n test: ["CMD-SHELL", \'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"\']\n interval: 10s\n timeout: 10s\n retries: 5\n restart: always\n\n airflow-worker:\n <<: *airflow-common\n command: celery worker\n healthcheck:\n test:\n - "CMD-SHELL"\n - \'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"\'\n interval: 10s\n timeout: 10s\n retries: 5\n restart: always\n\n airflow-init:\n <<: *airflow-common\n command: version\n environment:\n <<: *airflow-common-env\n _AIRFLOW_DB_UPGRADE: \'true\'\n _AIRFLOW_WWW_USER_CREATE: \'true\'\n _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}\n _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}\n\n flower:\n <<: *airflow-common\n command: celery flower\n ports:\n - 5555:5555\n healthcheck:\n test: ["CMD", "curl", "--fail", "http://localhost:5555/"]\n interval: 10s\n timeout: 10s\n retries: 5\n restart: always\n\nvolumes:\n postgres-db-volume:\n这是回溯docker compose up
[+] Running 7/7\n \xe2\xa0\xbf Container test_redis_1 Recreated 1.2s\n \xe2\xa0\xbf Container test_postgres_1 Recreated 0.2s\n \xe2\xa0\xbf Container test_airflow-worker_1 Created 0.1s\n \xe2\xa0\xbf Container test_airflow-init_1 Recreated 0.2s\n \xe2\xa0\xbf Container test_airflow-webserver_1 Created 0.1s\n \xe2\xa0\xbf Container test_flower_1 Created 0.1s\n \xe2\xa0\xbf Container test_airflow-scheduler_1 Created 0.1s\nAttaching to airflow-init_1, airflow-scheduler_1, airflow-webserver_1, airflow-worker_1, flower_1, postgres_1, redis_1\npostgres_1 | \npostgres_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization\npostgres_1 | \npostgres_1 | 2021-06-05 19:35:20.148 UTC [1] LOG: starting PostgreSQL 13.3 (Debian 13.3-1.pgdg100+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit\npostgres_1 | 2021-06-05 19:35:20.148 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432\npostgres_1 | 2021-06-05 19:35:20.148 UTC [1] LOG: listening on IPv6 address "::", port 5432\npostgres_1 | 2021-06-05 19:35:20.150 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"\npostgres_1 | 2021-06-05 19:35:20.154 UTC [27] LOG: database system was shut down at 2021-06-05 19:35:16 UTC\npostgres_1 | 2021-06-05 19:35:20.158 UTC [1] LOG: database system is ready to accept connections\nredis_1 | 1:C 05 Jun 2021 19:35:21.647 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo\nredis_1 | 1:C 05 Jun 2021 19:35:21.647 # Redis version=6.2.4, bits=64, commit=00000000, modified=0, pid=1, just started\nredis_1 | 1:C 05 Jun 2021 19:35:21.647 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf\nredis_1 | 1:M 05 Jun 2021 19:35:21.647 * monotonic clock: POSIX clock_gettime\nredis_1 | 1:M 05 Jun 2021 19:35:21.647 * Running mode=standalone, port=6379.\nredis_1 | 1:M 05 Jun 2021 19:35:21.647 # Server initialized\nredis_1 | 1:M 05 Jun 2021 19:35:21.649 * Ready to accept connections\nairflow-init_1 | BACKEND=postgresql+psycopg2\nairflow-init_1 | DB_HOST=postgres\nairflow-init_1 | DB_PORT=5432\nairflow-init_1 | \nairflow-scheduler_1 | BACKEND=postgresql+psycopg2\nairflow-scheduler_1 | DB_HOST=postgres\nairflow-scheduler_1 | DB_PORT=5432\nairflow-scheduler_1 | \nairflow-scheduler_1 | BACKEND=postgresql+psycopg2\nairflow-scheduler_1 | DB_HOST=postgres\nairflow-scheduler_1 | DB_PORT=5432\nairflow-scheduler_1 | \nairflow-worker_1 | BACKEND=postgresql+psycopg2\nairflow-worker_1 | DB_HOST=postgres\nairflow-worker_1 | DB_PORT=5432\nflower_1 | BACKEND=postgresql+psycopg2\nflower_1 | DB_HOST=postgres\nflower_1 | DB_PORT=5432\nairflow-worker_1 | \nairflow-worker_1 | BACKEND=postgresql+psycopg2\nairflow-worker_1 | DB_HOST=postgres\nairflow-worker_1 | DB_PORT=5432\nflower_1 | \nflower_1 | BACKEND=postgresql+psycopg2\nflower_1 | DB_HOST=postgres\nflower_1 | DB_PORT=5432\nairflow-webserver_1 | BACKEND=postgresql+psycopg2\nairflow-webserver_1 | DB_HOST=postgres\nairflow-webserver_1 | DB_PORT=5432\nairflow-worker_1 | \nflower_1 | \nairflow-webserver_1 | \nairflow-init_1 | DB: postgresql+psycopg2://airflow:***@postgres/airflow\nairflow-init_1 | [2021-06-05 19:35:59,163] {db.py:695} INFO - Creating tables\nairflow-init_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.\nairflow-init_1 | INFO [alembic.runtime.migration] Will assume transactional DDL.\nflower_1 | [2021-06-05 19:36:08,037] {command.py:137} INFO - Visit me at http://0.0.0.0:5555\nflower_1 | [2021-06-05 19:36:08,359] {command.py:142} INFO - Broker: redis://redis:6379/0\nflower_1 | [2021-06-05 19:36:08,391] {command.py:145} INFO - Registered tasks: \nflower_1 | [\'airflow.executors.celery_executor.execute_command\',\nflower_1 | \'celery.accumulate\',\nflower_1 | \'celery.backend_cleanup\',\nflower_1 | \'celery.chain\',\nflower_1 | \'celery.chord\',\nflower_1 | \'celery.chord_unlock\',\nflower_1 | \'celery.chunks\',\nflower_1 | \'celery.group\',\nflower_1 | \'celery.map\',\nflower_1 | \'celery.starmap\']\nflower_1 | [2021-06-05 19:36:08,666] {mixins.py:229} INFO - Connected to redis://redis:6379/0\nflower_1 | [2021-06-05 19:36:11,593] {inspector.py:42} WARNING - Inspect method scheduled failed\nflower_1 | [2021-06-05 19:36:11,609] {inspector.py:42} WARNING - Inspect method conf failed\nflower_1 | [2021-06-05 19:36:11,617] {inspector.py:42} WARNING - Inspect method reserved failed\nflower_1 | [2021-06-05 19:36:11,619] {inspector.py:42} WARNING - Inspect method registered failed\nflower_1 | [2021-06-05 19:36:11,655] {inspector.py:42} WARNING - Inspect method active_queues failed\nflower_1 | [2021-06-05 19:36:11,659] {inspector.py:42} WARNING - Inspect method stats failed\nflower_1 | [2021-06-05 19:36:11,662] {inspector.py:42} WARNING - Inspect method revoked failed\nflower_1 | [2021-06-05 19:36:11,664] {inspector.py:42} WARNING - Inspect method active failed\nairflow-scheduler_1 | ____________ _____________\nairflow-scheduler_1 | ____ |__( )_________ __/__ /________ __\nairflow-scheduler_1 | ____ /| |_ /__ ___/_ /_ __ /_ __ \\_ | /| / /\nairflow-scheduler_1 | ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /\nairflow-scheduler_1 | _/_/ |_/_/ /_/ /_/ /_/ \\____/____/|__/\nairflow-scheduler_1 | [2021-06-05 19:36:19,515] {scheduler_job.py:1253} INFO - Starting the scheduler\nairflow-scheduler_1 | [2021-06-05 19:36:19,527] {scheduler_job.py:1258} INFO - Processing each file at most -1 times\nairflow-scheduler_1 | [2021-06-05 19:36:19,588] {dag_processing.py:254} INFO - Launched DagFileProcessorManager with pid: 107\nairflow-scheduler_1 | [2021-06-05 19:36:19,604] {scheduler_job.py:1822} INFO - Resetting orphaned tasks for active dag runs\nairflow-scheduler_1 | [2021-06-05 19:36:19,657] {settings.py:52} INFO - Configured default timezone Timezone(\'UTC\')\nairflow-worker_1 | Starting flask\nairflow-worker_1 | * Serving Flask app "airflow.utils.serve_logs" (lazy loading)\nairflow-worker_1 | * Environment: production\nairflow-worker_1 | WARNING: This is a development server. Do not use it in a production deployment.\nairflow-worker_1 | Use a production WSGI server instead.\nairflow-worker_1 | * Debug mode: off\nairflow-worker_1 | [2021-06-05 19:36:28,740] {_internal.py:113} INFO - * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)\nairflow-init_1 | Upgrades done\nairflow-worker_1 | /home/airflow/.local/lib/python3.6/site-packages/celery/platforms.py:801 RuntimeWarning: You\'re running the worker with superuser privileges: this is\nairflow-worker_1 | absolutely not recommended!\nairflow-worker_1 | \nairflow-worker_1 | Please specify a different user using the --uid option.\nairflow-worker_1 | \nairflow-worker_1 | User information: uid=501 euid=501 gid=0 egid=0\nairflow-worker_1 | \nairflow-webserver_1 | ____________ _____________\nairflow-webserver_1 | ____ |__( )_________ __/__ /________ __\nairflow-webserver_1 | ____ /| |_ /__ ___/_ /_ __ /_ __ \\_ | /| / /\nairflow-webserver_1 | ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /\nairflow-webserver_1 | _/_/ |_/_/ /_/ /_/ /_/ \\____/____/|__/\nairflow-webserver_1 | [2021-06-05 19:36:32,432] {dagbag.py:487} INFO - Filling up the DagBag from /dev/null\npostgres_1 | 2021-06-05 19:36:38.577 UTC [150] LOG: unexpected EOF on client connection with an open transaction\nairflow-worker_1 | [2021-06-05 19:36:39,670: INFO/MainProcess] Connected to redis://redis:6379/0\nairflow-worker_1 | [2021-06-05 19:36:39,819: INFO/MainProcess] mingle: searching for neighbors\nairflow-webserver_1 exited with code 137\nairflow-worker_1 | [2021-06-05 19:36:41,084: INFO/MainProcess] mingle: all alone\nairflow-worker_1 | [2021-06-05 19:36:41,196: INFO/MainProcess] celery@a2177ac7a506 ready.\npostgres_1 | 2021-06-05 19:36:43.263 UTC [131] LOG: unexpected EOF on client connection with an open transaction\nairflow-worker_1 | [2021-06-05 19:36:43,502: INFO/MainProcess] Events of group {task} enabled by remote.\npostgres_1 | 2021-06-05 19:36:48.829 UTC [132] LOG: unexpected EOF on client connection with an open transaction\nflower_1 exited with code 137\nairflow-worker_1 exited with code 137\n然后它进入循环重试。在一个实验中，当我尝试一项一项地运行服务时，我发现一旦启动，airflow-worker所有其他服务就开始失败。airflow-webservice工人立即死亡并尝试再次重生但不能。
| 归档时间： |
|
| 查看次数： |
7137 次 |
| 最近记录： |