气流：DockerOperator 失败并显示权限被拒绝错误

Question

我正在尝试通过 Airflow 运行 docker 容器，但Permission Denied出现错误。我看过一些相关的帖子，有些人似乎已经解决了这个sudo chmod 777 /var/run/docker.sock问题，这充其量是一个有问题的解决方案，但它仍然对我不起作用（即使在重新启动 docker 之后。如果有人设法解决了这个问题，请让我知道！

这是我的 DAG：

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.docker_operator import DockerOperator

args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2020, 6, 21, 11, 45, 0),
    'retries': 1,
    'retry_delay': timedelta(minutes=1),
}

dag = DAG(
    "docker",
    default_args=args,
    max_active_runs=1,
    schedule_interval='* * * * *',
    catchup=False
)

hello_operator = DockerOperator(
    task_id="run_docker",
    image="alpine:latest",
    command="/bin/bash echo HI!",
    auto_remove=True,
    dag=dag
)

这是我得到的错误：

[2020-06-21 14:01:36,620] {taskinstance.py:1145} ERROR - ('Connection aborted.', PermissionError(13, 'Permission denied'))
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1262, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1308, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1257, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1262, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1308, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1257, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

这是我的设置：

Dockerfile：

FROM apache/airflow
RUN pip install --upgrade --user pip && \
    pip install --user psycopg2-binary && \
    pip install --user docker
COPY airflow/airflow.cfg /opt/airflow/

docker-compose.yaml：

version: "3"

services:

  postgres:
    image: "postgres:9.6"
    container_name: "postgres"
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow
    ports:
    - "5432:5432"
    volumes:
    - ./data/postgres:/var/lib/postgresql/data

  initdb:
    image: learning/airflow
    entrypoint: airflow initdb
    depends_on:
      - postgres

  webserver:
    image: learning/airflow
    restart: always
    entrypoint: airflow webserver
    environment:
      - EXECUTOR=Local
    healthcheck:
      test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-webserver.pid ]"]
      interval: 1m
      timeout: 5m
      retries: 3
    ports:
      - "8080:8080"
    depends_on:
      - postgres
    volumes:
    - ./airflow/dags:/opt/airflow/dags
    - ./airflow/plugins:/opt/airflow/plugins
    - ./data/logs:/opt/airflow/logs
    - /var/run/docker.sock:/var/run/docker.sock

  scheduler:
    image: learning/airflow
    restart: always
    entrypoint: airflow scheduler
    healthcheck:
      test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-scheduler.pid ]"]
      interval: 1m
      timeout: 5m
      retries: 3
    depends_on:
      - postgres
    volumes:
      - ./airflow/dags:/opt/airflow/dags
      - ./airflow/plugins:/opt/airflow/plugins
      - ./data/logs:/opt/airflow/logs
      - /var/run/docker.sock:/var/run/docker.sock

Answer 1

即使知道这个问题已经很老了，我的回答仍然可以帮助其他遇到这个问题的人。

\n

我在以下链接中找到了一个优雅（且实用）的解决方案：

\n

https://onedevblog.com/how-to-fix-a-permission-denied-when-using-dockeroperator-in-airflow/

\n

引用文章：

\n

\n

有一种更优雅的方法，其中包含 \xe2\x80\x9cwrapping\xe2\x80\x9d 服务周围的文件（可通过 TCP 访问）。

\n

\n

--

\n

根据上面的链接，解决方案是：

\n

\n
• 添加附加服务以通过使用docker-proxy访问本地主机 docker ( /var/run/docker.sock) 。tcp://docker-proxy:2375socat
\n

\n

version: \'3.7\'\nservices:\n  docker-proxy:\n    image: bobrik/socat\n    command: "TCP4-LISTEN:2375,fork,reuseaddr UNIX-CONNECT:/var/run/docker.sock"\n    ports:\n      - "2376:2375"\n    volumes:\n      - /var/run/docker.sock:/var/run/docker.sock\n

\n

\n
• 将 kwarg 替换docker_url=\'unix://var/run/docker.sock\'为docker_url=\'tcp://docker-proxy:2375\'for all DockerOperators。
\n

\n

Answer 2

如果卷已映射到容器，请在主机上运行 chmod：

chmod 777 /var/run/docker.sock

为我解决了。

Answer 3

在 中添加另一个引导/至/var/run/docker.sock（位于 之前的部分的源处:）volumes，如下所示：

volumes:
    //var/run/docker.sock:/var/run/docker.sock