Docker Airflow - 错误:无法执行“--user”安装。用户站点包在此 virtualenv 中不可见

sog*_*ogu 3 python pip docker dockerfile airflow

目标

  • 自2022年9月19日起 Apache Airflow 2.4.0 发布
  • Airflow 支持ExternalPythonOperator
  • 我也询问了主要贡献者,我应该能够将 2 个 python 虚拟环境添加到 Airflow Docker 2.4.1 的基础映像中,并能够在 DAG 内运行单个任务。
  • 我的目标是使用从本地requirements.txt构建的多个主机python virtualenvs。
  • 使用ExternalPythonOperator来运行它们(我的每个dags只执行一个定时的python函数)

代码

Dockerfile

FROM apache/airflow:2.4.1-python3.8
RUN python3 -m venv /opt/airflow/venv1
COPY requirements.txt .
RUN . /opt/airflow/venv1/bin/activate && pip install -r requirements.txt

Run Code Online (Sandbox Code Playgroud)

端子输入

docker build -t my-image-apache/airflow:2.4.1 .

端子输出

[+] Building 4.3s (9/9) FINISHED                                                                                                                
 => [internal] load build definition from Dockerfile                                                                                       0.0s
 => => transferring dockerfile: 1.55kB                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                          0.0s
 => => transferring context: 2B                                                                                                            0.0s
 => [internal] load metadata for docker.io/apache/airflow:2.4.1-python3.8                                                                  1.2s
 => [auth] apache/airflow:pull token for registry-1.docker.io                                                                              0.0s
 => CACHED [1/4] FROM docker.io/apache/airflow:2.4.1-python3.8@sha256:5f9f4eff86993e11893f371f591aed73cf2310a96d84ae8fddec11857c6345da     0.0s
 => [internal] load build context                                                                                                          0.0s
 => => transferring context: 37B                                                                                                           0.0s
 => [2/4] RUN python3 -m venv /opt/airflow/venv1                                                                                           2.2s
 => [3/4] COPY requirements.txt .                                                                                                          0.0s
 => ERROR [4/4] RUN . /opt/airflow/venv1/bin/activate && pip install -r requirements.txt                                                   0.8s
------                                                                                                                                          
 > [4/4] RUN . /opt/airflow/venv1/bin/activate && pip install -r requirements.txt:
#9 0.621 ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
#9 0.763 WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
#9 0.763 You should consider upgrading via the '/opt/airflow/venv1/bin/python3 -m pip install --upgrade pip' command.
------
executor failed running [/bin/bash -o pipefail -o errexit -o nounset -o nolog -c . /opt/airflow/venv1/bin/activate && pip install -r requirements.txt]: exit code: 1
Run Code Online (Sandbox Code Playgroud)

尝试过的解决方案

FROM apache/airflow:2.4.1-python3.8

ENV VIRTUAL_ENV=/opt/airflow/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# Install dependencies:
COPY requirements.txt .
RUN pip install -r requirements.txt
Run Code Online (Sandbox Code Playgroud)

与上面相同的错误

FROM apache/airflow:2.4.1-python3.8
ADD . /opt/airflow/
WORKDIR /opt/airflow/

RUN python -m venv venv
RUN venv/bin/pip install --upgrade pip
RUN venv/bin/pip install -r requirements.txt

Run Code Online (Sandbox Code Playgroud)

与上面相同的错误

sog*_*ogu 7

Dockerfile [正确]

FROM apache/airflow:2.4.1-python3.8

# Compulsory to switch parameter
ENV PIP_USER=false

#python venv setup
RUN python3 -m venv /opt/airflow/venv1

# Install dependencies:
COPY requirements.txt .

# --user   <--- WRONG, this is what ENV PIP_USER=false turns off
#RUN /opt/airflow/venv1/bin/pip install --user -r requirements.txt  <---this is all wrong
RUN /opt/airflow/venv1/bin/pip install -r requirements.txt
ENV PIP_USER=true
Run Code Online (Sandbox Code Playgroud)