如何正确缓存Python依赖项

Question

我正在尝试在Dockerfile中构建一个Python项目。我想缓存依赖项，然后在以后使用该缓存，如下所示：

RUN pip3 download -d "/pth/to/downloaded/files" -r /temp/requirements.txt -c /temp/constraints.txt

# much later on in the Dockerfile:

RUN pip3 install --download-cache="/pth/to/downloaded/files" -r requirements.txt -c constraints.txt

---

问题：假设pip3 download命令正确，--download-cache在查看pip3 --help输出时不再看到选项-是否可以使用新选项pip3 install来引用由生成的依赖项缓存pip3 download？

现在，我收到此错误：

用法：pip install [选项]

没有这样的选项：--download-cache

Answer 1

该--download-cache选项已在pip版本8中删除，因为默认情况下它现在使用缓存。因此，您根本不需要指定此选项。我不确定该pip download -d <dir>选项的目的是什么，但是显然它没有在目标目录中创建缓存。您也可以忽略该-d <dir>选项。

以下Dockerfile可以工作：

FROM python:3.7
COPY constraints.txt requirements.txt ./
RUN pip3 download -d .pipcache -r requirements.txt -c constraints.txt
COPY test.txt ./
RUN pip3 install -r requirements.txt -c constraints.txt

如果添加--cache-dir <dir>到都在download和install命令，它也能发挥作用。因此，以下Dockerfile也适用：

FROM python:3.7
COPY constraints.txt requirements.txt ./
RUN pip3 download --cache-dir ./tmp/pipcache -r requirements.txt -c constraints.txt
COPY test.txt ./
RUN pip3 install --cache-dir ./tmp/pipcache -r requirements.txt -c constraints.txt

输出示例（仅在pep8和pylint中requirements.txt）：

首轮：

Sending build context to Docker daemon  5.632kB
Step 1/5 : FROM python:3.7
 ---> a4cc999cf2aa
Step 2/5 : COPY constraints.txt requirements.txt ./
 ---> 411eaa3d36ff
Step 3/5 : RUN pip3 download -r requirements.txt -c constraints.txt
 ---> Running in 6b489df74137
Collecting pep8==1.7.1 (from -c constraints.txt (line 17))
  Downloading https://files.pythonhosted.org/packages/42/3f/669429ce58de2c22d8d2c542752e137ec4b9885fff398d3eceb1a7f5acb4/pep8-1.7.1-py2.py3-none-any.whl (41kB)
  Saved /pep8-1.7.1-py2.py3-none-any.whl
Collecting pylint==2.3.1 (from -c constraints.txt (line 22))
  Downloading https://files.pythonhosted.org/packages/60/c2/b3f73f4ac008bef6e75bca4992f3963b3f85942e0277237721ef1c151f0d/pylint-2.3.1-py3-none-any.whl (765kB)
  Saved /pylint-2.3.1-py3-none-any.whl
Collecting mccabe==0.6.1 (from -c constraints.txt (line 14))
  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
  Saved /mccabe-0.6.1-py2.py3-none-any.whl
Collecting astroid==2.2.5 (from -c constraints.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/d5/ad/7221a62a2dbce5c3b8c57fd18e1052c7331adc19b3f27f1561aa6e620db2/astroid-2.2.5-py3-none-any.whl (193kB)
  Saved /astroid-2.2.5-py3-none-any.whl
Collecting isort==4.3.19 (from -c constraints.txt (line 10))
  Downloading https://files.pythonhosted.org/packages/ae/ae/5ef4b57e15489754b73dc908b656b02ab0e6d37b190ac78dd498be8b577d/isort-4.3.19-py2.py3-none-any.whl (42kB)
  Saved /isort-4.3.19-py2.py3-none-any.whl
Collecting lazy-object-proxy==1.4.1 (from -c constraints.txt (line 12))
  Downloading https://files.pythonhosted.org/packages/43/a5/1b19b094ad19bce55b5b6d434020f5537b424fd2b3cff0fbef23d7bb5a95/lazy_object_proxy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (49kB)
  Saved /lazy_object_proxy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting wrapt==1.11.1 (from -c constraints.txt (line 39))
  Downloading https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz
  Saved /wrapt-1.11.1.tar.gz
Collecting six==1.12.0 (from -c constraints.txt (line 28))
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
  Saved /six-1.12.0-py2.py3-none-any.whl
Collecting typed-ast==1.3.5 (from -c constraints.txt (line 37))
  Downloading https://files.pythonhosted.org/packages/17/9e/00918af7bdd616decb5b7ad06a9cd0a4a247d2fccaa630ab448a57e68b98/typed_ast-1.3.5-cp37-cp37m-manylinux1_x86_64.whl (736kB)
  Saved /typed_ast-1.3.5-cp37-cp37m-manylinux1_x86_64.whl
Successfully downloaded pep8 pylint mccabe astroid isort lazy-object-proxy wrapt six typed-ast
Removing intermediate container 6b489df74137
 ---> 8ac3be432c58
Step 4/5 : COPY test.txt ./
 ---> 5cac20851967
Step 5/5 : RUN pip3 install -r requirements.txt -c constraints.txt
 ---> Running in 394847f09e9b
Collecting pep8==1.7.1 (from -c constraints.txt (line 17))
  Using cached https://files.pythonhosted.org/packages/42/3f/669429ce58de2c22d8d2c542752e137ec4b9885fff398d3eceb1a7f5acb4/pep8-1.7.1-py2.py3-none-any.whl
Collecting pylint==2.3.1 (from -c constraints.txt (line 22))
  Using cached https://files.pythonhosted.org/packages/60/c2/b3f73f4ac008bef6e75bca4992f3963b3f85942e0277237721ef1c151f0d/pylint-2.3.1-py3-none-any.whl
Collecting astroid==2.2.5 (from -c constraints.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/d5/ad/7221a62a2dbce5c3b8c57fd18e1052c7331adc19b3f27f1561aa6e620db2/astroid-2.2.5-py3-none-any.whl
Collecting mccabe==0.6.1 (from -c constraints.txt (line 14))
  Using cached https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Collecting isort==4.3.19 (from -c constraints.txt (line 10))
  Using cached https://files.pythonhosted.org/packages/ae/ae/5ef4b57e15489754b73dc908b656b02ab0e6d37b190ac78dd498be8b577d/isort-4.3.19-py2.py3-none-any.whl
Collecting lazy-object-proxy==1.4.1 (from -c constraints.txt (line 12))
  Using cached https://files.pythonhosted.org/packages/43/a5/1b19b094ad19bce55b5b6d434020f5537b424fd2b3cff0fbef23d7bb5a95/lazy_object_proxy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting six==1.12.0 (from -c constraints.txt (line 28))
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting wrapt==1.11.1 (from -c constraints.txt (line 39))
  Using cached https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz
Collecting typed-ast==1.3.5 (from -c constraints.txt (line 37))
  Using cached https://files.pythonhosted.org/packages/17/9e/00918af7bdd616decb5b7ad06a9cd0a4a247d2fccaa630ab448a57e68b98/typed_ast-1.3.5-cp37-cp37m-manylinux1_x86_64.whl
Building wheels for collected packages: wrapt
  Building wheel for wrapt (setup.py): started
  Building wheel for wrapt (setup.py): finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/89/67/41/63cbf0f6ac0a6156588b9587be4db5565f8c6d8ccef98202fc
Successfully built wrapt
Installing collected packages: lazy-object-proxy, six, wrapt, typed-ast, astroid, isort, mccabe, pep8, pylint
Successfully installed astroid-2.2.5 isort-4.3.19 lazy-object-proxy-1.4.1 mccabe-0.6.1 pep8-1.7.1 pylint-2.3.1 six-1.12.0 typed-ast-1.3.5 wrapt-1.11.1
Removing intermediate container 394847f09e9b
 ---> 68e65a214a32
Successfully built 68e65a214a32
Successfully tagged test:latest

第二次运行（更改test.txt以触​​发对Docker第4层和第5层的重建之后）：

Sending build context to Docker daemon  5.632kB
Step 1/5 : FROM python:3.7
 ---> a4cc999cf2aa
Step 2/5 : COPY constraints.txt requirements.txt ./
 ---> Using cache
 ---> 411eaa3d36ff
Step 3/5 : RUN pip3 download -r requirements.txt -c constraints.txt
 ---> Using cache
 ---> 8ac3be432c58
Step 4/5 : COPY test.txt ./
 ---> 7ab5814153b7
Step 5/5 : RUN pip3 install -r requirements.txt -c constraints.txt
 ---> Running in 501da787ab07
Collecting pep8==1.7.1 (from -c constraints.txt (line 17))
  Using cached https://files.pythonhosted.org/packages/42/3f/669429ce58de2c22d8d2c542752e137ec4b9885fff398d3eceb1a7f5acb4/pep8-1.7.1-py2.py3-none-any.whl
Collecting pylint==2.3.1 (from -c constraints.txt (line 22))
  Using cached https://files.pythonhosted.org/packages/60/c2/b3f73f4ac008bef6e75bca4992f3963b3f85942e0277237721ef1c151f0d/pylint-2.3.1-py3-none-any.whl
Collecting astroid==2.2.5 (from -c constraints.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/d5/ad/7221a62a2dbce5c3b8c57fd18e1052c7331adc19b3f27f1561aa6e620db2/astroid-2.2.5-py3-none-any.whl
Collecting mccabe==0.6.1 (from -c constraints.txt (line 14))
  Using cached https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Collecting isort==4.3.19 (from -c constraints.txt (line 10))
  Using cached https://files.pythonhosted.org/packages/ae/ae/5ef4b57e15489754b73dc908b656b02ab0e6d37b190ac78dd498be8b577d/isort-4.3.19-py2.py3-none-any.whl
Collecting typed-ast==1.3.5 (from -c constraints.txt (line 37))
  Using cached https://files.pythonhosted.org/packages/17/9e/00918af7bdd616decb5b7ad06a9cd0a4a247d2fccaa630ab448a57e68b98/typed_ast-1.3.5-cp37-cp37m-manylinux1_x86_64.whl
Collecting six==1.12.0 (from -c constraints.txt (line 28))
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting wrapt==1.11.1 (from -c constraints.txt (line 39))
  Using cached https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz
Collecting lazy-object-proxy==1.4.1 (from -c constraints.txt (line 12))
  Using cached https://files.pythonhosted.org/packages/43/a5/1b19b094ad19bce55b5b6d434020f5537b424fd2b3cff0fbef23d7bb5a95/lazy_object_proxy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl
Building wheels for collected packages: wrapt
  Building wheel for wrapt (setup.py): started
  Building wheel for wrapt (setup.py): finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/89/67/41/63cbf0f6ac0a6156588b9587be4db5565f8c6d8ccef98202fc
Successfully built wrapt
Installing collected packages: typed-ast, six, wrapt, lazy-object-proxy, astroid, isort, mccabe, pep8, pylint
Successfully installed astroid-2.2.5 isort-4.3.19 lazy-object-proxy-1.4.1 mccabe-0.6.1 pep8-1.7.1 pylint-2.3.1 six-1.12.0 typed-ast-1.3.5 wrapt-1.11.1
Removing intermediate container 501da787ab07
 ---> b377fe561e97
Successfully built b377fe561e97
Successfully tagged test:latest

注意：官方文档很有帮助。

Answer 2

正确缓存 python 依赖项意味着每当构建 Dockerfile 时，它​​都会重新使用安装 python 依赖项。我尝试使用下面的 DockerFile 来构建 docker。

要求.txt

Django==2.1.7

Docker文件

FROM python:3
ENV PYTHONUNBUFFERED 1
ADD requirements.txt ./requirements.txt
RUN pip3 install -r requirements.txt

构建docker镜像

Django==2.1.7

第一次构建docker

第二次构建docker。它默认使用缓存中的依赖项。

如果您对requirements.txt 进行任何更改或库版本更改，则它不会使用缓存。并安装新的库版本。

谢谢。