使用 Conda + Poetry 进行机器学习项目有意义吗?让我分享一下我(新手)的理解,请指正或赐教:
据我了解,Conda和Poetry有不同的目的,但很大程度上是多余的:
我的想法是同时使用两者并划分它们的角色:让 Conda 担任环境管理器,让 Poetry 担任包管理器。我的推理是(听起来)Conda 最适合管理环境,可用于编译和安装非 python 包,尤其是 CUDA 驱动程序(用于 GPU 功能),而 Poetry 作为 Python 包管理器比 Conda 更强大。
通过在 Conda 环境中使用 Poetry,我成功地相当轻松地完成了这项工作。诀窍是不要使用 Poetry 来管理 Python 环境:我没有使用poetry shellor poetry run、 onlypoetry init等命令poetry install(激活 Conda 环境后)。
为了充分披露,我的environment.yml文件(针对Conda)如下所示:
name: N
channels:
- defaults
- conda-forge
dependencies:
- python=3.9
- cudatoolkit
- cudnn
Run Code Online (Sandbox Code Playgroud)
我的诗歌.toml …
我有一个Python(2.7)应用程序,它在我的dockerfile中启动:
CMD ["python","main.py"]
Run Code Online (Sandbox Code Playgroud)
main.py在启动时打印一些字符串,然后进入循环:
print "App started"
while True:
time.sleep(1)
Run Code Online (Sandbox Code Playgroud)
只要我使用-it标志启动容器,一切都按预期工作:
$ docker run --name=myapp -it myappimage
> App started
Run Code Online (Sandbox Code Playgroud)
我可以稍后通过日志看到相同的输出:
$ docker logs myapp
> App started
Run Code Online (Sandbox Code Playgroud)
如果我尝试使用-d标志运行相同的容器,容器似乎正常启动,但我看不到任何输出:
$ docker run --name=myapp -d myappimage
> b82db1120fee5f92c80000f30f6bdc84e068bafa32738ab7adb47e641b19b4d1
$ docker logs myapp
$ (empty)
Run Code Online (Sandbox Code Playgroud)
但容器似乎还在运行;
$ docker ps
Container Status ...
myapp up 4 minutes ...
Run Code Online (Sandbox Code Playgroud)
Attach也不显示任何内容:
$ docker attach --sig-proxy=false myapp
(working, no output)
Run Code Online (Sandbox Code Playgroud)
任何想法都出错了?在后台运行时"打印"的行为会不同吗?
Docker版本:
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.2
Git commit …Run Code Online (Sandbox Code Playgroud) 我想跑opam init,但出现错误:
(iit_synthesis) brandomiranda~ \xe2\x9d\xaf docker build -t brandojazz/pycoq:test_brando ~/pycoq/tutorial/\n[+] Building 1.5s (12/19) \n => [internal] load build definition from Dockerfile 0.0s\n => => transferring dockerfile: 2.33kB 0.0s\n => [internal] load .dockerignore 0.0s\n => => transferring context: 2B 0.0s\n => [internal] load metadata for docker.io/library/ubuntu:20.04 0.2s\n => CACHED https://api.github.com/repos/IBM/pycoq/git/refs/heads/main 0.0s\n => [ 1/15] FROM docker.io/library/ubuntu:20.04@sha256:fd92c36d3cb9b1d027c4d2a72c6bf0125da82425fc2ca37c414d4f010180dc19 0.0s\n => CACHED [ 2/15] RUN apt-get update && apt-get install -y --no-install-recommends ssh git m4 libgmp-dev opam wget ca-certificates rsync strace …Run Code Online (Sandbox Code Playgroud) 我正在尝试在构建 conda 环境的服务器上构建 Docker 容器。除了启用 CUDA 的 PyTorch(我可以在没有 CUDA 的情况下让 PyTorch 工作,但是没问题)之外的所有其他要求都得到满足。如何确保 PyTorch 正在使用 CUDA?
这是Dockerfile:
# Use nvidia/cuda image
FROM nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04
# set bash as current shell
RUN chsh -s /bin/bash
# install anaconda
RUN apt-get update
RUN apt-get install -y wget bzip2 ca-certificates libglib2.0-0 libxext6 libsm6 libxrender1 git mercurial subversion && \
apt-get clean
RUN wget --quiet https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh -O ~/anaconda.sh && \
/bin/bash ~/anaconda.sh -b -p /opt/conda && \
rm ~/anaconda.sh && \
ln -s /opt/conda/etc/profile.d/conda.sh …Run Code Online (Sandbox Code Playgroud) 我有以下内容Dockerfile。
FROM continuumio/miniconda3:4.5.11
# create a new user (defaults to 'al-khawarizmi')
USER root
ARG username=al-khawarizmi
RUN useradd --create-home --home-dir /home/${username} ${username}
ENV HOME /home/${username}
# switch to newly created user to avoid running container as root
USER ${username}
WORKDIR $HOME
# build and activate the specified conda environment from a file (defaults to 'environment.yml')
ARG environment=environment.yml
COPY ${environment} .
RUN conda env create --file ${environment} && \
echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
echo "conda activate $(head …Run Code Online (Sandbox Code Playgroud) 我已经尝试了好几个小时,无法在构建过程中弄清楚如何在Dockerfile中激活和切换anaconda环境。
这是初始代码:
FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu16.04
# Set user
ENV SETUSER myuser
RUN useradd -m $SETUSER
USER $SETUSER
WORKDIR /home/$SETUSER
# Install Anaconda
RUN wget https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh
RUN bash Anaconda3-2019.03-Linux-x86_64.sh -b
RUN rm Anaconda3-2019.03-Linux-x86_64.sh
# Set path to conda
ENV CONDA_ENV_NAME mynewenv
RUN /home/$SETUSER/anaconda3/condabin/conda create -q --name $CONDA_ENV_NAME python=3.6 && \
/home/$SETUSER/anaconda3/condabin/conda clean --yes --all
RUN /home/$SETUSER/anaconda3/condabin/conda activate base #Just for testing anaconda environment
Run Code Online (Sandbox Code Playgroud)
首先,Docker中的anaconda会抱怨外壳设置不正确,因此在conda create命令之后添加了:
RUN /home/$SETUSER/anaconda3/condabin/conda init bash
RUN /bin/bash -c "source /home/$SETUSER/.bashrc"
RUN /home/$SETUSER/anaconda3/condabin/conda activate base
Run Code Online (Sandbox Code Playgroud)
在构建docker镜像后运行3个命令即可(即在调用docker run container-name之后交互运行),但是由于某种原因,在构建 …
我正在尝试在 docker 容器内设置 conda 环境,但它总是在 dockerfile 中的环境创建阶段失败,并出现错误:
conda activate" fails because shell is not initialized
下面是我的 Dockerfile:
FROM pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime
WORKDIR /app
# SET BASH AS CURRENT SHELL
RUN chsh -s /bin/bash
SHELL ["/bin/bash", "-c"]
# INSTALL ANACONDA
RUN apt-get update -y
RUN apt-get install -y wget && \
apt-get clean
RUN rm -rf /opt/conda && \
wget --quiet https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh -O ~/anaconda.sh && \
/bin/bash ~/anaconda.sh -b -p /opt/conda && \
rm ~/anaconda.sh && \
ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \ …Run Code Online (Sandbox Code Playgroud)