在 cron 中运行时,运行“docker-compose run”挂起

kuk*_*krt 7 ubuntu cron docker docker-compose

环境

? uname -a
Linux ubuntu 5.4.0-66-generic #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

?
docker -v Docker 版本 20.10.4,构建 d3cb89e

? docker-compose -v
docker-compose 版本 1.28.4,构建 cabd5cfb

? lsb_release -a
没有可用的 LSB 模块。
发行商 ID:Ubuntu
描述:Ubuntu 20.04.2 LTS
版本:20.04
代号:focus

问题

我一直在使用 ubuntu 18.04 版,我的 docker 和 docker-compose 运行得很好。我没有定期更新服务器,但就在 2 天前,我做了sudo apt-get update && apt-get upgrade。然后一切都开始变得奇怪了。

我曾经像这样从主机的 cron 调用 docker-compose 服务。

0 * * * * cd /project/folder && docker-compose run --rm service_name
Run Code Online (Sandbox Code Playgroud)

但是更新升级之后,这些cron作业就开始挂了。

  • 当我检查top命令时,我可以看到进程悬空
  • 我可以看到容器程序(命令)甚至没有启动

因此,每次 cron 作业尝试调用这些服务时,进程都在堆积。

我做过的事

我卸载了 docker、docker-compose,重新启动并重新安装它们,但它没有帮助。我什至将操作系统从 18.04 升级到 20.04,但也没有用。

https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository
https://docs.docker.com/compose/install/

do-release-upgrade # to upgrade from v18 to v20
Run Code Online (Sandbox Code Playgroud)

为了进一步调查,我添加了--verbose这样的内容并记录了下来。

0 * * * * cd /project/folder && docker-compose run --rm --verbose service_name >> /project/folder/cron.log 2>&1
Run Code Online (Sandbox Code Playgroud)

似乎它悬而未决,/usr/bin/docker start --attach --interactive并没有真正发生任何事情。

当我cd /project/folder && docker-compose run --rm --verbose service_name >> /project/folder/cron.log 2>&1仅从终端运行相同的命令时,它会按预期运行。

我使一切变得简单,如下所示,但是通过 cron 运行它仍然挂在中间。

任何人都知道是什么导致了这种情况?

docker-compose.yml

0 * * * * cd /project/folder && docker-compose run --rm service_name
Run Code Online (Sandbox Code Playgroud)

文件

FROM alpine:latest
Run Code Online (Sandbox Code Playgroud)

日志

compose.config.config.find: Using configuration files: ./docker-compose.yml
compose.cli.docker_client.get_client: docker-compose version 1.28.4, build cabd5cfb
docker-py version: 4.4.3
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost
compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '20.10.4', 'Details': {'ApiVersion': '1.41', 'Arch': 'amd64', 'BuildTime': '2021-02-25T07:03:23.000000000+00:00', 'Experimental': 'false', 'GitCommit': '363e9a8', 'GoVersion': 'go1.13.15', 'KernelVersion': '5.4.0-66-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.4.3', 'Details': {'GitCommit': '269548fa27e0089a8b8278fc4fc781d7f65a939b'}}, {'Name': 'runc', 'Version': '1.0.0-rc92', 'Details': {'GitCommit': 'ff819c7e9184c13b7c2607fe6c30ae19403a7aff'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=20.10.4, ApiVersion=1.41, MinAPIVersion=1.12, GitCommit=363e9a8, GoVersion=go1.13.15, Os=linux, Arch=amd64, KernelVersion=5.4.0-66-generic, BuildTime=2021-02-25T07:03:23.000000000+00:00
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('aaa_default')
compose.cli.verbose_proxy.proxy_callable: docker inspect_network -> {'Attachable': True,
 'ConfigFrom': {'Network': ''},
 'ConfigOnly': False,
 'Containers': {},
 'Created': '2020-07-19T12:11:28.401034027+09:00',
 'Driver': 'bridge',
 'EnableIPv6': False,
 'IPAM': {'Config': [{'Gateway': '192.168.80.1', 'Subnet': '192.168.80.0/20'}],
          'Driver': 'default',
          'Options': None},
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('aaa_default')
compose.cli.verbose_proxy.proxy_callable: docker inspect_network -> {'Attachable': True,
 'ConfigFrom': {'Network': ''},
 'ConfigOnly': False,
 'Containers': {},
 'Created': '2020-07-19T12:11:28.401034027+09:00',
 'Driver': 'bridge',
 'EnableIPv6': False,
 'IPAM': {'Config': [{'Gateway': '192.168.80.1', 'Subnet': '192.168.80.0/20'}],
          'Driver': 'default',
          'Options': None},
...
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('aaa_test')
compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64',
 'Author': '',
 'Comment': '',
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['/bin/sh'],
            'Domainname': '',
            'Entrypoint': None,
            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'],
...
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.parallel.feed_queue: Pending: {<Service: test>}
compose.parallel.feed_queue: Starting producer thread for <Service: test>
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
Creating aaa_test_run ... 
compose.parallel.feed_queue: Pending: {ServiceName(project='aaa', service='test', number=1)}
compose.parallel.feed_queue: Starting producer thread for ServiceName(project='aaa', service='test', number=1)
compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('aaa_test')
compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64',
 'Author': '',
 'Comment': '',
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['/bin/sh'],
            'Domainname': '',
            'Entrypoint': None,
            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'],
...
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='aaa_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None)
compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [],
 'Links': [],
 'LogConfig': {'Config': {}, 'Type': ''},
 'NetworkMode': 'aaa_default',
 'PortBindings': {},
 'VolumesFrom': []}
compose.cli.verbose_proxy.proxy_callable: docker create_container <- (command='ash -c "date"', volumes={}, tty=False, stdin_open=True, detach=False, ports=[], name='aaa_test_run_b01e0a8c11e3', environment=[], labels={'com.docker.compose.project': 'aaa', 'com.docker.compose.service': 'test', 'com.docker.compose.oneoff': 'True', 'com.docker.compose.project.working_dir': '/home/ac/projects/playground/aaa', 'com.docker.compose.project.config_files': 'docker-compose.yml', 'com.docker.compose.slug': 'b01e0a8c11e319685038ac27761c14b08d5fa025dee67abab9da546afadc28f', 'com.docker.compose.version': '1.28.4'}, image='aaa_test', host_config={'NetworkMode': 'aaa_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'aaa_default': {'Aliases': ['test'], 'IPAMConfig': {}}}})
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': 'c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b',
 'Warnings': []}
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b')
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['-c', 'date'],
 'Config': {'AttachStderr': True,
            'AttachStdin': True,
            'AttachStdout': True,
            'Cmd': ['ash', '-c', 'date'],
            'Domainname': '',
            'Entrypoint': None,
            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'],
            'Hostname': 'c743a18406c5',
...
compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='aaa', service='test', number=1)
Creating aaa_test_run ... done
compose.parallel.feed_queue: Pending: set()
compose.parallel.parallel_execute_iter: Finished processing: <Service: test>
compose.parallel.feed_queue: Pending: set()
compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b', 'aaa_default')
compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None
compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b', 'aaa_default', aliases=[], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None)
compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None
compose.cli.main.call_docker: /usr/bin/docker start --attach --interactive c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b
Run Code Online (Sandbox Code Playgroud)

DrM*_*ers 10

更新:这应该在 Docker 20.10.5 中修复,所以升级可能比降级更可取!

如果您降级到 Docker 20.10.3,问题应该可以解决。我相信这是此版本中引入的错误,如下所述:https : //github.com/moby/moby/issues/42093

恶心的问题!希望尽快发布修复...

如果降级是不是你的选择,你可以尝试设置COMPOSE_INTERACTIVE_NO_CLI环境变量(以1所讨论的)在这里-例如,在前面加上:

COMPOSE_INTERACTIVE_NO_CLI=1
Run Code Online (Sandbox Code Playgroud)

到您的 crontab 配置,以便docker-compose通过 cron 启动的任务应绕过此错误作为短期解决方法