docker-compose网络创建让我脱离了ssh

Question

由于docker-compose网络的缘故,我需要帮助了解我的网络日志.

我正在进入一个虚拟机,我有两个带有docker-compose的项目.第一个是简单地启动docker-compose up.当我尝试启动第二个时,我的ssh会话冻结,我再也无法进入虚拟机.经过大量的试验和错误,在阅读本文之后,我尝试将第二个项目的docker-compose.yml文件追加到以下内容:

networks:
  default:
    external:
      name: ffamfe_default

哪个ffamfe_default是docker-compose up第一个项目创建的网络的名称.有了这个,docker-compose up第二个项目并没有让我退出ssh会话.

我尾在日志中/var/log/*.log,和这里的输出与中网部分docker-compose.yml文件(不带时间戳前缀:Jan 19 09:13:42 hostname kernel: [420096.305357]):

aufs au_opts_verify:1597:dockerd[13813]: dirperm1 breaks the protection by the permission bits on the lower branch
device veth6a84537 entered promiscuous mode
IPv6: ADDRCONF(NETDEV_UP): veth6a84537: link is not ready
eth0: renamed from veth2480623
IPv6: ADDRCONF(NETDEV_CHANGE): veth6a84537: link becomes ready
br-fe0deb0149df: port 18(veth6a84537) entered forwarding state
br-fe0deb0149df: port 18(veth6a84537) entered forwarding state
aufs au_opts_verify:1597:dockerd[25317]: dirperm1 breaks the protection by the permission bits on the lower branch
device veth1a3c1e3 entered promiscuous mode
IPv6: ADDRCONF(NETDEV_UP): veth1a3c1e3: link is not ready
br-fe0deb0149df: port 22(veth1a3c1e3) entered forwarding state
br-fe0deb0149df: port 22(veth1a3c1e3) entered forwarding state
eth0: renamed from veth54e576d
IPv6: ADDRCONF(NETDEV_CHANGE): veth1a3c1e3: link becomes ready
br-fe0deb0149df: port 22(veth1a3c1e3) entered disabled state
veth54e576d: renamed from eth0
br-fe0deb0149df: port 22(veth1a3c1e3) entered disabled state
device veth1a3c1e3 left promiscuous mode
br-fe0deb0149df: port 22(veth1a3c1e3) entered disabled state
br-fe0deb0149df: port 18(veth6a84537) entered forwarding state

这里是没有该networks部分的输出(即当我被踢出ssh会话时):

IPv6: ADDRCONF(NETDEV_UP): br-55349b03453a: link is not ready
aufs au_opts_verify:1597:dockerd[26982]: dirperm1 breaks the protection by the permission bits on the lower branch
aufs au_opts_verify:1597:dockerd[26982]: dirperm1 breaks the protection by the permission bits on the lower branch
aufs au_opts_verify:1597:dockerd[3051]: dirperm1 breaks the protection by the permission bits on the lower branch
device veth7a1bcde entered promiscuous mode
IPv6: ADDRCONF(NETDEV_UP): veth7a1bcde: link is not ready
br-55349b03453a: port 1(veth7a1bcde) entered forwarding state
br-55349b03453a: port 1(veth7a1bcde) entered forwarding state
br-55349b03453a: port 1(veth7a1bcde) entered disabled state
eth0: renamed from veth5d8a2ea
IPv6: ADDRCONF(NETDEV_CHANGE): veth7a1bcde: link becomes ready
br-55349b03453a: port 1(veth7a1bcde) entered forwarding state
br-55349b03453a: port 1(veth7a1bcde) entered forwarding state
IPv6: ADDRCONF(NETDEV_CHANGE): br-55349b03453a: link becomes ready
aufs au_opts_verify:1597:dockerd[13814]: dirperm1 breaks the protection by the permission bits on the lower branch
aufs au_opts_verify:1597:dockerd[13814]: dirperm1 breaks the protection by the permission bits on the lower branch
aufs au_opts_verify:1597:dockerd[13922]: dirperm1 breaks the protection by the permission bits on the lower branch
device veth3253bd4 entered promiscuous mode
IPv6: ADDRCONF(NETDEV_UP): veth3253bd4: link is not ready
br-55349b03453a: port 2(veth3253bd4) entered forwarding state
br-55349b03453a: port 2(veth3253bd4) entered forwarding state
br-55349b03453a: port 2(veth3253bd4) entered disabled state
eth0: renamed from veth9c8aaa3
IPv6: ADDRCONF(NETDEV_CHANGE): veth3253bd4: link becomes ready
br-55349b03453a: port 2(veth3253bd4) entered forwarding state
br-55349b03453a: port 2(veth3253bd4) entered forwarding state
br-55349b03453a: port 2(veth3253bd4) entered disabled state
veth9c8aaa3: renamed from eth0
br-55349b03453a: port 2(veth3253bd4) entered disabled state
device veth3253bd4 left promiscuous mode
br-55349b03453a: port 2(veth3253bd4) entered disabled state
br-55349b03453a: port 1(veth7a1bcde) entered forwarding state
br-55349b03453a: port 1(veth7a1bcde) entered disabled state
veth5d8a2ea: renamed from eth0
br-55349b03453a: port 1(veth7a1bcde) entered disabled state
device veth7a1bcde left promiscuous mode
br-55349b03453a: port 1(veth7a1bcde) entered disabled state

我真的不明白如何阅读这些日志. 这也是我的ifconfig.有人可以帮助我阅读日志并找出问题所在吗？

Answer 1

诊断

我们的团队正在使用运行 Ubuntu 18.04 的 AWS EC2 实例作为开发服务器。我们最近收到报告称 docker-compose 破坏了 SSH 连接。即使重新启动后，开发服务器仍然无法访问。于是我开始调查。

我能够通过仅使用 docker 进行复制来排除 docker-compose 的原因。

ubuntu@ip-172-31-115-116:~$ docker network create -d bridge my-bridge-network
aca5884d60f146cef81ac55c8cccd231a43f40927d645168642d9b28c5e009a6

ubuntu@ip-172-31-115-116:~$ docker network prune
WARNING! This will remove all custom networks not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Networks:
my-bridge-network

ubuntu@ip-172-31-115-116:~$ docker network create -d bridge my-bridge-network
f0a7a06a9627bc2de00eb60091a92010451690626d95e077f622f3058cc3a07c

ubuntu@ip-172-31-115-116:~$ docker network prune
WARNING! This will remove all custom networks not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Networks:
my-bridge-network

ubuntu@ip-172-31-115-116:~$ docker network create -d bridge my-bridge-network
Connection reset by 172.31.115.116 port 22

然后我想到了根本原因。

根本原因

• 我们的 docker-compose 文件使用桥接网络模式，默认情况下会创建一个新的桥接网络。当docker-compose down或docker network prune运行时，桥接网络将被拆除。而接下来docker-compose run还是docker network create会创建一个新的桥接网络。
• docker0 网桥适配器的默认 IP 范围是172.17.0.0/16.
• 当我第一次运行该docker network create -d bridge my-bridge-network 命令时，它为172.18.0.0/16.
• 第二个网桥适配器是为172.19.0.0/16.
• 自然，第三个桥接适配器是为172.20.0.0/16. 但是，这是我们的工程 VPN IP 范围。因此重叠导致服务器无法与我们的笔记本电脑通信。

解决方案

解决方案是确保新的 docker 网桥网络将跳过我们的 VPN IP 范围。

临时解决方案

如果我们将跳过的 IP 范围添加到系统路由表中，docker 会自动跳过它们。因此，只要开发服务器重新启动，我们就可以运行以下脚本。

sudo route add -net [our VPN IP range] netmask 255.255.0.0 gw [our gateway]

这个方案不完美，重启机器后新路由会被丢弃。

主要解决方案

我们应该将路由更改永久应用于所有开发服务器。

echo "            routes:" | sudo tee -a /etc/netplan/50-cloud-init.yaml
echo "            - to: [our VPN IP range]" | sudo tee -a /etc/netplan/50-cloud-init.yaml
echo "              via: [our gateway]" | sudo tee -a /etc/netplan/50-cloud-init.yaml
sudo netplan apply

Docker IP 更改

我们还计划修改 docker default-address-pools 以重新定义 docker IP 范围。参考https://github.com/docker/compose/issues/4336#issuecomment-457326123。我会说修改/etc/docker/daemon.json更好。