如何让Linux为不同PC上的不同桥接设备生成不同的MAC地址？

Question

我刚刚将两台服务器从Debian 10 (Buster) 升级到Debian 11 (Bullseye)。此后，我再也无法通过网络联系到他们。经过一番排查，发现问题如下：

Both machines have a bridge device configured. Obviously, the algorithm which Debian uses to assign MAC addresses to bridge devices has changed from version 10 to 11. After the upgrade, the bridge device on the first server had the same MAC address as the bridge device on the second server, which for sure has not been the case before.

One of the answers there claims that a bridge is a purely internal device and that therefore a bridge's MAC address does not matter. However, this is obviously wrong. At least in my case, packets from both machines were outgoing with the hardware source address being the bridge's MAC address, and the network ports on both machines were processing incoming packets only if they were destined for the bridge's MAC address.

Since that MAC address was the same on both machines, the network became unusable, which is completely logical and understandable.

How can I make Debian generate different MAC addresses for bridge devices which are on different machines (or even on the same machine, but that's currently not my issue)?

Answer 1

Browsing in Internet I found this bug report on systemd-udev related to Debian 11 bridges: systemd-udev interferes with MAC addresses of interfaces it's not supposed to do #21185:

ash.in.ffho.net:~# for n in 0 1 2 3; do ip l add br$n type bridge; done
ash.in.ffho.net:~# ip -br l

br0              DOWN           d2:9e:b3:32:53:42 <BROADCAST,MULTICAST> 
br1              DOWN           e2:00:44:2c:5b:70 <BROADCAST,MULTICAST> 
br2              DOWN           0e:99:b7:42:f0:25 <BROADCAST,MULTICAST> 
br3              DOWN           a6:3f:5f:b5:9a:d6 <BROADCAST,MULTICAST> 
ash.in.ffho.net:~# for n in 0 1 2 3; do ip link del br${n}; done
ash.in.ffho.net:~# for n in 0 1 2 3; do ip l add br$n type bridge; done
ash.in.ffho.net:~# ip -br l

br0              DOWN           d2:9e:b3:32:53:42 <BROADCAST,MULTICAST> 
br1              DOWN           e2:00:44:2c:5b:70 <BROADCAST,MULTICAST> 
br2              DOWN           0e:99:b7:42:f0:25 <BROADCAST,MULTICAST> 
br3              DOWN           a6:3f:5f:b5:9a:d6 <BROADCAST,MULTICAST>

As you can see, the bridges were created with low-level commands, but they always inherit the same MAC address value: a systemd component interferes and sets the MAC address. One can see this in action using ip monitor link:

22: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 0a:ae:c3:0d:ec:68 brd ff:ff:ff:ff:ff:ff
22: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 1a:d0:fc:63:c1:71 brd ff:ff:ff:ff:ff:ff
Deleted 22: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 1a:d0:fc:63:c1:71 brd ff:ff:ff:ff:ff:ff
23: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 4e:e9:11:dd:a5:aa brd ff:ff:ff:ff:ff:ff
23: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 1a:d0:fc:63:c1:71 brd ff:ff:ff:ff:ff:ff

You can see how the MAC address initially random is overwritten to a fixed one, twice to the same value for a given bridge name.

An other side effect is that when interface is set administratively UP, the bridge operational status becomes DOWN instead of UNKNOWN initially because of this (see these answers of mine on SU and SF mentioning behaviors about DOWN and UNKNOWN: How does Linux determine the default MAC address of a bridge device? , linux ipv6 bridge address does not work when mac address is forced). Anyway this doesn't matter anymore once its first bridge port is attached.

Doing the same experiment inside a network namespace (eg: ip add netns experiment and ip netns exec experiment bash -l before running above commands twice) where systemd-udevd does not interfere will show the usual behavior of having different random addresses each time.

This is an effect of systemd ecosystem and doesn't happen on systems not running systemd (or older versions of systemd). One proposed fix is to use:

# /etc/systemd/network/90-bridge.link
[Match]
OriginalName=br*

[Link]
MACAddressPolicy=random

but it appears the real fix is to change the file that participates in generating this "stable random" value, as described there: https://wiki.debian.org/MachineId

Each machine should have a different value. This is especially important for cloned VMs from a base template. The relation between machine-id and the way the bridge "stable" MAC address is generated is mentioned in the patch having implemented the (quite breaking) change:

=== This patch

This patch means that we will set a "stable" MAC for pretty much any virtual device by default, where "stable" means keyed off the machine-id and interface name.

It was also mentioned that this would be having impacts , but this was shrugged off.

This is not limited to interfaces of type bridge but to any interface that would generate a random MAC address: for example types veth, macvlan tuntap are also affected.

I could verify that the same bridge name would get a different "stable random" value after doing the operations described in Debian's link:

rm -f /etc/machine-id /var/lib/dbus/machine-id
dbus-uuidgen --ensure=/etc/machine-id
dbus-uuidgen --ensure

现在ip monitor在删除和重新创建时为相同的网桥名称提供一个新的 MAC 地址：32:ee:c8:92:9f:e8 而不是 1a:d0:fc:63:c1:71 brtest0。

Deleted 23: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 1a:d0:fc:63:c1:71 brd ff:ff:ff:ff:ff:ff
24: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether da:72:b6:63:23:e5 brd ff:ff:ff:ff:ff:ff
24: brtest0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 32:ee:c8:92:9f:e8 brd ff:ff:ff:ff:ff:ff

结论：

由于桥接 MAC 地址现在是手动设置的，桥接器将不再继承设置为桥接端口的其他接口的 MAC 地址之一，包括通常的永久（物理或虚拟机）接口，这些接口预计具有不同的 MAC 地址。两个系统使用相同machine-id且相同的网桥名称（例如：br0），且此类网桥参与路由（即：网桥上配置了 IP 地址，但即使没有，网桥也可以根据其设置发出与桥接相关的其他帧）同一 LAN 上的网络将发出具有相同源 MAC 地址（网桥）的帧，可能会中断路径中的交换机，并且无论如何都会忽略来自对等方的相同源 MAC 地址。