Recently, I realized I had misunderstood how Docker containers communicate across networks.
Initially, I thought containers could communicate with other networks without needing to attach to multiple networks.
But according to Docker’s documentation, containers can only communicate across networks if they are explicitly attached to more than one network.
This left me puzzled. While the documentation seemed to validate my understanding, my recent experiences told a different story.
Here’s what I originally thought:
- Bridge interfaces are created on the Docker host.
- Each bridge has a gateway IP assigned.
- Inside the container, it operates in an isolated network namespace, with a veth pair linking the container to the bridge.
On the Docker host:
- IP forwarding is enabled.
- NAT is applied via iptables to enable internet access for the containers.
cat /proc/sys/net/ipv4/ip_forward
1
iptables -t nat -L -v -n
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
7 420 MASQUERADE all -- * !br-907f0c015fa0 172.23.0.0/16 0.0.0.0/0
8 504 MASQUERADE all -- * !br-923e0836e2fd 172.21.0.0/16 0.0.0.0/0
Since the gateway ip is the Docker host itself, I assumed that inter-network traffic would be routed between containers through the gateway.
container1
14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:17:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.23.0.2/16 brd 172.23.255.255 scope global eth0
valid_lft forever preferred_lft forever
container2
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:15:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.21.0.2/16 brd 172.21.255.255 scope global eth0
valid_lft forever preferred_lft forever
docker host
3: br-923e0836e2fd: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:9a:b1:50:33 brd ff:ff:ff:ff:ff:ff
inet 172.21.0.1/16 brd 172.21.255.255 scope global br-923e0836e2fd
valid_lft forever preferred_lft forever
inet6 fe80::42:9aff:feb1:5033/64 scope link
valid_lft forever preferred_lft forever
5: br-907f0c015fa0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:e7:c6:c0:af brd ff:ff:ff:ff:ff:ff
inet 172.23.0.1/16 brd 172.23.255.255 scope global br-907f0c015fa0
valid_lft forever preferred_lft forever
inet6 fe80::42:e7ff:fec6:c0af/64 scope link
valid_lft forever preferred_lft forever
However, after doing more research, I found a discussion here - https://forums.docker.com/t/understanding-iptables-rules-added-by-docker/77210/3. It turns out Docker adds some security measures to its bridge network using iptables rules.
Docker uses two specific chains, DOCKER-ISOLATION-STAGE-1 and DOCKER-ISOLATION-STAGE-2, to block communication between containers on different networks. While NAT still works for internet access, these rules prevent communication between containers via the gateway. As a result, for containers to communicate across networks, they must be attached to multiple networks, enabling communication at L2.
iptables -t -L -v -n
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- br-923e0836e2fd !br-923e0836e2fd 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-2 all -- br-907f0c015fa0 !br-907f0c015fa0 0.0.0.0/0 0.0.0.0/0
2757 1311K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (3 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * br-923e0836e2fd 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- * br-907f0c015fa0 0.0.0.0/0 0.0.0.0/0
1829 983K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
If we remove the rules in the DOCKER-ISOLATION-STAGE-1 chain, Docker containers will be able to communicate across networks again.
This makes me reconsider the motivation behind this design. Technically, Docker doesn’t require multiple networks for communication between containers. However, this would complicate network isolation, as users would need to manually handle iptables rules.
The purpose of this the isolation 1+2 chain are to block inter-container communication at gateway (L3). Forcing communication through L2 by adding multiple networks to the container.
This also cleared up a question I had. In K8s, we can apply security policies between pods, but Docker doesn’t have a built-in mechanism for this. My mindset was that we needed setting up firewall for that. Actually we can do that in Docker simply by using docker network.
For examples,
- To allow only the API to communicate with the database, connect the API to the database network. Other components won’t have access to the database unless they’re part of that network.
- To block internet access for a container, use an internal network.
- To restrict external traffic, we can create an ingress container. Backends that need to be exposed externally can connect to the ingress network, and iptables rules can be applied to the DOCKER_USER chain to control the traffic.