Multi-host docker cluster using OVS/VxLAN on CentOS 7
The community version of RHEL 7; Community Enterprise OS (CentOS) 7 comes with docker 1.8.x. Which does not support multi-host networking as it only support linux bridges. In other words a container on Host A can't talk to a container on Host B. If you have a cluster of 5 nodes then you can't utilize them all.
In this article I'm going to show you how to setup a multi-host docker cluster where containers on different hosts can talk to each other without NATing (Network Address Translation). I'm going to use OpenVSwitch using VxLAN tunnels (or GRE tunnels) both are Linux kernel technologies that are true and tested (they are used in production for years).
The Open Virtual Switch (OVS) is an L2 switch just like physical switch, it has ports where you plug ethernet plugs and it works at MAC address / ARP level to connect them.
Docker 1.9.x shipped in Fedora 23 do support this feature via something called overlay network but even in that case I still prefer OVS method.
I don't like creating tunnels in user space (like CoreOS Flannel) specially in languages with automatic garbage collection. Remember all inter-container-communication is going via this tunnel it would certainly be a bottleneck.
In this article I'll keep SELinux enabled and I'll use "firewalld"
My Setup - your mileage may vary!I used some VM nodes, each has it's own LAN IP and docker bridge (called docker0 or br0) on each node should has it's own non-overlapping subnet addresses that belong to a bigger subnet in the virtual switch (called sw0).
I used the same 172.x.y.z/16 for the bridge and 172.x.y.z/12 for the bigger switch, one might like to use 10.x.y.z/16 and 10.x.y.z/8.
so we have
- eth0: 192.168.122.81
- sw0: 172.30.0.1/12
- br0 / docker0: 172.18.0.1/16
- eth0: 192.168.122.82
- sw0: 172.30.0.2/12
- br0 / docker0: 172.19.0.1/16
- eth0: 192.168.122.83
- sw0: 172.30.0.3/12
- br0 / docker0: 172.20.0.1/16
- ... and so on
The address of each host in the virtual switch is 172.30.0.1/12 ,172.30.0.2/12, 172.30.0.3/12 as the number of the node and /12 means that it can route IPs in the range 172.16.0.0 - 172.31.255.255 as they belong to the same network.
I made the docker bridge starts from 172.18.y.z on node 1 then add 1 for each other node. and it's /16 which means it only routes IPs in the range 172.18.0.0-172.18.255.255 (in other words packets with IPs not in that range does not get routed to this bridge but to sw0 switch)
In a similar way if you choose the IP of docker bridge to be 10.1.0.1/16 for node 1 (and 10.2.0.1/16 for node two ..etc.) this will cover IPs in range 10.1.0.0 - 10.1.255.255 to be routed for single host communication and 10.200.0.1/8, 10.200.0.2/8, 10.200.0.3/8 for sw0 on each node which will cover range 10.0.0.0-10.255.255.255 for communications across multiple hosts
Step By Step
You need to run all those steps on all nodes. Let's install some dependencies, we need to some repos, let's enable "Extra Packages for Enterprise Linux" (EPEL) and "RedHat Community Distribution of Openstack" (RDO)
yum install -y epel-release https://www.rdoproject.org/repos/rdo-release.rpm yum install -y firewalld docker openvswitch bridge-utils nc yum update -y systemctl start openvswitch firewalld systemctl enable openvswitch firewalld
Let's open VxLAN ports (on production you should restrict this to a trusted network interface or network as below)
firewall-cmd --add-port=4789/udp --add-port=8472/udp firewall-cmd --permanent --add-port=4789/udp --add-port=8472/udp
You might replace that with GRE ports if you want GRE instead of VxLAN.
firewall-cmd --permanent --zone=trusted --add-source=172.16.0.0/12
Now let's customize default docker bridge "docker0" on each node by editing /etc/sysconfig/docker-network so that it looks like this
# /etc/sysconfig/docker-network DOCKER_NETWORK_OPTIONS="--bip=172.18.0.1/16 --iptables=false --ip-masq=false --ip-forward=true"
be sure to replace 172.18.0.1/16 with 172.19.0.1/16 on node 2.
Another option is to create a different bridge br0 using
brctl addbr br0 ip addr add 172.18.0.1/16 dev br0 && ip link set dev br0 up
and to make that permanent instead of the above just create the file ifcfg-br0
cat /etc/sysconfig/network-scripts/ifcfg-br0 DEVICE=br0 TYPE=Bridge IPADDR=172.18.0.1 NETMASK=255.255.0.0 ONBOOT=yes BOOTPROTO=none DELAY=0
and pass "-b=br0" instead of "--bip" to docker. I prefer the first method of just customize "docker0" via "--bip".
Now let's verify that works, let's start docker and run a a busybox container and type "ip a"
systemctl start docker systemctl enable docker ip a # verify docker0 exists and the needed IP address docker run -ti busybox /bin/sh # ip a
Now containers like the busybox above can't talk to each other from different hosts. Let's proceed.
Adding Virtual Switch
Let's create a switch (aka. bridge) on each node and connect it with other nodes using a VxLAN tunnel, just type
ovs-vsctl add-br sw0 ip addr add 172.30.0.1/12 dev sw0 && ip link set dev sw0 up ovs-vsctl add-port sw0 vxlan0 -- set Interface vxlan0 type=vxlan options:remote_ip=192.168.122.82 ovs-vsctl add-port sw0 vxlan0 -- set Interface vxlan0 type=vxlan options:remote_ip=192.168.122.83
here 172.30.0.1/12 is sw0 of node 1 and tell it how to reach other nodes like 192.168.122.82
on the second node
ovs-vsctl add-br sw0 ip addr add 172.30.0.2/12 dev sw0 && ip link set dev sw0 up ovs-vsctl add-port sw0 vxlan0 -- set Interface vxlan0 type=vxlan options:remote_ip=192.168.122.81 ovs-vsctl add-port sw0 vxlan0 -- set Interface vxlan0 type=vxlan options:remote_ip=192.168.122.83
and so on. You can use GRE instead of VxLAN.
The good thing about ovs-vsct is that they persist after reboot
Now on each node just create a pair of Virtual Ethernet that from one end connects to the docker bridge "docker0" and from the other end the virtual switch "sw0", we shall call them veth_sw0 and veth_d0 for the switch end and the docker end, this is done by typing:
ip link add veth_sw0 type veth peer name veth_d0 ovs-vsctl add-port sw0 veth_sw0 brctl addif br0 veth_d0 ip link set dev veth_sw0 up ip link set dev veth_d0 up
Verify that by typing "ovs-vsctl show" and "ip a"
# ovs-vsctl show
Verifying cross host connectivity
now on on host start a container
docker run -ti busybox /bin/sh
# ip a
# nc -l -p 8080 0.0.0.0
and on the other host start a container and try to reach the first one (where OTHER_IP is the one seen above in "ip a" output)
docker run -ti busybox /bin/sh
# ip a
# echo test | nc $OTHER_IP 8080
containers can reach hosts using their sw0 addresses (like 172.30.0.1)
Talking to the outside world
Let's add NATing rule so that containers can talk to outside world, so if a packet coming from a container (172.16.0.0/12) want to talk something to the outside world (let's say google.com) in other words its destination does not belong to 172.16.0.0/12 then it needs NAT ie. it will go through hosts's 192.168.x.y then to the outside world
firewall-cmd --direct --add-rule ipv4 nat POSTROUTING 0 -s 172.16.0.0/12 ! -d 172.16.0.0/12 -j MASQUERADE
firewall-cmd --permanent --direct --add-rule ipv4 nat POSTROUTING 0 -s 172.16.0.0/12 ! -d 172.16.0.0/12 -j MASQUERADE
to test this
docker run -ti busybox /bin/sh
# ping google.com