jessie

Create percona xtradb cluster using rkt containers on multiple hosts over flannel overlay network

Keresztes Peter-Zoltan

Jun 17, 2017 • 6 min read

Since the new technologies are using containers I have decided to give rkt a try to see what it can do. Rkt is developed by CoreOs and was announced in 2014. It got alot of attention in February 2016 when it was released version 1.0. Rkt can run docker images therefore there is no more need for docker to run the already built images.

It's enough about rkt since there are quite a lot of informations out there which does not need to be repeated. To cover the setup we are using 3 deployed virtual machines with public IP's running Debian jessie. We are running an etcd v3.2.0 cluster on the hosts running rkt. To make it simple we do not use ssl/tls certificates to encrypt the internode traffic. To run the containers in background we are running it using systemd.

Let's see how it is done.

First we are creating an etcd cluster id by simple browing http://54.215.153.3/new:

#curl http://54.215.153.3/new
https://discovery.etcd.io/<etcd cluster discovery id>

Once we have the cluster discovery id we can start running the rkt container using the following command:

#systemd-run --slice=machine \
--unit=etcd3 \
rkt run \
--net=host \
--dns=8.8.8.8 \
--hostname=$(hostname) \
--set-env=ETCD_INITIAL_ADVERTISE_PEER_URLS=http://$(hostname -I | awk '{print $1}'):2380 \
--set-env=ETCD_DISCOVERY=http://54.215.153.3/<etcd cluster discovery id> \
--set-env=ETCD_LISTEN_PEER_URLS=http://$(hostname -I | awk '{print $1}'):2380 \
--set-env=ETCD_LISTEN_CLIENT_URLS=http://$(hostname -I | awk '{print $1}'):2379,http://$(hostname -I | awk '{print $1}'):4001,http://127.0.0.1:2379,http://127.0.0.1:4001 \
--set-env=ETCD_ADVERTISE_CLIENT_URLS=http://$(hostname -I | awk '{print $1}'):2379 \
--set-env=ETCD_NAME=$(hostname)-etcd-cluster \
--set-env=ETCD_DATA_DIR=$(hostname) \
coreos.com/etcd:v3.2.0 \
-- --log-output=stderr --- \
--debug=true

some of the more important options are:

--net = setting the networking type, In this case we are using the host networking therfore the etcd will be binding on the hosts IP addresses.
--dns = we are telling to the rkt what dns servers to be used.
--hostname = we are setting the hostname of the container
--debug = enabling or disabling debugging of the rkt
coreos.com/etcd:v3.2.0 is the image name which is to be used for the rkt container.

After starting the container on all the 3 nodes we can check the containers by running:

#rkt list
UUID		APP	IMAGE NAME				STATE	CREATED		STARTED		NETWORKS
bda8c7d7	etcd	coreos.com/etcd:v3.2.0			running	1 hour ago	1 hour ago

Now that we have the etcd cluster running it's time to configure the flanneld.
For this we need to install go version 1.5 or higher.

We have downloaded go versio 1.8. using the following link.

#cd /usr/src
#wget https://storage.googleapis.com/golang/go1.8.3.linux-amd64.tar.gz
#tar -xzvf go1.8.3.linux-amd64.tar.gz
#export GOPATH=/usr/src/go
#export GOROOT=/usr/src/go
#export PATH=$PATH:$GOROOT

Now it's time to download the flannel. This installer is a little bit weird cause it will need to be installed in under $GOPATH and with a specific folder path.

#apt-get update
#apt-get -y install build-essential linux-libc-dev git
#mkdir -p src/github.com/coreos
#git clone https://github.com/coreos/flannel.git
#cd flannel
#CGO_ENABLED=1 make dist/flanneld
#cp dist/flanneld /sbin/flanneld

Now that we have flanneld installed it's type to configure it. Flannel does not have a configuration file it's configuration it's registered into etcd. That is the reason why we have created the etcd cluster first.
First we need to create the flannel network config for rkt into /etc/rkt/net.d:

#cat /etc/rkt/net.d/10-test.conf
{
	"name": "test",
	"type": "flannel"
}

Now we can register the flannel overlay network configuration using the following command:

#rkt enter --app=etcd bda8c7d7 etcdctl set /xtradb/network/config '
{
    "Network": "10.0.0.0/8",
    "SubnetLen": 20,
    "SubnetMin": "10.10.0.0",
    "SubnetMax": "10.99.0.0",
    "Backend": {
        "Type": "vxlan",
        "VNI": 100,
        "Port": 8472 }
}'
{"Network": "10.0.0.0/8", "SubnetLen": 20, "SubnetMin": "10.10.0.0", "SubnetMax": "10.99.0.0", "Backend": { "Type": "vxlan", "VNI": 100, "Port": 8472 } }

Now we can start running flanneld on each host. As for rkt we will be running flanneld using systemd:

#systemd-run \
--unit=flannel \
--slice=machine \
/sbin/flanneld \
-public-ip="$(hostname -I | awk '{print $1}')" \
-etcd-prefix \
/xtradb/network \
-etcd-endpoints=http://<host1_IP>:2379,http://<host2_IP>:2379,http://<host3_IP>:2379 \
-iface=eth0 \
--ip-masq=true \
-v=1

If we are watching the journalctl then we will see the following output:

Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.748117     516 main.go:331] Using interface with name eth0 and address <host1_IP>
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.749289     516 main.go:344] Using <host1_IP> as external address
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.750008     516 main.go:148] Created subnet manager: &{registry:0xc4202dec90}
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.750588     516 main.go:151] Installing signal handlers
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.780903     516 main.go:248] Found network config - Backend type: vxlan
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.821736     516 local_manager.go:181] Picking subnet in range 10.10.0.0 ... 10.99.0.0
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.842398     516 local_manager.go:167] Allocated lease (10.12.16.0/20) to current node (<host1_IP>)
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.857298     516 ipmasq.go:47] Adding iptables rule: -s 10.0.0.0/8 -d 10.0.0.0/8 -j RETURN
Jun 17 17:22:41 test1 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Jun 17 17:22:41 test1 kernel: nf_conntrack version 0.5.0 (8002 buckets, 32008 max)
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.920789     516 ipmasq.go:47] Adding iptables rule: -s 10.0.0.0/8 ! -d 224.0.0.0/4 -j MASQUERADE
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.931444     516 ipmasq.go:47] Adding iptables rule: ! -s 10.0.0.0/8 -d 10.0.0.0/8 -j MASQUERADE
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.937259     516 main.go:197] Wrote subnet file to /run/flannel/subnet.env
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.938328     516 main.go:202] Finished starting backend.
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.940547     516 vxlan_network.go:56] Watching for L3 misses
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.941405     516 vxlan_network.go:64] Watching for new subnet leases
Jun 17 17:22:41 test1 flanneld[516]: I0617 17:22:41.946535     516 main.go:285] Waiting for 22h59m59.876144802s to renew lease
Jun 17 17:22:51 test1 flanneld[516]: I0617 17:22:51.471353     516 vxlan_network.go:157] Handling initial subnet events
Jun 17 17:22:59 test1 flanneld[516]: I0617 17:22:59.391501     516 vxlan_network.go:116] Subnet added: 10.13.16.0/20

Now we have both etcd and flannel running and it's time to start the xtradb cluster nodes which will be registering to the etcd to create the cluster on different hosts. To create the nodes run the following command on each node. It is recommended to wait for the first node to start up before you start the next nodes.

#systemd-run \
--slice=machine \
--unit=xtradb \
rkt run \
--hostname=$(hostname)-container \
--set-env=MYSQL_ROOT_PASSWORD=somepass \
--set-env=DISCOVERY_SERVICE=$(hostname -I | awk '{print $1}'):2379 \
--set-env=CLUSTER_NAME=somecluster \
--set-env=XTRBACKUP_PASSWORD=xtr4b4ckup \
--net=default,test \
--port=3306-tcp:13601 \
--insecure-options=image \
docker://quay.io/zozo6015/percona-xtradb:latest \
--name=xtradb \
--debug=true

I am using a slightly modified docker percona xtradb image since the script inside which ensures if the cluster nodes are registered into etcd were not working properly so I had to fix the script for my usage. But the official image should be working as well out of the box.

Once all the 3 nodes are started properly you can check the mysql cluster status using the following commands:

#rkt list
UUID		APP	IMAGE NAME				STATE	CREATED		STARTED		NETWORKS
bda8c7d7	etcd	coreos.com/etcd:v3.2.0			running	1 hour ago	1 hour ago	
dd10431a	xtradb	quay.io/zozo6015/percona-xtradb:latest	running	1 hour ago	1 hour ago	test:ip4=10.12.16.2, default:ip4=172.16.28.2
#rkt enter dd10431a mysql -u root -psomepass
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1238
Server version: 5.7.18-15-57-log Percona XtraDB Cluster (GPL), Release rel15, Revision 7693d6e, WSREP version 29.20, wsrep_29.20

Copyright (c) 2009-2017 Percona LLC and/or its affiliates
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show status like 'wsrep_%';
....
| wsrep_incoming_addresses         | 10.15.160.2:3306,10.12.16.2:3306,10.13.16.2:3306 |
| wsrep_cluster_size               | 3                                                |
....
+----------------------------------+--------------------------------------------------+
67 rows in set (0.00 sec)
mysql>

As results this is a very simple way of creating a percona xtradb cluster using rkt. Of course we could use some mounted folders from the host to ensure that we don't loose any files in case the xtradb cluster goes down and the container needs to be redeployed but that is not covered into this tutorial since it's coverage inside the rkt documentation is good enough.

Sign up for more like this.