Backing up Kubernetes ETCD database to remote server privately over NetFoundry Network.




The etcd server is the only stateful component of the Kubernetes cluster. Kubernetes stores all API objects and settings on the etcd server. Etcd backup is enough to restore the Kubernetes cluster’s state completely. Kubernetes disaster recovery plans often include backing up the etcd cluster and using infrastructure as code to create new cloud servers.


While backing up your etcd data to a local volume might be good for your solution, other might find it useful to have a remote copy of etcd which can be scripted automatically. The NetFoundry platform enables administrators to create secure network connectivity between Kubernetes Clusters and remote ETCD backup servers. All connectivity is private without the need for open Firewall ports or whitelisting addresses.


This demo provides the steps required to configure the network and specific AppWAN for the solution.


Assumptions /Pre-requisites


1. NetFoundry Platform account created. 


2. NetFoundry Network created.


3. NetFoundry Hosted Edge Router created.


3. NetFoundry Edge Router policy created.


4. You have an active Kubernetes Cluster.


5. Created a Kubernetes cluster endpoint and deployed to cluster.


6. Active remote Linux host to be used as ETCD backup/restore server.



NOTE: The diagram below shows a K8's cluster deployed in Oracle with a Controller and 3 Nodes.  It also shows a couple of AppWANs designed to provide connectivity to either a Private Cloud virtual machine or a Public Cloud virtual machine. Installed on either Node will be a Ziti-Edge-Tunnel POD deployed as a Daemonset. This DaemonSet will allow for private inbound/outbound communications to cluster from remote host.  While many use cases are possible with Kubernetes workloads, this guide will focus on the Etcd backup/restore connectivity.





Deployments steps


Step 1. Ensure ETCD and ETCDCTL are installed on your K8s controller

sudo apt update 
sudo apt install vim wget curl
export RELEASE=$(curl -s|grep tag_name | cut -d '"' -f 4) 
tar xvf etcd-${RELEASE}-linux-amd64.tar.gz
cd etcd-${RELEASE}-linux-amd64
sudo mv etcd etcdctl /usr/local/bin
Create data directory...e.g.
mkdir /opt/snapshots

NOTE: Run a local test to the controller by supplying a local data directory and the controller IP address or

TCDCTL_API=3 etcdctl --endpoints=https://CONTROLLER IP:2379 --cacert=/home/ziggy/etcd-backup/keys/ca.crt --cert=/home/ziggy/etcd-backup/keys/server.crt --key=/home/ziggy/etcd-backup/keys/server.key snapshot save /CONTROLLER Directory...e.g.../opt/snapshots/


Step 2. Ensure ETCD and ETCDCTL are installed on your ETCD backup host.

Same as above


Step 3. Create Endpoint for remote host and enroll/register with your NetFoundry network.



Step 4. Create a service for the etcd backup service. The endpoint hosting the service will be the DaemonSet endpoint and the IP will be retrieved from the controller with the following command:


kubectl get pods --all-namespaces -o wide






Step 5. Create AppWAN -- For your remote host to access the cluster. Hit + within the AppWAN menu.



Step 6 Perform Backup of etcd to remote host. Backup is initiated from the remote host.

Notice the connection to Private IP of ETCD container and the resulting remoteest3.db saved in desired location.


ziggy@ALIENWARE:~$ ETCDCTL_API=3 etcdctl --endpoints= --cacert=/home/ziggy/etcd-backup/keys/ca.crt --cert=/home/ziggy/etcd-backup/keys/server.crt --key=/home/ziggy/etcd-backup/keys/server.key snapshot save ~/etcd-backup/remotetest3.db
{"level":"info","ts":1660050320.816254,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"/home/ziggy/etcd-backup/remotetest3.db.part"}
{"level":"info","ts":"2022-08-09T09:05:21.115-0400","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1660050321.116608,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":""}
{"level":"info","ts":"2022-08-09T09:05:21.876-0400","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
{"level":"info","ts":1660050321.8901336,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"","size":"3.2 MB","took":1.0726409}
Snapshot saved at /home/ziggy/etcd-backup/remotetest3.db






Was this article helpful?
2 out of 3 found this helpful



Please sign in to leave a comment.