Demonstrating Ziti Tunnelers Across Multi-Cloud Kubernetes Use Cases

Follow

This article explores the deployment and use of various tunnelers in a multi-cloud Kubernetes environment, highlighting practical scenarios and step-by-step setup instructions. The architecture demonstrates how different Ziti tunneler modes can be leveraged to securely connect workloads across on-premises, Azure AKS, and AWS EKS clusters.

Architecture Diagram

 

Scenario

A company operates a K3s Kubernetes cluster on-premises, where all pods need to access a frontend web application hosted on Azure AKS. The frontend app, in turn, connects to a CouchDB database running on an EKS. The goal is to securely enable these connections using OpenZiti tunnelers, each tailored to the specific connectivity and security requirements of the components involved.

Solution Components

• On-Premises K3s Cluster: Uses Ziti Edge Tunneler as a node DaemonSet to provide network-wide access via DNS.

• Azure AKS Frontend: Deploys a frontend web app with a Ziti tunneler sidecar for bi-directional connectivity to the database.

• AWS EKS Database: Runs CouchDB with Ziti Host tunneler in hosting mode, restricting access to only the frontend app.

1. On-Premises K3s Cluster Setup

Objective:
Enable all pods on the on-prem K3s cluster to securely access the frontend app on Azure AKS via the Ziti network.

Steps:

1. Create Ziti Identity:
   Generate an identity on the Management Orchestration Platform (MOP) for enrollment with Ziti Edge Tunneler (ZET).

2. Deploy Ziti Edge Tunneler as DaemonSet:
   Use Helm to install ZET across all nodes:

   helm repo add openziti https://docs.openziti.io/helm-charts/
   helm pull openziti/ziti-edge-tunnel
   helm install ziti-edge-tunnel openziti/ziti-edge-tunnel --set-file zitiEnrollToken=NODE-PROXY-K3S.jwt --set pvc.accessMode=ReadWriteOnce

   After successful enrollment, the identity will appear online in the Ziti console.

   

3. Configure DNS Resolution:
   To resolve Ziti domains within pods, customize CoreDNS. For a single-node cluster, use the following ConfigMap:

   Note: If the on-prem K3S has multiple worker nodes, configure node-local-dns, as detailed in the node proxy/daemonset guide. Each worker node's pods must resolve Ziti DNS locally to avoid receiving incorrect Ziti intercept addresses from other nodes

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: coredns-custom
     namespace: kube-system
   data:
     ziti.server: |
       ziti.aks {  # replace with your domain
         forward . 100.64.0.2
       }

4. Apply and restart CoreDNS:

   kubectl apply -n kube-system -f coredns-custom.yml
   kubectl rollout restart -n kube-system deployment/coredns

5. Deploy a Browser Pod for Testing:
   Launch a Firefox browser pod to access the web app from on-prem:
   Firefox Browser Helm Chart

   helm repo add firefox-browser https://mrnim94.github.io/firefox-browser-helm
   helm install my-firefox-browser firefox-browser/firefox-browser --version 1.1.0

2. Azure AKS Frontend Setup

Objective:
Deploy a frontend web application with a Ziti tunneler sidecar to enable secure, bi-directional communication with the CouchDB database.

Steps:

1. Create Ziti Identity:
   Generate an identity on MOP for the sidecar tunneler.

2. Enroll Identity and Create Kubernetes Secret:
   Download and install Ziti CLI, then enroll the identity. Click here for the reference.

   ziti edge enroll SIDECAR-PROXY-AKS.jwt
   kubectl create secret generic "sidecar-client-identity" --from-file=/tmp/SIDECAR-PROXY-AKS.json

3. Configure DNS for the Pod:
   Find the CoreDNS cluster service IP:

   kubectl --namespace kube-system get services kube-dns

   Update the deployment manifest to use this IP as a fallback resolver for non-Ziti DNS queries.

4. Deploy the Frontend Application with Ziti Sidecar:
   Use the following manifest (update DNS and identity details as needed): Apply the frontend_with_ziti_tunnel.yml manifest, which contains the frontend app with Ziti tunnel, and update the details below to match your setup before deployment. The frontend app includes a simple web UI that checks and displays the connectivity status to the CouchDB database

apiVersion: apps/v1
kind: Deployment
metadata:
 name: connectivity-checker
spec:
 replicas: 1
 selector:
   matchLabels:
     app: connectivity-checker
 template:
   metadata:
     labels:
       app: connectivity-checker
   spec:
     containers:
     - name: nginx  # frontend app container
       image: nginx:alpine
       ports:
       - containerPort: 80
       volumeMounts:
       - name: html
         mountPath: /usr/share/nginx/html
       - name: nginx-conf
         mountPath: /etc/nginx/nginx.conf
         subPath: nginx.conf
     - name: ziti-tunnel  # ziti-tunnel container
       image: openziti/ziti-tunnel
       args: ["tproxy"]
       env:
       - name: ZITI_IDENTITY_BASENAME
         value: SIDECAR-PROXY-AKS  # the filename in the volume is SIDECAR-PROXY-AKS.json
       volumeMounts:
       - name: sidecar-client-identity
         mountPath: /netfoundry
         readOnly: true
       securityContext:
         capabilities:
           add:
           - NET_ADMIN
     dnsPolicy: None
     dnsConfig:
       nameservers:
         - 127.0.0.1   # all containers in pod must try this nameserver first
         - 10.0.0.10  # change this to match the CoreDNS cluster service address you found in the first step
       searches:
         - cluster.local  # you must supply all search domains when overriding cluster DNS
     restartPolicy: Always
     volumes:
     - name: html
       configMap:
         name: connectivity-checker-html
     - name: nginx-conf
       configMap:
         name: connectivity-checker-nginx
     - name: sidecar-client-identity
       secret:
         secretName: sidecar-client-identity
---
apiVersion: v1
kind: ConfigMap
metadata:
 name: connectivity-checker-nginx
data:
 nginx.conf: |
   worker_processes 1;
   events { worker_connections 1024; }
   http {
     # Use Kubernetes DNS — check CoreDNS IP via `kubectl get svc -n kube-system`
     resolver 127.0.0.1 10.0.0.10 valid=10s ipv6=off;
     server {
       listen 80;
       server_name _;
       location / {
         root /usr/share/nginx/html;
         index index.html;
       }
       # Dynamic backend — won't crash if DB not yet deployed
       location /db {
         set $backend "http://couchdb.ziti:5984"; #ziti intercept that will look for db
         proxy_pass $backend;
         proxy_connect_timeout 1s;
         proxy_read_timeout 2s;
         proxy_intercept_errors on;
         error_page 502 503 504 = @offline;
       }
       # Fallback response when DB is down
       location @offline {
         default_type application/json;
         return 200 '{"status":"offline"}';
       }
     }
   }
---
apiVersion: v1
kind: ConfigMap
metadata:
 name: connectivity-checker-html
data:
 index.html: |
   <!DOCTYPE html>
   <html>
   <head>
     <title>DB Connectivity Checker</title>
     <style>
       body { font-family: sans-serif; text-align: center; padding-top: 50px; }
       .status { font-size: 1.5em; margin-top: 20px; }
       .online { color: green; }
       .offline { color: red; }
     </style>
   </head>
   <body>
     <h1>CouchDB Connectivity Checker</h1>
     <div class="status" id="status">Checking...</div>
     <script>
       async function checkDB() {
         try {
           const res = await fetch('/db/');
           const data = await res.json();
           if (data.status === 'offline') {
             document.getElementById('status').innerHTML = 'CouchDB is not reachable';
             document.getElementById('status').className = 'status offline';
           } else {
             document.getElementById('status').innerHTML = 'CouchDB is reachable';
             document.getElementById('status').className = 'status online';
           }
         } catch (e) {
           document.getElementById('status').innerHTML = 'CouchDB is not reachable';
           document.getElementById('status').className = 'status offline';
         }
       }
       checkDB();
       setInterval(checkDB, 5000);
     </script>
   </body>
   </html>

---
apiVersion: v1
kind: Service
metadata:
 name: connectivity-checker
spec:
 selector:
   app: connectivity-checker
 ports:
 - port: 80
   targetPort: 80

After deployment, the identity should appear online.

3. AWS EKS Database Setup

Objective:
Deploy CouchDB on AWS EKS and expose it securely using Ziti Host tunneler in hosting mode.

Steps:

1. Create Ziti Identity:
   Generate an identity on MOP for the Ziti Host tunneler.

2. Deploy Ziti Host Tunneler:
   Install Ziti Host using Helm:

   helm repo add openziti https://docs.openziti.io/helm-charts/
   helm repo update
   helm install ziti-host openziti/ziti-host --set-file zitiEnrollToken=REVERSE-PROXY-AWS.jwt

   After deployment, the identity should appear online.

   

3. Deploy CouchDB:
   Install CouchDB using the official Helm chart:

   helm repo add couchdb https://apache.github.io/couchdb-helm
   helm repo update
   helm install couchdb --version=4.6.2 --set couchdbConfig.couchdb.uuid=decafbaddecafbaddecafbaddecafbed couchdb/couchdb

4. Service Creation and Connectivity Testing

Service Creation:

Creating a simple service on the V8 network (e.g., from NODE-PROXY-K3S to SIDECAR-PROXY-AKS, and from SIDECAR-PROXY-AKS to REVERSE-PROXY-AWS) automatically generates the necessary access policies and configurations.

AKS-SERVICE:

• Create a simple service to establish access from on-prem to AKS in our demo, connecting NODE-PROXY-K3S to SIDECAR-PROXY-AKS as shown below.

• As mentioned, configurations and service policies will be created automatically

Testing:

• Let's test the connectivity. I accessed the Firefox browser pod at on-prem using port forwarding and successfully reached the frontend web app. The CouchDB is unreachable, which is expected since we have not created a service to access the database.

AWS Service:

• Create a service to allow the frontend app to connect to CouchDB. In this demo, that means connecting SIDECAR-PROXY-AKS Identity to REVERSE-PROXY-AWS

• Service-related configs and policies are created automatically as shown below

Testing:

• Refresh the browser. The front-end app now communicates with CouchDB as shown below

Conclusion

This demonstration showcases how ziti tunnelers can be effectively used to secure and simplify connectivity across distributed, multi-cloud Kubernetes environments. By leveraging different tunneler modes—DaemonSet, sidecar, and host—you can tailor access and security to the needs of each component, ensuring robust, policy-driven connectivity without exposing services to the public internet.

References:

• OpenZiti Helm Charts

• Firefox Browser Helm Chart

• CouchDB Helm Chart

• OpenZiti Releases