Ingress High Availability in Public Clouds

Follow

Overview

As customers extend Ziti network into all public clouds, we obviously prefer they would use SDKs and embed Ziti into their cloud native apps. This is not always possible for various reasons. Since Ziti Network is highly distributed by design, the ingress to the network from not native Ziti Apps must be as well.

The preferred architecture for this is to be cloud native and utilize tools/platforms available in various clouds, more specifically network load balancers. The following requirements are the desired attributes of cloud load balancers.

1. Network type, i.e. L3/L4

2. Transparent - doesn't close or originate flows or interact with the payload of the flow and protocol handshakes always occur directly between the client and the back-end pool vm

3. The original source IP address is preserved

4. TCP and UDP support using the same front end private IP

5. Hash Algorithm:

   1. Five-Tuple (source IP and port, destination IP and port, protocol)

   2. Three-Tuple (source IP, destination IP, protocol) - session affinity, i.e. more predictable to use the same ER for network ingress per protocol

   3. Two-Tuple (source IP, destination IP) - same as (b) but no protocol dependency

6. Backend VM Weight - desirable

7. Health Check Types - configurable destination port, interval, retries, timeout for all types

   1. http (optional body look-up)

   2. tcp (required at the very least)

   3. udp (optional)

Notes

In Ziti v.0.20.10, the router health check endpoint was introduced. This can be used to check on the Edge Router connectivity status to the controller. i.e. ER online or offline as seen by the controller. We also tested using a local service that would originate and terminate on the same local ER as the health check endpoint. The idea for that was to verify the data plane between the tunneler and the edge router as seen by sessions originating on the local tunneler. This part may not be as critical for the tunneler/router combo for the simple fact that the code is integrated now. 

Here is a simple health check configuration option using the existing edge router health checks.

1. protocol option = https

2. port option = 8081

3. path option = /health-checks

DNS Resolution

If destinations in Ziti services are referenced by FQDNs, hostnames or sub/top domain wildcard names, then DNS resolution requests are only needed to be forwarded to the network load balancers (nlb) and not the client sessions.  Each router's resolver must have a unique IP Pool configured as well, i.e. ER1's IP Pool = 100.64.0.0/13, ER2's IP Pool = 100.72.0.0/13, etc. The default is set to 100.64.0.0/10 across all routers. In this example, this means that /10 prefix is divided into into 8 unique  /13 Prefix Pools that can be used with up to 8 BE Routers. It can be less or more Prefix Pools based on the BE Router count and number of services required. The router registration command can be used to change the default pool prefix as follows at the router registration:

AZ1

sudo router-registration --dnsIPRange 100.64.0.0/13 ${Registration Short Key}
static route needed 
 - 100.64.0.0/13 via AZ1 Edge Router's IP

AZ2

sudo router-registration --dnsIPRange 100.72.0.0/13 ${Registration Short Key}
static route needed 
 - 100.72.0.0/13 via AZ2 Edge Router's IP

Client Stub Resolver

Note: This works with Azure Load Balancer, other clouds would need to follow the Single Ziti Resolver section

nameserver NLB Front IP

Single Ziti Resolver

Since not all NLBs have ability to proxy dns requests addressed to their front IP, most would need the same revolver IP to be configured across all BEs and a static route set up to reach it through NLBs as indicated below.

Note: AWS GLB uses Geneve protocol to tunnel packets through GLB, and one would need to enable ebpf on BE ERs. Pleases see AWS GLB for more details

AZ1

LANIF="AZ1 Edge Router Intf"
sudo set-ip.sh -tp=static -ip=100.127.255.254 -pr=32 -in="lo" -f
sudo router-registration --dnsIPRange 100.64.0.0/13 --tunnel_ip 100.127.255.254 --lanIf $LANIF ${Registration Short Key} 
static routes needed
 - 100.64.0.0/13 via AZ1 Edge Router's IP
 - 100.127.255.254 via NLB Front IP

AZ2

LANIF="AZ2 Edge Router Intf"
sudo set-ip.sh -tp=static -ip=100.127.255.254 -pr=32 -in="lo" -f
sudo router-registration --dnsIPRange 100.72.0.0/13 --tunnel_ip 100.127.255.254 --lanIf $LANIF ${Registration Short Key}
static routes needed
 - 100.72.0.0/13 via AZ2 Edge Router's IP
 - 100.127.255.254 via NLB Front IP

Client Stub Resolver

nameserver 100.127.255.254

 

Deployment Diagram