AWS Cloud - Ingress High Availability

Follow

Overview

The network load balancer that can be deployed in the AWS Cloud fits the architecture and features we are looking for as described in this article Ingress High Availability in Public Clouds . More details on GLB  features can be found in AWS Documentation at this link for those interested https://docs.aws.amazon.com/elasticloadbalancing/latest/gateway/introduction.html. 

Important Notes:

The GLB in AWS uses the geneve protocol to provide transparency through the load balancer to the backend servers. This means that the backend servers must be able to handle the geneve decapsulation before the clients packets (i.e. inner IP) are forwarded to the application.

Our implementation uses ebpf program that is attached to the interface at the Traffic Control (TC) subsystem, which has a function that checks for IP/UDP headers and Destination Port 6081. When the UDP Header Type and Port 6081 are detected, then it will remove the outer ip/udp/geneve headers if the geneve header verification is a success. Finally, the inner ip packet is handed over to the socket lookup function to find out if  there is a valid listen socket for that service. If not, the packet is dropped, otherwise the packet is forwarded.

Important to note is that the program does not keep any geneve header information, so it does not send the replies back to the load balancer to handle the return traffic. The replies are directly forwarded to clients. This means that the load balancer and clients need to be in the same VPC. We don't support inter VPC or Account connections at this time. The bidirectional communication with the load balancer over the geneve tunnel is on the roadmap and can be implemented if there is real interest and demand for it.

Here is the link to the README for source code (zfw) if you are interested to find out more details about it. By the way, it does a lot more than just the geneve header decapsulation. Furthermore, if an openziti user would like to deploy this on the openziti network, it will work there too. Please follow README. If you run into issues and need some guidance, you can reach us on the openziti discourse.

Configuration Steps to enable EBPF on CloudZiti Edge Routers:

Important Note: 
The minimum Network Version needed to run Diverter on Edge Routers is 
- OpenZiti version: 0.27.3
- CloudZiti version: 7.3.83
Edge Router Image Version needed is
$ cat /etc/nfrelease 
2.1.20230614 or higher

Additionally, follow direction to deploy health checker script to enhance 
the fabric/network failure detections. More details can be found at this link

Lastly, if any of the services are configured as FQDNs and/or Wildcard Urls, 
one needs to configured a single resolver address across all backend routers
to simplify and speed up the name resolution. Best time to do that is at the router 
registration. Run these:
sudo /opt/netfoundry/set-ip.sh -tp=static -ip=100.127.255.254 -pr=32 -in="lo" -f
sudo router-registration --dnsIPRange 100.64.0.0/13 --lanIf enp0s5 \
                         --tunnel_ip 100.127.255.254 {{ edgeRouterKey }}
dnsIPRange will be different for each BE Router and not overlapping, i.e. 
BE01 = 100.64.0.0/13 (~ 0.5M addresses)
BE02 = 100.72.0.0/13

1. Follow AWS Router Deployment to deploy ER from the AWS Marketplace.
2. After the successful deployment and registration with CloudZiti Network, login into it with ssh.
3. Run diverter-update to upload the latest binaries.
4. Then, you run diverter-enable command

Additional Details for Diverter can be found at this link

$ diverter-update
(Reading database ... 109608 files and directories currently installed.)
Preparing to unpack /tmp/zfw.deb ...
Unpacking zfw-router (0.4.2) over (0.4.2) ...
Setting up zfw-router (0.4.2) ...
Not enough privileges or ebpf not enabled!
Run as "sudo" with ingress tc filter [filter -X, --set-tc-filter] set on at least one interface

$ diverter-enable
Detected Netfoundry install/registration!
Installing symlink from /opt/openziti/ziti-router to /opt/netfoundry/ziti/ziti-router!
Attempting to add ebpf ingress to: enp0s5
Ebpf not running no maps to clear
tc parent add : enp0s5
Set tc filter enable to 1 for ingress on enp0s5
Attached /opt/openziti/bin/zfw_tc_ingress.o to enp0s5
Rule added
Rule added (v6)
Adding TCP mapping
Adding TCP mapping
lookup success
Adding TCP mapping
Adding UDP mapping
Successfully converted ziti-router.service. Restarting!

Configuration Steps for GLB:

1. Log into the AWS Cloud Console 

2. Navigate to Services → EC2 → Target Groups

3. Create Target Group as shown - edge router health checks are set by default to https, 8081, /health-checks 
4. 
5. Click Next and register the backend targets (i.e. edge routers) if they created prior to the GLB deployment, otherwise you can add them after.
6. Click on Create target group.

7. Navigate to Services → EC2 → Load Balancers

8. Click on “Create Load Balancer” and select “Gateway Load Balancer”

9. Fill in the basic configuration (ipv4 address is only supported) and click on "Create load balancer"  

10. If targets were not added in steps 6, you can navigate to Services → EC2 → Target Groups
11. Select the GLB created in step 7 and click on Targets Tab → Edit
12. Select your edge routers and click on Add to registered, then save
13. If the health checks reach the target edge routes successfully, then the target status should look like this
14. Now, you need to create endpoint connections to the subnets, where the clients live. Navigate to Services → EC2 → Load Balancers
15. Select the GLB created in step 7 and go to Integrated services tab
16. Click on Create Endpoint service
17. You can also navigate to Endpoint services as Services → VPC→ Endpoint services, click on Create endpoint service
18. Fill the required information and click on Create. Note the service id, you will need it later, i.e. com.amazonaws.vpce.us-east-2.vpce-svc-03f2f3fa067344454
19. Navigate to Allow principals Tab and click on Allow principals
20. Add the appropriate service permissions, i.e. "arn:aws:iam::aws-account-id:user/*"
21. Then select Endpoint connections Tab and click on Create endpoint connections
22. Fill in the configurations, and click on Create Endpoint
23. Repeat previous step for all the subnets that you are deploying this connection into.
24. Navigate over to Endpoint services and click on Accept endpoint connection request for all the connections created in the previous step
25. Once the status is available on all connections, you can navigate over to the routing tables in your respective subnets and create routes with the next hop being these connections listed here. Note endpoint ids, you will need them to create routes.
26. Create routes to GLB from Clients
27. Repeat the previous step for every subnet that the connection was created into.
28. If FQDNs and Wildcard Urls are used, one needs to configure routes to reach the DNS Resolver through GLB and direct static routes to each edge routers.
29. Done!