Introduction
With the deployment of openziti version 0.27.3, the new ingress service interception has been enabled called the diverter. The diverter is based on the ebpf technology, where a small restricted c-based program is attached to the ingress interface in the kernel space along with ebpf hash maps. The hash maps can be updated from the user space with the service prefixes and read by the diverter from the kernel space. The diverter makes service interceptions based on the entries in the hash maps.
Diverter Lifecycle
The diverter helper scripts have been created to help with the manageability of the ebpf program and maps after the CloudZiti managed Edge Router is deployed and registered. Here is the detail of the Diverter menu options, which can be viewed by running nfhelp from the router cli shell as shown below.
$ nfhelp
...
Diverter Commands:
diverter-enable - enable diverter ebpf program
diverter-disable - disable diverter ebpf program
diverter-status - check if diverter ebpf program is enabled
diverter-add-user-rules - add all user ingress rules to ebpf map configured in /opt/openziti/bin/user/user_rules.sh
diverter-update - update the diverter ebpf bytecode to latest version
diverter-trace - show ebpf trace logs
zfw - link to the zfw program used to manage ebpf map content
...
The diverter is not enabled by default and the binaries are not pre-loaded on the image. The binaries can be downloaded by running the following command:
diverter-update
The update command will upload all the necessary files associated with the diverter deployment. The helper functions were written to provide idempotency to the ebpf deployment, which means that every time they are run the same outcome should be produced. Thus, the update can be rerun anytime to update the diverter files if a new version becomes available.
Additionally, switching between the tproxy and diverter modes can be done. All the necessary changes will be preformed to switch between the two modes and the edge router will be restarted every time.
diverter-enable -----> ebpf mode is invoked
diverter-disable -----> tproxy with iptables is invoked
Command Line Options to Check Rules Status
CLI Commands zt-intercept and zt-firewall-rules automatically detect which mode is enabled and show the output sourced from the corresponding data source.
Support Commands:
...
zt-firewall-rules - print current filter table of NF-INTERCEPT chain - allow traffic inbound
zt-intercepts - print current mangle table of NF-INTERCEPT chain - intercept
...
Ebpf Mode Examples:
$ zt-intercepts
target proto origin destination mapping: interface list
-------- ----- ----------------- ------------------ ------------------------------------------------------- -----------------
TPROXY tcp 0.0.0.0/0 100.64.1.21/32 dpts=1000:1999 TPROXY redirect 127.0.0.1:39581 []
TPROXY tcp 0.0.0.0/0 100.64.1.21/32 dpts=2000:2999 TPROXY redirect 127.0.0.1:39581 []
TPROXY tcp 0.0.0.0/0 100.64.1.21/32 dpts=3000:3999 TPROXY redirect 127.0.0.1:39581 []
TPROXY tcp 0.0.0.0/0 100.64.1.21/32 dpts=4000:4999 TPROXY redirect 127.0.0.1:39581 []
TPROXY tcp 0.0.0.0/0 100.64.1.21/32 dpts=44300:44399 TPROXY redirect 127.0.0.1:39581 []
TPROXY tcp 0.0.0.0/0 100.64.1.21/32 dpts=8000:8999 TPROXY redirect 127.0.0.1:39581 []
TPROXY tcp 0.0.0.0/0 100.64.1.21/32 dpts=9000:9999 TPROXY redirect 127.0.0.1:39581 []
TPROXY tcp 0.0.0.0/0 100.64.1.21/32 dpts=50000:59900 TPROXY redirect 127.0.0.1:39581 []
...
TPROXY tcp 0.0.0.0/0 100.64.1.23/32 dpts=5000:5099 TPROXY redirect 127.0.0.1:39877 []
TPROXY tcp 0.0.0.0/0 100.64.1.23/32 dpts=6000:6099 TPROXY redirect 127.0.0.1:39877 []
TPROXY tcp 0.0.0.0/0 100.64.1.23/32 dpts=111:111 TPROXY redirect 127.0.0.1:39877 []
TPROXY tcp 0.0.0.0/0 100.64.1.23/32 dpts=30000:39900 TPROXY redirect 127.0.0.1:39877 []
Rule Count: 2301
prefix_tuple_count: 193 / 100000
$ zt-firewall-rules
target proto origin destination mapping: interface list
-------- ----- ----------------- ------------------ ------------------------------------------------------- -----------------
PASSTHRU udp 0.0.0.0/0 100.127.255.254/32 dpts=53:53 PASSTHRU to 100.127.255.254/32 []
PASSTHRU tcp 0.0.0.0/0 100.127.255.254/32 dpts=53:53 PASSTHRU to 100.127.255.254/32 []
PASSTHRU tcp 0.0.0.0/0 10.40.101.37/32 dpts=443:443 PASSTHRU to 10.40.101.37/32 []
PASSTHRU tcp 0.0.0.0/0 10.40.101.37/32 dpts=8081:8081 PASSTHRU to 10.40.101.37/32 []
Rule Count: 4
prefix_tuple_count: 193 / 100000
Custom Ingress FW Rules
The ingress firewall rules are entered based on the router configuration present in the edge router configuration file. Users can enter their own custom rules using zfw alias command.
Alias nfhelp command
zfw -I -c 192.168.1.108 -m 32 -l 8000 -h 8000 -t 0 -p tcp
Direct command
sudo /usr/sbin/zfw -I -c 192.168.1.108 -m 32 -l 8000 -h 8000 -t 0 -p tcp
If one wants these rules to survive reboot, they need to be added to a user script located in /opt/openziti/bin/user folder. The script may need to be created if not there already.
$ cat /opt/openziti/bin/user/user_rules.sh
#!/bin/bash
sudo /usr/sbin/zfw -I -c 192.168.1.108 -m 32 -l 8000 -h 8000 -t 0 -p tcp
Note:
More details about zfw can be found in the README file located in the zfw repo.
EBPF Useful Support Commands
Some useful commands to help you debug service issues.
$ diverter-status
lo: 1
--------------------------
icmp echo :1
verbose :0
ssh disable :0
per interface :0
tc ingress filter :0
tc egress filter :0
tun mode intercept :0
vrrp enable :0
eapol enable :0
--------------------------
enp0s5: 2
--------------------------
icmp echo :0
verbose :0
ssh disable :0
per interface :0
tc ingress filter :1
tc egress filter :0
tun mode intercept :0
vrrp enable :0
eapol enable :0
--------------------------
# increase verbosity of logs, i.e enable/disable debug mode
$ zfw -v enp0s5
Set verbose to 1 for enp0s5
$ zfw -v enp0s5 -d
Set verbose to 0 for enp0s5
# Using wireshark like capture on the interface, i.e. zfw --monitor (or -M) {interface name}
# can filter by using grep
$ zfw -M enp0s5
Aug 22 2023 15:50:05.605031204 : enp0s5 : INGRESS : TCP :10.40.101.54:45922 > 100.64.0.158:8080 | tproxy ---> 127.0.0.1:35439
Aug 22 2023 15:50:05.668845650 : enp0s5 : INGRESS : TCP :10.40.101.54:55518 > 100.64.0.172:8080 | tproxy ---> 127.0.0.1:35303
Aug 22 2023 15:50:05.730050538 : enp0s5 : INGRESS : TCP :10.40.101.54:54706 > 100.64.0.143:8080 | tproxy ---> 127.0.0.1:39607
Aug 22 2023 15:50:05.783907437 : enp0s5 : INGRESS : TCP :10.40.101.54:58462 > 100.64.0.79:8080 | tproxy ---> 127.0.0.1:44069
Aug 22 2023 15:50:05.894732847 : enp0s5 : INGRESS : TCP :10.40.101.54:34414 > 100.64.0.58:8080 | tproxy ---> 127.0.0.1:40739
Aug 22 2023 15:50:05.946104430 : enp0s5 : INGRESS : TCP :10.40.101.54:51212 > 100.64.0.174:8080 | tproxy ---> 127.0.0.1:35751
Aug 22 2023 15:50:06.235142043 : enp0s5 : INGRESS : TCP :10.40.101.54:55656 > 100.64.0.77:8080 | tproxy ---> 127.0.0.1:38985
Aug 22 2023 15:50:06.289569267 : enp0s5 : INGRESS : TCP :10.40.101.54:45878 > 100.64.0.153:8080 | tproxy ---> 127.0.0.1:43221
Aug 22 2023 15:50:06.341535673 : enp0s5 : INGRESS : TCP :10.40.101.54:49310 > 100.64.0.128:8080 | tproxy ---> 127.0.0.1:39731
Aug 22 2023 15:50:06.398646475 : enp0s5 : INGRESS : TCP :10.40.101.54:44668 > 100.64.0.51:8080 | tproxy ---> 127.0.0.1:43273
Aug 22 2023 15:50:06.572781760 : enp0s5 : INGRESS : TCP :10.40.101.54:59272 > 100.64.0.84:8080 | tproxy ---> 127.0.0.1:44471
Aug 22 2023 15:50:06.635721683 : enp0s5 : INGRESS : TCP :10.40.101.54:51044 > 100.64.0.25:8080 | tproxy ---> 127.0.0.1:34199
Aug 22 2023 15:50:06.690551210 : enp0s5 : INGRESS : TCP :10.40.101.54:48262 > 100.64.0.60:8080 | tproxy ---> 127.0.0.1:39409