Securing Azure AKS API Service with OpenZiti Module and NGINX

 

Introduction:

In this guide, we are going to walk through the steps that need to be taken to deploy Ziti Nginx Module onto an NGINX server hosted in Azure. The typical use case for this is an existing deployment where there is a reverse proxy already deployed (i.e. NGINX Server) in the cloud. All clients requests are proxied through the NGINX and then forwarded to backend servers. The proxy is typically exposed to the internet, which means it is open to security threats/breaches, DDoS attacks, etc. Having a module like Ziti Nginx with the NetFoundry CloudZiti network will mitigate these threats by making access to the API service private.

The article will cover steps to build the NGNIX module, deploy it onto Nginx Server, create a Service on CloudZiti to test connectivity to the backend service such as the AKS Service via a zero trust, private overlay of CloudZiti. The guide considers a "Teams / Growth" plan. 

Prerequisites:

Architecture:

nginx_module_arch.svg

The diagram above depicts the architecture of the API Service hosted on Kubernetes behind a NGNIX Reverse Proxy in Azure, where a client machine with ZDE installed can securely reach it via NGNIX  OpenZiti Module.

Initial Checks:

You will need the following:
  1. Azure account and permissions to create resources via Terraform.
  2. Azure Cli installed
  3. Go installed. We suggest the latest versions
  4. Terraform installed
Let's run quick commands to ensure we have everything we need installed on the client ( i.e. machine from where the Azure Services  can be reached securely via CloudZiti network) 
> az version
{
"azure-cli": "2.43.0",
"azure-cli-core": "2.43.0",
"azure-cli-telemetry": "1.0.8",
"extensions": {}
}
> go version
go version go1.19.1 windows/amd64
> terraform --version
Terraform v1.3.9

 

Once the proper permissions in Azure are verified and compatible versions of Azure Cli, Golang and Terraform are installed, go ahead and clone Nginx Module Solution Repo locally

 

Pre-requisites :

1. The edge routers and endpoints need to reach the NetFoundry controller and NF-hosted edge routers for registration and operations. Please make sure to have the required ports, IPs, and URLs reachable if you have firewall ACL policies.

2. NetFoundry traffic should be bypassed from any deep packet inspection in between since our solution involves mTLS and E2E encryption. 

The guide has the details. 

https://support.netfoundry.io/hc/en-us/articles/4402361752717-Firewall-Requirements

________________________________________________________________________________________________________________

 

Step 1. Create your NetFoundry account

If you are an existing customer having a NetFoundry Cloud account, proceed to step 2. Else get an account.

Step 2 : Create your NF hosted public routers: 

Once the network is up (i.e. network controller only), navigate to the edge router page and click on the help message at the bottom of the page. Enter an address that matches the region you provided in the previous step and the fabric, i.e. 2 Public Edge Routers, will be deployed for you along with the Edge Router Policy.
create_fabric.png

 

Note:  For "Enterpriser Plans" you can create your NF hosted public edge routers in your region and cloud provider of choice.

 

Step 3: Create Server and Client Endpoints

To create endpoints, navigate to Endpoints in the navigation panel to the left and click + to create a new endpoint. Let's name the client application client-nginx and assign Attribute: #clients. Repeat this step for the server module application server-nginx and Attribute: #servers.
Refer to the support guide on how to create and manage endpoints.

Step 4: Deploy Nginx Server and AKS Cluster Infrastructure 

Now that we have the enrollment tokens for Nginx Module and Client Go Apps, we can go ahead and deploy the infrastructure in Azure using terraform. The templates included in this project will create the following resources (and their associated resources):
  • Virtual Network
  • AKS Private Cluster with Kubenet CNI
  • Nginx Server
  • Security Group with only SSH port open to Internet
You will need to set up a few environmental variables. Deploy infrastructure by running the following commands:
Important Note: By default, the terraform expects the ssh public key to be in the home directory (i.e. "~/.ssh/id_rsa.pub"). You can pass a custom path using -var ssh_public_key="your/path/pub_key"

 

Linux Client

export ARM_SUBSCRIPTION_ID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
export ARM_CLIENT_ID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
export ARM_CLIENT_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
export ARM_TENANT_ID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

Windows Client

$env:ARM_SUBSCRIPTION_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
$env:ARM_CLIENT_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
$env:ARM_CLIENT_SECRET = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$env:ARM_TENANT_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
git clone https://github.com/netfoundry/ziti-cookbook.git
cd ziti-cookbook/NginxModuleSolution/terraform/tf-provider
terraform init
terraform plan -out aks
terraform apply "aks"

Wait for resources to be deployed successfully...

Once completed, grab  cluster_private_fqdn and nginx_public_ip_address under "outputs" as shown in the example below. 

Apply complete! Resources: 8 added, 0 changed, 0 destroyed.

Outputs:

cluster_name = "akssandeastus"
cluster_private_fqdn = "akssand-e3b5e367.97690473-9b65-433e-aa79-76cc7685d852.privatelink.eastus.azmk8s.io"
nginx_public_ip_address = [
"172.174.95.189",
]

Step 5 - Build and Deploy Nginx Ziti Module

  • Upload nginx-server.jwt file to the nginx server.
scp -i "ssh private key" ../../files/identity_tokens/server-nginx.jwt ziggy@${nginx_public_ip_address}:
  • Login to Nginx Server host using ssh.
ssh -i "ssh private key" ziggy@${nginx_public_ip_address}

Important Note: The package updates listed in the next code snippet will be performed by the cloud-init script. They are posted here just incase they are not executed during the initial boot-up for any reason. You can skip the next step and only come back to it if "Build the module" step fails due to packages not present.

  • Update all existing and add essentials packages for the build process. Note: May need to click ok on messages during the install to leave current defaults on
sudo apt update && sudo apt upgrade -y 
sudo apt install libpcre3-dev libz-dev libuv1-dev cmake build-essential jq -y
sudo reboot # if required
  • Build the module

Note: Steps copied from cmake build. One can always check if the steps have changed since the publishing of this article.

git clone https://github.com/openziti/ngx_ziti_module.git; cd ngx_ziti_module/; mkdir cmake-build; cd cmake-build
cmake ../
make
  • Copy the module to the Nginx's modules folder
sudo cp ngx_ziti_module.so /etc/nginx/modules/
ls /etc/nginx/modules/
ngx_ziti_module.so
  • To enable the ziti module in the configuration file, run the command as shown in the the next code block to replace the existing content. 

Important Note: The identity path points to the nginx directory. The identity file will be moved there after it is enrolled. Don't forget to replace ${cluster_private_fqdn} with your own. Also, the http configuration block is the default configuration that comes with nginx. One can try to access port 80 from the internet to test if it is exposed or not, i.e. curl http://${nginx_public_ip_address}.

sudo tee /etc/nginx/nginx.conf << EOF

load_module modules/ngx_ziti_module.so;

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log debug;
pid        /var/run/nginx.pid;

thread_pool ngx_ziti_tp threads=32 max_queue=65536;

events {
  worker_connections  1024;
}

ziti identity1 {
  identity_file /etc/nginx/server-nginx.json;
  bind nginx-service-01 {
      upstream ${cluster_private_fqdn}:443;
  }
}

http {
  include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

  sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

  include /etc/nginx/conf.d/*.conf;
}
EOF

 

Step 6 - Enroll Nginx Module Identity to the network

  • Install Ziti Cli first
cd ~
wget $(curl -s https://api.github.com/repos/openziti/ziti/releases/latest | jq -r .assets[].browser_download_url | grep "linux-amd64")
tar -xf $(curl -s https://api.github.com/repos/openziti/ziti/releases/latest | jq -r .assets[].name | grep "linux-amd64") 'ziti/ziti' --strip-components 1; rm $(curl -s https://api.github.com/repos/openziti/ziti/releases/latest | jq -r .assets[].name | grep "linux-amd64")
  • Enroll Server Identity
sudo ./ziti edge enroll -j server-nginx.jwt -o /etc/nginx/server-nginx.json
  • Restart the Nginx process
sudo systemctl restart nginx.service
  • Check if Ziti Identity initialized successfully
cat /var/log/nginx/error.log
...
...
2022/12/28 21:28:59 [debug] 16347#16347: enter: ngx_ziti_init_process
2022/12/28 21:28:59 [warn] 16347#16347: initializing block identity1
(16347)[        0.000]    INFO ziti_log_set_level set log level: root=1

Step 7 - Build Client App 

Note: May need to install gcc compiler on windows tdm-gcc

git clone https://github.com/openziti-test-kitchen/kubeztl; cd kubeztl; mkdir build; go mod init kubeztl; go mod tidy; go build -o build ./...
  • Test Client app - kubeztl
build/kubeztl version -o json 
ERROR failure to post auth Post "https://b5767e83-dd46-42f7-bb80-32712f35bfe8.production.netfoundry.io:443/edge/client/v1/authenticate?method=cert": dial tcp: lookup b5767e83-dd46-42f7-bb80-32712f35bfe8.production.netfoundry.io: no such host
{
"clientVersion": {
"major": "",
"minor": "",
"gitVersion": "v0.0.0-master+$Format:%H$",
"gitCommit": "$Format:%H$",
"gitTreeState": "",
"buildDate": "1970-01-01T00:00:00Z",
"goVersion": "go1.19.1",
"compiler": "gc",
"platform": "windows/amd64"
},
"kustomizeVersion": "v4.5.7"
}
Unable to connect to the server: failed to dial: no apiSession, authentication attempt failed: Post "https://b5767e83-dd46-42f7-bb80-32712f35bfe8.production.netfoundry.io:443/edge/client/v1/authenticate?method=cert": dial tcp: lookup b5767e83-dd46-42f7-bb80-32712f35bfe8.production.netfoundry.io: no such host
If you get the error as shown above, then the client app was built successfully.

Step 8 - Client Identity Enrollment to the network

One way to enroll the client identity is to use ZDE and skip this step. Here is the link to the endpoint installation and enrollment guide. 

Important Note: If your are using a linux client, you can follow the same commands as described for the nginx module above in Enroll NGINX Module Identity section

  • Install Ziti Cli on Windows using powershell
wget $($(Invoke-RestMethod -uri https://api.github.com/repos/openziti/ziti/releases/latest).assets |  where-object { $_.name -match "windows"}).browser_download_url -UseBasicParsing -OutFile $($(Invoke-RestMethod -uri https://api.github.com/repos/openziti/ziti/releases/latest).assets |  where-object { $_.name -match "windows"}).name

Expand-Archive
-LiteralPath $($(Invoke-RestMethod -uri https://api.github.com/repos/openziti/ziti/releases/latest).assets |  where-object { $_.name -match "windows"}).name -DestinationPath ./; Move-Item -Path ziti/ziti.exe -Destination ./; Remove-Item -Path ziti/ -Recurse -Force

rm
$($(Invoke-RestMethod -uri https://api.github.com/repos/openziti/ziti/releases/latest).assets |  where-object { $_.name -match "windows"}).name
  • Enroll Client Identity
./ziti edge enroll -j files/identity_tokens/client-nginx.jwt -o files/identity_tokens/client-nginx.json

Step 9 - Create Service and AppWans: 

In order for Nginx Module to be recognized on the fabric, it must be hosted as Service.
For this, we'll create a Simple Service. On the nfconsole.io dashboard, select Services and + on the top right. Then select Simple Service and Create Service. We'll name the service nginx-service-01 and give it the service attribute nginx-services and leave the edge router attributes as default.
This service will be hosted on the sdk embedded endpoint, i.e nginx module. Thus, intercept host information is irrelevant in our case, but it needs to be filled in the current UI version. Use name=none.io, port 1 and enter #servers group tag in the hosted endpoints field. Select Yes to choose Native Application SDK Based Option.

create_service_nginx.png

As mentioned before, in order to allow our local Endpoint (kubeztl) to access our hosted Endpoint (nginx module), we'll need to assign them to the same AppWAN. Under AppWANs on our nfconsole.io dashboard, click the plus sign to create a new AppWAN. Let's name it nginx-appwan with our service attribute nginx-services and endpoint attributes clients.
create_appwan_nginx.png

## It's Alive!! ##

Everything should be connected as expected now. Let's look again to ensure everything is as expected.
Now the event history should look as follows:
management_events_nginx.png

 

Step 7 : Let's test our service

We will use our zitified kubectl client to interact with the AKS API Control Plane to list, create, delete context, containers, etc.
Note: ZDE can also be used along with Kubectl, i.e. not zitified version to test this service.
  • Configure your local kube configuration file using azure cli
az login # if not already logged in
# Windows
$env:RG_NAME = 'resource group name'
$env:ARM_SUBSCRIPTION_ID =  'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
az aks get-credentials --resource-group $env:RG_NAME --name {cluster_name} --subscription $env:ARM_SUBSCRIPTION_ID
# Linux
$RG_NAME = 'resource group name'
$ARM_SUBSCRIPTION_ID =  'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
az aks get-credentials --resource-group $RG_NAME --name {cluster_name} --subscription $ARM_SUBSCRIPTION_ID
  • Change directory to kubeztl and check the installed context in the config file
cd kubeztl
build/kubeztl config  get-contexts
CURRENT   NAME            CLUSTER         AUTHINFO                                      NAMESPACE
*         akssandeastus   akssandeastus   clusterUser_<$RG_NAME>_akssandeastus
  • Let's check the status of nodes in the cluster. One should be expected!
build/kubeztl get nodes
ERROR   Service Name not provided # if we dont provide ziti service name and identity configuration file path

build/kubeztl get nodes --zConfig ziti-cookbook/NginxModuleSolution/files/identity_tokens/client-nginx.json --service nginx-service-01
NAME                                STATUS   ROLES   AGE    VERSION
aks-agentpool-26146717-vmss000000   Ready    agent   153m   v1.23.12

build/kubeztl cluster-info --zConfig ziti-cookbook/NginxModuleSolution/files/identity_tokens/client-nginx.json --service nginx-service-01
Kubernetes control plane is running at https://akssand-2887a9f2.eb3aa8fb-333a-47c3-b657-61035c4c4f98.privatelink.eastus.azmk8s.io:443
addon-http-application-routing-nginx-ingress is running at http://20.246.189.101:80 http://20.246.189.101:443
CoreDNS is running at https://akssand-2887a9f2.eb3aa8fb-333a-47c3-b657-61035c4c4f98.privatelink.eastus.azmk8s.io:443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://akssand-2887a9f2.eb3aa8fb-333a-47c3-b657-61035c4c4f98.privatelink.eastus.azmk8s.io:443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

***Dark Access enabled !!!***

  • Another test if you are up for it!
To access this app once deployed, you can follow sidecar solution
  • Deploying Helloworld App.
build/kubeztl  --zConfig ziti-cookbook/NginxModuleSolution/files/identity_tokens/client-nginx.json --service nginx-service-01 apply -f ziti-cookbook/NginxModuleSolution/files/templates/helloworld.yaml
deployment.apps/helloworld created

build/kubeztl  get pods --zConfig ziti-cookbook/NginxModuleSolution/files/identity_tokens/client-nginx.json --service nginx-service-01
NAME                         READY   STATUS    RESTARTS   AGE
helloworld-ff5c98fbf-xq5w2   1/1     Running   0          2m11s

build/kubeztl  --zConfig ../client-nginx.json --service nginx-service-01 get pods -o wide
NAME                         READY   STATUS    RESTARTS   AGE     IP           NODE                                NOMINATED NODE   READINESS GATES
helloworld-ff5c98fbf-xq5w2   1/1     Running   0          4m36s   10.17.1.16   aks-agentpool-26146717-vmss000000   <none>           <none>
  • Deleting Helloworld App
build/kubeztl  delete deployment helloworld --zConfig ziti-cookbook/NginxModuleSolution/files/identity_tokens/client-nginx.json --service nginx-service-01
deployment.apps "helloworld" deleted

build/kubeztl  --zConfig ziti-cookbook/NginxModuleSolution/files/identity_tokens/client-nginx.json --service nginx-service-01 get pods -o wide
No resources found in default namespace.
Was this article helpful?
1 out of 1 found this helpful