NetFoundry makes it easy to instantly spin up highly secure, performant, edge, app or device -to-cloud networks for workloads in AWS. Our secure private overlays on the internet offer private, zero trust networking to S3 bucket and objects for apps, devices and users.
With NetFoundry, you can extend secure zero trust connections to S3 buckets and objects that are not public, following least privilege access and micro segmentation principles. Smart fabric from NetFoundry provides you with optimal latency routes for your apps. This ensures you can build highly secure and performant connectivity in minutes using cloud-native tools without the burden of a direct connect solution.
This guide explores how to leverage NetFoundry to establish secure access for your private Amazon S3 buckets from a python log pusher program running a Boto3 Client for S3. We'll establish the private overlay using a NetFoundry edge router in AWS and a NetFoundry Python SDK imported in the Py log pusher program. You can follow the approach for any Py based app or explore our SDKs for other programming languages. We would be exploring how to access the bucket and objects from a laptop that is running a NetFoundry Ziti Desktop Edge software client.
Getting started:
What you need to get started:
1. A NetFoundry Cloud account - Go through the steps to create a free trial account if you don't have one.
2. A network in your account with at least one public router. The articles will guide you through this process.
- How to create a network
- Provision a public router ( NetFoundry hosted)
- Firewall policy requirements to provide outbound only access to the NetFoundry network
Solution Architecture:
Setting up S3 bucket, VPC interface endpoint and policies in AWS:
A. S3 Bucket:
You can follow the steps outlined in the following document to create a S3 bucket.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html
Create the bucket that is not public
B. Interface VPC endpoint:
Create a new VPC or use an existing VPC to provision your VPC endpoint.
Create an interface VPC endpoint to controls access to S3 bucket. The S3 bucket would be reached within the VPC via the interface VPC endpoint.
Refer the following document for details on using interface VPC endpoint to access a S3 bucket - https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html
Interface VPC endpoint are represented by one or more elastic network interfaces (ENIs) that are assigned private IP addresses from subnets in your VPC which will be used to configure NetFoundry service.
You can attach an endpoint policy to your VPC endpoint that controls access to Amazon S3. The policy in the snapshot below provides the interface VPCE access to all resources within the VPC.
3. S3 Bucket Policy:
Once the VPCe and S3 bucket are provisioned, the next step is to provision the S3 bucket policy. With Amazon S3 bucket policies, you can secure access to objects in your buckets, so that only services or users with the appropriate permissions can access them. For this demo, we're using a bucket policy to restrict access by a VPC endpoint . Below is a bucket policy that allows access to the specific VPCe " vpce-0db2ff4e77e2622ba" to perform actions on the S3 bucket "sdktestingwithboto3" and its objects
Spin up a NetFoundry edge router in AWS:
The NetFoundry edge router is the WAN gateway in the VPC that helps you to reach the S3 bucket via the VPC endpoint over a private and secure zero trust overlay. The ER is deployed on a EC2 instance. You can also deploy the ER in a container.
Follow the instructions to spin up the NetFoundry edge router in AWS from the AWS marketplace. The router has to be spun up in the same VPC as your interface VPC endpoint or have the reachability to the VPC endpoint if it is spun up in a different VPC.
The router should show registered and online once provisioned successfully.
Create your identity, service and service policy:
You can access your S3 bucket via a NetFoundry endpoint software on your laptop or via ziti python SDK embedded into your application that requires access to S3 bucket.
- Create your identity - Create an identity each to access the S3 bucket from the NetFoundry Ziti Py SDK and the Ziti desktop edge
-
Create your service for accessing S3 bucket via interface VPC endpoint.
The following URL is used for private access to S3 buckets associated with the interface VPC endpoint from the boto3 client for S3 imported in the python program.
https://sdktestingwithboto3.vpce-0db2ff4e77e2622ba-uf7ato7f.s3.ap-southeast-1.vpce.amazonaws.com
From the laptop running ziti desktop edge, the following URL is used to access the object netfoundry.jpg
https://sdktestingwithboto3.s3.ap-southeast-1.amazonaws.com/netfoundry.jpg
The service configuration is configured with the "wildcard" domain name using the DNS name of the interface VPC endpoint and the private IP address attached to elastic network interface (ENI) .
The identity is that of the customer edge router that was provisioned in AWS from the marketplace.
Port 443 has been selected for private access within the VPC via the interface VPC endpoint.
-
Create your service policy to allow your identity (or identities) to access the S3 Access service
Create a service policy to allow the identities for devices or the router identity deployed in your factory or site to access the S3 bucket over the highly secure NetFoundry cloud network.
The service policy that allows identities to access the S3 bucket has been created as shown below:
Accessing S3 bucket and objects using a Boto3 client + Ziti Python SDK over the NetFoundry Cloud:
Download demo python program that imports AWS's boto3 and ziti SDK to access the private buckets via the Ziti. The Py program generates logfiles and uploads them to the S3 bucket, creating a folder and uploading the log files to the folder over the NetFoundry Cloud network.
https://github.com/openziti-test-kitchen/boto-demo/tree/main
Use the following commands to run the Py program. You can run export ZITI_LOG=4 to view the logs
python boto-demo-main/s3z/s3z.py \
--ziti-identity-file "/d/S3/identityname.json" \
--bucket-name "sdktestingwithboto3" \
--bucket-endpoint "https://bucket.vpce-0db2ff4e77e2622ba-uf7ato7f.s3.ap-southeast-1.vpce.amazonaws.com" \
--push-log-dir "logs" \
--object-prefix "foldername"
The identityname.json is the json file of the identity that is registered with the NetFoundry network.
How to enroll the jwt obtained from the console and generate the json -
python -m openziti enroll --jwt identityname.jwt --identity identityname.json
https://github.com/openziti/ziti-sdk-py/blob/main/sample/README.md#get-and-enroll-an-identity
The bucket name is the name of your S3 bucket
The bucket-endpoint is the DNS of the S3 VPC endpoint prefixed with your bucket name
The object-prefix is the name of the folder you want to create in S3
Once you execute the Py program, you will find the output like the one below
The folder is created in your S3 bucket
And the log files are uploaded in the folder
Accessing S3 bucket and objects using a Ziti desktop edge over the NetFoundry Cloud:
Install the desktop edge based on the OS of your device - https://netfoundry.io/downloads/
You'll find the installation instructions for the endpoints in the respective sections.
Register your identity to the network from your endpoint software.
With the required service policy in place, you should see the S3 service listed on your endpoint.
Access the S3 object from your device over the NetFoundry network
Comments
0 comments