Journey to Secure Azure Kubernetes Service

My experience

When most of the organizations are currently moving to the trend to implement Microservices pattern, Containerization, Modernization, and etc. They might consider using Kubernetes as one of the options.

However, most of the time we might face challenges like the workloads or the projects that need to be fully secured or the traffic must be controlled in some of the restricted environment. Some of the concerns will be, how can we fully deploy the cluster within the vnet without any communication through public ip.

With my recent experience on the preview of the cluster egress with UDR (preview only, please do not use it for production) and private cluster. I will write down the steps on the implementation with the reference materials.

Design

Diagram 1 : AKS Private Cluster with UDR to Proxy Egress

Implementation

Step 1 : To get started, need to make sure that you have sufficient privileges to create AKS and some of the restricted subscription they do remove the user rights for creating new resource group, new vnet, and etc. To mitigate that you might want to create a service principal with “Contributor” to the subscription.

Step 2 : After creating service principal, you need to create 2 subnets according to the diagram above, Proxy Subnet mainly to ensure the routing of all the AKS Management egress traffic and AKS Subnet to deploy all the worker nodes of AKS within the subnet.

AKS Subnet with NSG and Route Table

Step 3: Creating route table and associates to Proxy Subnet. In this case, the routing will be going directly to the private IP of the AzFW. AzFW will be acting as the proxy to control the egress management traffic of AKS.

Route table setup

Step 4 : For the AKS management egress traffic, there are certain rules and fqdn need to be allowed and you can refer to here. So based on the documentation, the table below is the setup of the Application Rules in AzFirwall. You also need to setup the rules for the NTP time for linux nodes.

AzFw Application Rules

Step 5: After provisioning network and route table, you can provision the private AKS with the following scripts using Azure Cloud Shell.

AKS CLI Scripts

Step 6 : After provisioning of AKS, you will notice that Private Endpoint will be created inside the AKS resource group starting with the prefix of MC_*. With the private endpoint being setup for AKS, the communication between worker nodes and AKS control plane API Server will be fully private and via private IP.

Private Endpoint for AKS

Step 7: Subsequently you can setup the network security group to further tighten and restrict the Inbound and Outbound of AKS Subnet.

AKS Subnet NSG Inbound

AKS Subnet NSG Outbound

Step 8 : You will need to create a bastion and the jumphost in the vnet to perform the kubectl command to the API Server.

Step 9 : After everything in place, you can start performing the kubectl command within the bastion and the first thing you might want to quickly ensure is that the kubernetes is running by checking all the pods created in all namespaces are running.

kubectl get all --all-namespaces

Step 10 (optional) : You can perform a quick test to deploy container image into the kubernetes environment.

kubectl create deployment --image=nginx nginx
kubectl get pods

Sample deployments into AKS

All the above steps are for references only and please let me know or feedback on any information that is inaccurate.

Thank You for your time and Enjoy your Secured AKS.