When most of the organizations are currently moving to the trend to implement Microservices pattern, Containerization, Modernization, and etc. They might consider using Kubernetes as one of the options.
However, most of the time we might face challenges like the workloads or the projects that need to be fully secured or the traffic must be controlled in some of the restricted environment. Some of the concerns will be, how can we fully deploy the cluster within the vnet without any communication through public ip.
With my recent experience on the preview of the cluster egress with UDR (preview only, please do not use it for production) and private cluster. I will write down the steps on the implementation with the reference materials.
Step 1 : To get started, need to make sure that you have sufficient privileges to create AKS and some of the restricted subscription they do remove the user rights for creating new resource group, new vnet, and etc. To mitigate that you might want to create a service principal with “Contributor” to the subscription.
Step 2 : After creating service principal, you need to create 2 subnets according to the diagram above, Proxy Subnet mainly to ensure the routing of all the AKS Management egress traffic and AKS Subnet to deploy all the worker nodes of AKS within the subnet.
Step 3: Creating route table and associates to Proxy Subnet. In this case, the routing will be going directly to the private IP of the AzFW. AzFW will be acting as the proxy to control the egress management traffic of AKS.
Step 4 : For the AKS management egress traffic, there are certain rules and fqdn need to be allowed and you can refer to here. So based on the documentation, the table below is the setup of the Application Rules in AzFirwall. You also need to setup the rules for the NTP time for linux nodes.
Step 5: After provisioning network and route table, you can provision the private AKS with the following scripts using Azure Cloud Shell.
Step 6 : After provisioning of AKS, you will notice that Private Endpoint will be created inside the AKS resource group starting with the prefix of MC_*. With the private endpoint being setup for AKS, the communication between worker nodes and AKS control plane API Server will be fully private and via private IP.
Step 7: Subsequently you can setup the network security group to further tighten and restrict the Inbound and Outbound of AKS Subnet.
Step 8 : You will need to create a bastion and the jumphost in the vnet to perform the kubectl command to the API Server.
Step 9 : After everything in place, you can start performing the kubectl command within the bastion and the first thing you might want to quickly ensure is that the kubernetes is running by checking all the pods created in all namespaces are running.
kubectl get all --all-namespaces
Step 10 (optional) : You can perform a quick test to deploy container image into the kubernetes environment.
kubectl create deployment --image=nginx nginxkubectl get pods
All the above steps are for references only and please let me know or feedback on any information that is inaccurate.
Thank You for your time and Enjoy your Secured AKS.