How To Create a Private AKS Cluster That Uses Azure Key Vault

Recently, I did a migration from IaaS (a solution composed essentially of virtual machines) to PaaS. I needed to create a complete private AKS cluster. In this article, I describe how I did it.

Together we’ll see:

1. The difference between a public and private AKS cluster

2. What the solution looks like

3. How to communicate with Azure Key Vault

1. The Difference Between a Public and Private AKS Cluster

A private AKS cluster means that the control plane is not accessible from the internet and has a private IP address.

By default, when we create an AKS cluster in Azure, the API server has a public IP address. It means that it is accessible from anywhere through the internet to issue commands to kubectl. This implementation is the simplest. With this solution, it is convenient to integrate with a cloud CI/CD tool like Azure DevOps when cloud agents need to communicate with the control plane.

However, some companies may need a little bit more security. In a scenario where we want to access the API server only from our private network (VNET) we can create a private AKS cluster. With this solution, it is impossible to communicate with the control plane from the internet. A private endpoint is added to the VNET and our AKS cluster gets a private IP address.

In both scenarios, the control plane is managed by Azure and the worker nodes (managed by the clients) have a private IP address in the VNET.

How does the control plane communicate with the worker nodes?

For a public cluster, since the API server has a public IP adress, there is a tunnel between the control plane and the worker nodes.

For a private cluster, the API server has a private IP address in the VNET. The tunnel stills exist but the traffic between the control plane and the worker nodes remains private.

2. What The Solution Looks Like

Now that you understand the difference between a public and private AKS cluster, let’s see how we can create a private cluster in Azure.

This solution, is composed of:

• 2 virtual networks: vnet-hub and vnet-aks. The vnet-hub is the central point. Usually, in a hybrid infrastructure, we’ll find a VPN gateway or an Express Route gateway to connect to the on-premise environment. The vnet-aks is dedicated to our cluster. All our worker nodes will be there.

• A bastion. Access to the VNET from the outside. It could be replaced by a Jumpbox.

• A Container Registry with private endpoint to ensure private traffic. The container registry will contain all our container images.

• A 1st Virtual Machine that we’ll use as an Azure DevOps agent (our cluster being private, we cannot used Azure DevOps hosted agents)

• A private DNS zone for the cluster: privatelink.canadacentral.azmk8s.io linked to the vnet-hub

• A private DNS zone for the container registry: privatelink.azurecr.io linked to the vnet-hub

• A private DNS zone for the Key Vault: privatelink.vaultcore.azure.net linked to the vnet-hub

• A 2nd Virtual Machine that is used as a DNS server. Both VNETs will use this VM as DNS server. A conditional forwarder to Azure IP (168.63.129.16) has to be configured for each private DNS zone. More information can be found here.

• An Application Gateway with a public and a private IP address (some applications will be exposed on the internet, some only internally)

• A Key Vault with private endpoint to ensure private traffic

• Our AKS cluster

3. How To Communicate With Azure Key Vault

When we were using virtual machines, we would store all secrets in an Azure Key Vault. Each virtual machine can have a system assigned identity. By allowing the identity of the VM to access the Key Vault, our application can fetch secrets from the Key Vault.

To have a similar experience with AKS, we can use the Azure Active Directory pod-managed identities add-on. This add-on allow us to link a managed identity to a pod. Like the VM, by allowing the identity of the pod to access the Key Vault, our application can fetch secrets from the Key Vault.

The only restriction is that our cluster has to use the Azure CNI as the network plugin.

This is it. I hope you found this article useful. In a next article, I’ll share how I deploy this solution with Terraform.