How To Secure Your Application With SSL/TLS Certificate In AKS

How can you configure SSL/TLS certificates in AKS?

Do you utilize TLS Kubernetes secrets? If you use GitOps, it's advisable not to store a duplicate of your certificate in your Git repository.

In this article, I will present two solutions for configuring your applications with SSL/TLS certificates while ensuring security and encryption.

In both solutions, the certificate is initially uploaded to an Azure Key Vault.

I will explain the difference between the two solutions and subsequently provide a step-by-step guide on implementing each.

To be able to follow the deployment steps, you will need to install:

• The Azure CLI and the Bicep tools.
• Docker Desktop.
• Kubectl tool.

Let’s get started!

Access to the code : GitHub repository

Solution 1: Application Gateway

Explanation

This solution works with the Azure Application Gateway and requires the Application Gateway Ingress Controller installed to the cluster.

What I like about this solution is that Kubernetes is not even aware of the certificate so we don’t have to worry about secret encryption in Kubernetes.

We configure the Application Gateway to directly retrieve the certificate from the key vault.

In order for this to work, a managed identity is assigned to the Application Gateway and has access to the Azure Key Vault secret.

If Azure RBAC is enabled on the Key Vault, the role Key Vault Secrets User must be assigned to Application Gateway managed identity.

If RBAC is not enabled, configure access policy for the identity with GET access for secrets.

It’s important to understand that we always retrieve the certificate as a secret.

Behind the scene, certificates are composed of three interrelated resources linked together as a Key Vault certificate; certificate metadata, a key, and a secret. See more in the Microsoft documentation.

Once the Application Gateway has access to the Key Vault, we can specify in the ingress resource which certificate to use with an annotation.

This solution works very well with GitOps as there is no certificate in the Git repository.

Deployment

Let’s see how this works in practice.

Clone the repository and navigate to the 01-agic folder.

From there, connect to your Azure subscription with the az login command.

Then, run the script.ps1 PowerShell script.

It will create a new resource group rg-tlscertificate-01 with an AKS cluster and AGIC enabled.

Notice the different identities and role assignments in the main.bicep file.

Among others, we have to assign the Contributor role to the AGIC Identity on the resource group.

resource agicRoleAssignmentForResourceGroup 'Microsoft.Authorization/roleAssignments@2020-10-01-preview' = {
  name: guid(aks.id, 'agic', contributorRoleDefinitionId)
  scope: resourceGroup()
  properties: {
    roleDefinitionId: contributorRoleDefinitionId
    principalId: aks.properties.addonProfiles.ingressApplicationGateway.identity.objectId
    principalType: 'ServicePrincipal'
  }
}

To allow the application gateway to retrieve the certificate from the Key Vault, we must also create an identity for the Application Gateway and assign the role Key Vault Secrets User to this identity on the Key Vault.

resource applicationGatewayRoleAssignmentForKeyVault 'Microsoft.Authorization/roleAssignments@2020-10-01-preview' = {
  name: guid(applicationGatewayIdentity.id, kvSecretUserRoleDefinitionId)
  scope: kv
  properties: {
    roleDefinitionId: kvSecretUserRoleDefinitionId
    principalId: applicationGatewayIdentity.properties.principalId
    principalType: 'ServicePrincipal'
  }
}

Once the infrastructure is ready, you can create a self signed certificate for mywebsite.local, upload it to the Key Vault and configure the Application Gateway to reference it.

You can do all of that with the script upload-certificate-agw.ps1.

IMPORTANT: you will have to change the variable $kvName with the name of your Key Vault. You can retrieve the Key Vault name from the Azure portal.

$kvName = 'kv-punjt2filcllc'

az keyvault certificate create --vault-name $kvName `
    -n my-certificate `
    -p `@policy.json

az network application-gateway ssl-cert create --gateway-name agw-01 `
    --name my-certificate `
    --resource-group rg-tlscertificate-01 `
    --key-vault-secret-id "https://$kvName.vault.azure.net/secrets/my-certificate"

Run the script.

We are now ready to deploy a web application to test everything.

Navigate to the 00-application folder.

Build and push the docker image to the registry. This is a default asp.net web application.

IMPORTANT: change the ACR name crpunjt2filcllc with yours.

az acr login -n crpunjt2filcllc
docker build -t crpunjt2filcllc.azurecr.io/app-01:01 -f .\TlsCertificate\Dockerfile .
docker push crpunjt2filcllc.azurecr.io/app-01:01

Go back to the 01-agic folder and deploy the deployment.yaml file to AKS with kubectl.

IMPORTANT: change the ACR name crpunjt2filcllc with yours.

cd ..\01-agic\
az aks get-credentials -n aks-01 -g rg-tlscertificate-01
kubectl apply -f .\deployment.yaml

Notice in the deployment.yaml how we reference the TLS certificate with the annotation appgw.ingress.kubernetes.io/appgw-ssl-certificate.

Notice also that we didn’t reference any secret in the tls block.

Lastly, we enabled HTTP to HTTPS redirection with the appgw.ingress.kubernetes.io/ssl-redirect annotation.

Those are specific annotations for AGIC. For the complete list, see AGIC documentation.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-01
  name: app-01
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-01
  template:
    metadata:
      labels:
        app: app-01
    spec:
      containers:
        - name: app-01
          image: crpunjt2filcllc.azurecr.io/app-01:01
          ports:
            - containerPort: 8080
          resources:
            limits:
              memory: "512Mi"
              cpu: "256m"
            requests:
              memory: "256Mi"
              cpu: "128m"
---
apiVersion: v1
kind: Service
metadata:
  name: app-01
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: app-01
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-01
  annotations:
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
    appgw.ingress.kubernetes.io/appgw-ssl-certificate: "my-certificate"
spec:
  ingressClassName: azure-application-gateway
  tls:
    - hosts:
        - mywebsite.local
  rules:
    - host: mywebsite.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-01
                port:
                  number: 80

To test the solution, since we don’t actually have a domain name mywebsite.local, retrieve the public IP address of the Application Gateway from the Azure Portal and update your hosts file accordingly.

Once the hosts file updated, open your browser and navigate to mywebsite.local.

Since we use a self signed certificate, we will have a certificate warning. Click on proceed to mywebsite.local.

Then, you’ll see the default asp.net web application.

When you are done, you can cleanup all resources by deleting the resource group with this command.

az group delete -n rg-tlscertificate-01 --yes

Solution 2: External Secrets

Explanation

In this solution, we do have a Kubernetes secret. But the important part is that this secret is not created by a Kubernetes administrator.

Instead, the secret is automatically created by the External Secret operator from the secret already in the Key Vault.

To allow the External Secret Store access to the Key Vault, we create a managed identity and use Workload identity federation to link a Kubernetes service account with this identity.

If RBAC is enabled on the Key Vault, the managed identity requires the role of Key Vault Secrets User; otherwise, it needs GET secret access in the access policies.

What I like about this solution is that it works with any ingress controller. Since a regular Kubernetes secret is created, it does not matter if we use Application Gateway Ingress Controller or NGINX Ingress Controller or the latest Application Gateway For Containers load balancer.

All those solutions will be able to fetch the Kubernetes secret.

That being said, there is a copy of the secret in AKS and you must enable encryption at rest with Key Management Service to keep this secure.

This solution also works very well with GitOps as we don’t keep the Kubernetes secret resource in Git but just the External Secret one.

Deployment

To test this solution, we will use the NGINX ingress controller. But, keep in mind that since we are using Kubernetes secret it would work with any ingress controller.

Clone the repository, navigate to the 02-external-secrets folder and run the PowerShell script script.ps1.

The script will create a new AKS cluster with all the necessary managed identities to configure KMS and workload identity in a resource group named rg-tlscertificate-01.

Next, open the script configure-kms-upload-certificate.ps1 and adjust the variables $kvName and $clusterName with your values. Retrieve the values from the Azure portal.

$kvName = 'kv-punjt2filcllc'
$keyName = 'aks-kms-key-01'
$clusterName = 'aks-01'
$resourceGroupName = 'rg-tlscertificate-01'

## Create a key in Key Vault. For some reason I wasn't able to do it with Bicep.
az keyvault key create --kty RSA `
                    --name $keyName `
                    --ops decrypt encrypt `
                    --size 2048 `
                    --vault-name $kvName
                    
$keyId = $(az keyvault key show --name $keyName --vault-name $kvName --query 'key.kid' -o tsv)

## Update cluster to enable KMS
az aks update --name $clusterName `
    --resource-group $resourceGroupName `
    --enable-azure-keyvault-kms `
    --azure-keyvault-kms-key-vault-network-access "Public" `
    --azure-keyvault-kms-key-id $keyId

## Create self signed certificate
az keyvault certificate create --vault-name $kvName `
    -n my-certificate `
    -p `@policy.json

Run the script. It will create a key in the Key Vault, configure KMS and create a self signed certificate for our website mywebsite.local.

Next, open the script configure-workload-identity.ps1 and update the variable $clusterName.

Once the variable updated, run the script. It will configure workload identity on the cluster.

• managed identity: id-workload-01
• federated identity: fid-workload-01
• service account in the default namespace: sa-workload-01

$clusterName = 'aks-01'
$resourceGroupName = 'rg-tlscertificate-01'
$identityName = 'id-workload-01'
$federatedIdentityName = 'fid-workload-01'
$namespace = 'default'
$serviceAccountName = 'sa-workload-01'

az aks get-credentials -n $clusterName -g $resourceGroupName

$clientId = $(az identity show --resource-group $resourceGroupName --name $identityName --query 'clientId' -o tsv)
$oidcIssuer = $(az aks show -n $clusterName -g $resourceGroupName --query "oidcIssuerProfile.issuerUrl" -o tsv)

$serviceAccount = @"
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: $clientId
  name: $serviceAccountName
  namespace: $namespace
"@

$serviceAccount | kubectl apply -f -

az identity federated-credential create `
    --name $federatedIdentityName `
    --identity-name $identityName `
    --resource-group $resourceGroupName `
    --issuer $oidcIssuer `
    --subject "system:serviceaccount:${namespace}:${serviceAccountName}" `
    --audience api://AzureADTokenExchange

Then, install ingress-nginx

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.0/deploy/static/provider/cloud/deploy.yaml

Install External Secrets:

kubectl apply -k https://github.com/external-secrets/external-secrets//config/crds/bases?ref=v0.9.11

Our cluster is now ready. Let’s build and push the Docker image before deploying the application.

Navigate to the 00-application folder.

Build and push the docker image to the registry. This is a default asp.net web application.

IMPORTANT: change the ACR name crpunjt2filcllc with yours.

az acr login -n crpunjt2filcllc
docker build -t crpunjt2filcllc.azurecr.io/app-01:01 -f .\TlsCertificate\Dockerfile .
docker push crpunjt2filcllc.azurecr.io/app-01:01

Now, go back to the 02-external-secrets folder.

Open the deployment.yaml file and update the image property l.19. Replace the name of the container registry with your container registry name: crpunjt2filcllc.azurecr.io/app-01:01.

Update also the name of the key vault l.74: https://kv-punjt2filcllc.vault.azure.net

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-01
  name: app-01
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-01
  template:
    metadata:
      labels:
        app: app-01
    spec:
      containers:
        - name: app-01
          image: crpunjt2filcllc.azurecr.io/app-01:01
          ports:
            - containerPort: 8080
          resources:
            limits:
              memory: "512Mi"
              cpu: "256m"
            requests:
              memory: "256Mi"
              cpu: "128m"
---
apiVersion: v1
kind: Service
metadata:
  name: app-01
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: app-01
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-01
  annotations:
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - mywebsite.local
      secretName: my-certificate
  rules:
    - host: mywebsite.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-01
                port:
                  number: 80
---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: azure-store
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://kv-punjt2filcllc.vault.azure.net"
      serviceAccountRef:
        name: sa-workload-01
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-certificate
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore
    name: azure-store
  target:
    template:
      type: kubernetes.io/tls
      engineVersion: v2
      data:
        tls.crt: "{{ .tls | b64dec | pkcs12cert }}"
        tls.key: "{{ .tls | b64dec | pkcs12key }}"
  data:
    - secretKey: tls
      remoteRef:
        key: secret/my-certificate

Notice how we use nginx ingress controller with the ingressClassName property, the secret name my-certificate in the tls block and configure SSL redirection with the annotation nginx.ingress.kubernetes.io/force-ssl-redirect: "true".

Notice also the SecretStore azure-store and the ExternalSecret my-certificate. Thanks to those 2 CRDs a Kubernetes secret will be automatically created from the secret in the Key Vault.

Deploy the application with the following command.

kubectl apply -f .\deployment.yaml

To test the application, retrieve the public IP address of the ingress with the following command:

kubectl get ingress

When you have the IP address, update your hosts file in order to point mywebsite.local to this address.

Once this is done, you can open your browser and navigate to mywebsite.local.

Since we use a self signed certificate, you will have a certificate warning. Click on proceed to mywebsite.local.

Then, you’ll see the default asp.net web application.

When you are done, you can cleanup all resources by deleting the resource group with this command.

az group delete -n rg-tlscertificate-01 --yes

Conclusion

In wrapping up, these two solutions stand out as my preferred methods for securing applications deployed to AKS with TLS certificates.

When leveraging AGIC, I find solution 1 particularly appealing due to its avoidance of secrets within Kubernetes.

However, while solution 1 is tailored to AGIC, solution 2 offers greater versatility. It's important to note that for solution 2, enabling encryption with KMS is necessary. Yet, this requirement may not pose an issue, considering you likely already have other types of secrets in your cluster requiring encryption as well.

With either solution, updating the certificate in the Key Vault will seamlessly propagate the updates to the certificates used by our applications.

I'd love to hear your thoughts on these solutions. Feel free to share your feedback in the comments below.