Configuring Secret Store CSI Driver with Terraform: A Guide to Secure Secrets Management in Azure Kubernetes Service

In my previous article, I talked about how I was able to use the External Secret operator to synchronize certificates (and also all kind of secrets) from the Azure Key Vault to AKS.

In this article, we’ll install the Secret Store CSI Driver in AKS. With this tool, you can also fetch secrets from Azure Key Vault. So, what are the differences between the Secret Store CSI Driver and External Secrets and which one to use?

This is what we’re going to talk about.

By the end of the article, you will know in which situation to use External Secrets or the Secret Store CSI Driver.

Finally, we will deploy the Secret Store CSI Driver with Terraform.

Let’s get started.

As usual the code is available on GitHub.

The Secret Store CSI Driver

The Secret Store CSI Driver allows you to mount a volume when your pod starts and create a file in this volume containing the secret fetched from the Azure Key Vault.

With the Secret Store CSI Driver, you don’t need to use Kubernetes secrets.

When the pod is deleted, the volume is cleaned up and deleted as well.

The primary purpose of the Secret Store CSI Driver is to avoid using Kubernetes secrets. However, if your application relies on environment variables to retrieve secrets, you can configure the Secret Store CSI Driver to create a Kubernetes secret and inject the secrets into the pod as environment variables.

When you delete your pod, the volume will be deleted as we mentioned and the Kubernetes secret too. So basically, with the Secret Store CSI Driver the secret lifetime is equal to your pod lifetime.

When to use External Secrets vs Secret Store CSI Driver

If you need to create Kubernetes secrets without deploying a pod, your best option is to use External Secrets.

For example, when installing an application you don't own, perhaps one developed by a third party, it's likely that the application will rely on Kubernetes secrets to manage sensitive data. In such cases, I recommend using External Secrets.

To put it simply, if your setup involves using Kubernetes secrets, External Secrets is the way to go. On the other hand, if you're developing an application and prefer not to use Kubernetes secrets, the Secret Store CSI Driver is a strong alternative.

Now, let's dive into how to configure the Secret Store CSI Driver in practice.

Create an AKS cluster with the Secret Store CSI Driver enabled with Terraform

We start by creating the resource group, the virtual network and subnet.

To get started, open the repository and navigate to the 01-aks folder.

### Resource group
resource "azurerm_resource_group" "rg" {
  name     = "rg-csi-driver-01"
  location = "Canada Central"
}

### Network
resource "azurerm_virtual_network" "vnet" {
  name                = "vnet-csi-driver-01"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  address_space       = ["10.0.0.0/16"]
}

resource "azurerm_subnet" "aks" {
  name                 = "snet-aks-01"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.0.0.0/24"]
}

Then we add the key vault.

### Key vault
resource "azurerm_key_vault" "kv" {
  name                        = "kv-2024070102"
  location                    = azurerm_resource_group.rg.location
  resource_group_name         = azurerm_resource_group.rg.name
  enabled_for_disk_encryption = true
  tenant_id                   = data.azurerm_client_config.current.tenant_id
  soft_delete_retention_days  = 7
  purge_protection_enabled    = false

  sku_name = "standard"

  enable_rbac_authorization = true
}

Next, we add the different identities used by the cluster. Notice we added an identity called id-csi-driver-workloadidentity-01. This identity will be used as a workload identity to authorize the workload to access to the Key Vault with the CSI driver. Since we need to retrieve secrets, we create a role assignment to assign the Key Vault Secrets User to this identity.

We also assign the Key Vault Secrets Officer role to the current user. It will allow us to create a new secret to the Key Vault.

### Identities
resource "azurerm_user_assigned_identity" "controlplane" {
  location            = azurerm_resource_group.rg.location
  name                = "id-csi-driver-controlplane-01"
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_user_assigned_identity" "kubelet" {
  location            = azurerm_resource_group.rg.location
  name                = "id-csi-driver-kubelet-01"
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_role_assignment" "controlplane_identity_contributor" {
  scope                = azurerm_user_assigned_identity.kubelet.id
  role_definition_name = "Managed Identity Contributor"
  principal_id         = azurerm_user_assigned_identity.controlplane.principal_id
}

resource "azurerm_role_assignment" "controlplane_resourcegroup_contributor" {
  scope                = azurerm_resource_group.rg.id
  role_definition_name = "Contributor"
  principal_id         = azurerm_user_assigned_identity.controlplane.principal_id
}

resource "azurerm_role_assignment" "cluster_admin" {
  scope                = azurerm_kubernetes_cluster.aks.id
  role_definition_name = "Azure Kubernetes Service RBAC Cluster Admin"
  principal_id         = data.azurerm_client_config.current.object_id
}

### CSI Driver identity
resource "azurerm_user_assigned_identity" "workload_identity" {
  location            = azurerm_resource_group.rg.location
  name                = "id-csi-driver-workloadidentity-01"
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_role_assignment" "secret_user" {
  scope                = azurerm_key_vault.kv.id
  role_definition_name = "Key Vault Secrets User"
  principal_id         = azurerm_user_assigned_identity.workload_identity.principal_id
}

resource "azurerm_role_assignment" "current_user_secret_officer" {
  scope                = azurerm_key_vault.kv.id
  role_definition_name = "Key Vault Secrets Officer"
  principal_id         = data.azurerm_client_config.current.object_id
}

We are now ready to create the AKS service.

### AKS
resource "azurerm_kubernetes_cluster" "aks" {
  name                   = "aks-csi-driver-01"
  location               = azurerm_resource_group.rg.location
  resource_group_name    = azurerm_resource_group.rg.name
  dns_prefix             = "aks-csi-driver-01"
  kubernetes_version     = "1.30"
  local_account_disabled = true
  sku_tier               = "Free"
  oidc_issuer_enabled    = true

  default_node_pool {
    name           = "default"
    node_count     = 1
    vm_size        = "Standard_D2s_v3"
    vnet_subnet_id = azurerm_subnet.aks.id
  }

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.controlplane.id]
  }

  kubelet_identity {
    client_id                 = azurerm_user_assigned_identity.kubelet.client_id
    object_id                 = azurerm_user_assigned_identity.kubelet.principal_id
    user_assigned_identity_id = azurerm_user_assigned_identity.kubelet.id
  }

  network_profile {
    network_plugin = "kubenet"
    dns_service_ip = "10.0.3.4"
    service_cidr   = "10.0.3.0/24"
  }

  azure_active_directory_role_based_access_control {
    admin_group_object_ids = var.cluster_admin_ids
    azure_rbac_enabled     = true
  }

  key_vault_secrets_provider {
    secret_rotation_enabled = true
  }

  depends_on = [
    azurerm_role_assignment.controlplane_identity_contributor,
    azurerm_role_assignment.controlplane_resourcegroup_contributor
  ]
}

Configure Workload Identity

To configure workload identity, we first need a service account in Kubernetes.

Connect to the cluster with the following az CLI command:

az aks get-credentials -n aks-csi-driver-01 -g rg-csi-driver-01

Navigate to the folder 02-workload-identity/manifests and apply the service-account.yaml manifest.

Do not forget to replace the client-id with the client id of the workload identity created earlier: id-csi-driver-workloadidentity-01.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: {client id of the identity id-csi-driver-workloadidentity-01}
  name: workload-identity-sa
  namespace: default

Run kubectl apply -f service-account.yaml.

Navigate back to folder 02-workload-identity to apply the terraform code.

We will create a new federated identity credential with the following code:

resource "azurerm_federated_identity_credential" "fid" {
  name                = "fid-csi-driver-workloadidentity-01"
  resource_group_name = data.azurerm_resource_group.rg.name
  audience            = ["api://AzureADTokenExchange"]
  issuer              = data.azurerm_kubernetes_cluster.aks.oidc_issuer_url
  parent_id           = data.azurerm_user_assigned_identity.workload_identity.id
  subject             = "system:serviceaccount:${var.service_account_namespace}:${var.service_account_name}"
}

Run terraform init and terraform apply to create the federated identity credential..

Test the installation

We will do 2 different tests. One without Kubernetes secret and one with a Kubernetes secret.

In both scenarios, we will use a secret named secret from the key vault created earlier. The value will be MySecretValue.

To add the secret to the key vault, run the following az CLI command:

az keyvault secret set --name secret --vault-name kv-2024070102 --value MySecretValue

Next, we will create some Kubernetes resources. To apply all manifests, navigate to the folder 02-workload-identity/manifests .

Test without Kubernetes secret

Create a secret provider. Do not forget to replace the clientId with the client ID of the workload identity. Replace the tenantId with your tenant ID as well.

The code is in the 02-workload-identity/manifests folder.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: no-secret-provider
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: {clientId of the workload identity}
    keyvaultName: kv-2024070102
    cloudName: "" # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects: |
      array:
        - |
          objectName: secret             # Set to the name of your secret
          objectType: secret              # object types: secret, key, or cert
          objectVersion: ""               # [OPTIONAL] object versions, default to latest if empty
    tenantId: {your tenantId} # The tenant ID of the key vault

Run kubectl apply -f secret-provider-class-no-secret.yaml.

Next, deploy a pod that will use this secret provider.

kind: Pod
apiVersion: v1
metadata:
  name: busybox-no-secret
  labels:
    azure.workload.identity/use: "true"
spec:
  serviceAccountName: "workload-identity-sa"
  containers:
    - name: busybox
      image: registry.k8s.io/e2e-test-images/busybox:1.29-4
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
        - name: secrets-store01-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: no-secret-provider

Run kubectl apply -f pod-no-secret.yaml

Once both manifests have been deployed, you can validate that the secret is correctly fetched from the key vault.

Make sure you run the command once the pod is actually running. Check this with the command kubectl get pod.

You should see something similar to this screenshot.

Once the pod is running, you can execute the following command:

kubectl exec busybox-no-secret -- cat /mnt/secrets-store/secret

 You should see the value of the secret: MySecretValue.

Test with Kubernetes secret

Create the secret provider. Execute the command kubectl apply -f secret-provider-class-with-secret.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: with-secret-provider
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: {clientID of the workload identity}
    keyvaultName: kv-2024070102
    cloudName: "" # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects: |
      array:
        - |
          objectName: secret             # Set to the name of your secret
          objectType: secret              # object types: secret, key, or cert
          objectVersion: ""               # [OPTIONAL] object versions, default to latest if empty
    tenantId: {your tenantId} # The tenant ID of the key vault
  secretObjects: # [OPTIONAL] SecretObjects defines the desired state of synced Kubernetes secret objects
    - data:
        - key: secret # data field to populate
          objectName: secret # name of the mounted content to sync; this could be the object name or the object alias
      secretName: my-secret # name of the Kubernetes secret object
      type: Opaque

Create a pod that uses the provider and the Kubernetes secret. Notice that we still need to mount a volume.

Execute the command kubectl apply -f pod-with-secret.yaml.

kind: Pod
apiVersion: v1
metadata:
  name: busybox-with-secret
  labels:
    azure.workload.identity/use: "true"
spec:
  serviceAccountName: "workload-identity-sa"
  containers:
    - name: busybox
      image: registry.k8s.io/e2e-test-images/busybox:1.29-4
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
        - name: secrets-store01-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      env:
        - name: my_secret
          valueFrom:
            secretKeyRef:
              name: my-secret
              key: secret
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: with-secret-provider

The secret value will be available in the volume as well as in the environment variable my_secret created from the Kubernetes secret.

Run the following commands to see the Kubenetes secret, the secret from the file and from the environment variable.

kubectl get secret my-secret -o yaml
kubectl exec busybox-with-secret -- cat /mnt/secrets-store/secret
kubectl exec busybox-with-secret -- printenv my_secret

Conclusion

To wrap up, we walked through configuring the Secret Store CSI Driver with Terraform to securely pull secrets from Azure Key Vault.

We also highlighted that if you're working with Kubernetes secrets, the External Secrets Operator might be a more suitable option.

Ultimately, the Secret Store CSI Driver is a solid choice when you'd rather avoid using Kubernetes secrets altogether.