Implementing Windows Authentication in Azure Kubernetes Services: A Step-by-Step Guide
Jan 03, 2024Last year, I had the task of migrating a significant number of services that were previously deployed on IIS (Internet Information Services) within Windows virtual machines, over to AKS (Azure Kubernetes Services).
These virtual machines were integrated into the company's domain, and the various services were dependent on Windows authentication to establish communication among themselves.
The objective was to transfer all of these services to AKS while minimizing the need for extensive development changes. A complete alteration of authentication methods across all services was not a viable option due to the considerable effort it would entail at the present time.
Hence, we had to find a solution that would facilitate the use of Windows Authentication within the AKS environment.
This article details the solution I chose to tackle this challenge. Spoiler alert: the use of GMSA (Group Managed Service Account) is what made this migration possible.
Create your test environment
To test the solution I describe in this article, you can create a complete environment in Azure. Follow these instructions to create your test environment:
1. Clone the github repository.
2. Open a terminal in the Infrastructure folder.
3. Login to Azure with az login and retrieve your user object ID.
az login
az ad signed-in-user show -o tsv --query id
4. Update the main.bicepparam file:
currentUserObjectId ⇒ value from step 3.
password ⇒ the password you will use throughout this solution.
The currentUserObjectId will be used for role assignments.
To simplify the process, all virtual machines will use the same username and password. Username is useradmin.
5. Run the script.ps1. The script creates a new resource group and deploys the following ressources:
A Virtual Network.
A Virtual machine called vm-dc-01 in the vnet and a domain called mycompany.local. vm-dc-01 is the domain controller.
A second virtual machine vm-01 part of the domain used to test our windows container with docker.
A bastion to allow remote connection to the different VMs.
An Azure key vault.
An Azure container registry.
A Kubernetes cluster to run the windows container.
6. Go to the Azure portal and connect to vm-01 with Bastion.
7. Execute the following powershell script from vm-01 to add the vm to the new created domain mycompany.local. Replace the password with the one from step 4.
$username = "useradmin";
$password = "JCH4rqwgvh3xqcqbf" ## must be replaced with password from step 4
$domain = "mycompany.local";
$joinCred = New-Object pscredential -ArgumentList ([pscustomobject]@{
UserName = $username;
Password = (ConvertTo-SecureString -String $password -AsPlainText -Force)[0]
});
Add-Computer -Domain $domain -Credential $joinCred -Force -Restart
8. Install Docker desktop to vm-01.
Make sure Docker desktop is running correctly. You may have to follow the instructions to install wsl.
9. Install az cli to vm-01.
Your environment is now ready.
Enable Windows Authentication in a Windows Container
Before moving to AKS, let’s create a Windows container that can use Windows authentication.
Here, you will use your local computer to build a Docker image.
You need Visual Studio and Docker Desktop installed on your local computer.
1. Create a web app with Windows authentication
Start by creating a new ASP.NET Core Web App project with Visual Studio. During the project creation process, choose 'Windows' as the Authentication type in the Additional information popup before clicking on the Create button.
Next, we add a health check endpoint with anonymous access for Kubernetes readiness and liveness probe.
Add the following code to program.cs.
[...]
builder.Services.AddHealthChecks();
[...]
app.MapHealthChecks("/health").AllowAnonymous();
We also have to configure IIS to enable anonymous authentication. Update the launchSettings.json file for that. This is only for test purpose when running the application with Visual Studio. The actual IIS configuration will be done in the Dockerfile.
Run the application with IIS Express and navigate to the /health endpoint in a new incognito window. You should see the word 'healthy'.
2. Create the Dockerfile
Add a Dockerfile to use a Windows base image, enable Windows authentication, install .net 7 and configure IIS. We will also configure LogMonitor for logging.
Add the file to the project solution, in the same directory of the program.cs file.
# escape=`
FROM mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2022
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'Continue'; $verbosePreference='Continue';"]
# Enable Windows Authentication
RUN Install-WindowsFeature Web-Windows-Auth –IncludeAllSubFeature
# Install .NET
ENV DOTNET_VERSION 7.0.10
ENV DOTNET_DOWNLOAD_URL <https://download.visualstudio.microsoft.com/download/pr/d489c5d0-4d0f-4622-ab93-b0f2a3e92eed/101a2fae29a291956d402377b941f401/dotnet-hosting-7.0.10-win.exe>
ENV DOTNET_DOWNLOAD_SHA cd52c9d45e63458ce1e84282b2d9b1432ced7ecfdbe1b3d57b3f0791d20c2ecfcc7ffb8da0916722cea4cce55cb5dea8f34bff489f7a04c14c104a974ff72379
RUN Invoke-WebRequest $Env:DOTNET_DOWNLOAD_URL -OutFile WindowsHosting.exe; `
if ((Get-FileHash WindowsHosting.exe -Algorithm sha512).Hash -ne $Env:DOTNET_DOWNLOAD_SHA) { `
Write-Host 'CHECKSUM VERIFICATION FAILED!'; `
exit 1; `
}; `
`
dir c:\Windows\Installer; `
Start-Process "./WindowsHosting.exe" '/install /quiet /norestart' -Wait; `
Remove-Item -Force -Recurse 'C:\ProgramData\Package Cache\*'; `
Remove-Item -Force -Recurse C:\Windows\Installer\*; `
Remove-Item -Force WindowsHosting.exe
RUN setx /M PATH $($Env:PATH + ';' + $Env:ProgramFiles + '\dotnet')
# Enable detection of running in a container
ENV DOTNET_RUNNING_IN_CONTAINER=true
# Configure IIS website
RUN Remove-Website -Name 'Default Web Site'; `
Set-ItemProperty IIS:\AppPools\DefaultAppPool -Name managedRuntimeVersion -Value ''; `
Set-ItemProperty IIS:\AppPools\DefaultAppPool -Name enable32BitAppOnWin64 -Value 0; `
Set-ItemProperty IIS:\AppPools\DefaultAppPool -Name processModel -value @{identitytype='ApplicationPoolIdentity'}; `
New-Website -Name 'webapp' `
-Port 80 -PhysicalPath 'C:\webapp' `
-ApplicationPool 'DefaultAppPool' -force;
RUN Import-Module IISAdministration; `
Start-IISCommitDelay; `
(Get-IISConfigSection -SectionPath 'system.webServer/security/authentication/windowsAuthentication').Attributes['enabled'].value = $true; `
(Get-IISConfigSection -SectionPath 'system.webServer/security/authentication/anonymousAuthentication').Attributes['enabled'].value = $true; `
Stop-IISCommitDelay;
# Create a directory for the webapp and copy it
RUN mkdir c:\webapp
COPY .\output\ c:\webapp
SHELL ["cmd", "/S", "/C"]
## Configure LogMonitor
WORKDIR /LogMonitor
COPY LogMonitorConfig.json .
RUN powershell.exe -command wget -uri <https://github.com/microsoft/windows-container-tools/releases/download/v1.1/LogMonitor.exe> -outfile LogMonitor.exe
# Change the startup type of the IIS service from Automatic to Manual
RUN sc config w3svc start=demand
# Enable ETW logging for the Web Site on IIS
RUN c:\windows\system32\inetsrv\appcmd.exe set config -section:system.applicationHost/sites /"[name='webapp'].logFile.logTargetW3C:"File,ETW"" /commit:apphost
EXPOSE 80
# Start "C:\LogMonitor\LogMonitor.exe C:\ServiceMonitor.exe w3svc"
ENTRYPOINT ["C:\\LogMonitor\\LogMonitor.exe", "C:\\ServiceMonitor.exe", "w3svc"]
Create a file named LogMonitorConfig.json in the same folder of the Dockerfile. It represents the LogMonitor configuration.
{
"LogConfig": {
"sources": [
{
"type": "EventLog",
"startAtOldestRecord": true,
"eventFormatMultiLine": false,
"channels": [
{
"name": "application",
"level": "Warning"
},
{
"name": "application",
"level": "Error"
}
]
},
{
"type": "File",
"directory": "c:\\inetpub\\logs",
"filter": "*.log",
"includeSubdirectories": true
},
{
"type": "ETW",
"eventFormatMultiLine": false,
"providers": [
{
"providerName": "IIS: WWW Server",
"providerGuid": "3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83",
"level": "Information"
},
{
"providerName": "Microsoft-Windows-IIS-Logging",
"providerGuid": "7E8AD27F-B271-4EA2-A783-A47BDE29143B",
"level": "Information"
}
]
}
]
}
}
And that's it for our Dockerfile.
3. Run the container
To run the container, first publish the binary with the following command. Execute the command from the “Application\RS.WindowsAuthDemo” directory.
dotnet publish -c release -o .\output
Switch Docker desktop to Windows containers.
Then, execute the following command to build the Docker image. Replace cryldkhz3mm5uwe with your container registry name. You can find it in the Azure portal.
docker build -t cryldkhz3mm5uwe.azurecr.io/gmsa:1.0 .
docker run -p 8080:80 cryldkhz3mm5uwe.azurecr.io/gmsa:1.0
Navigate to http://localhost:8080.
You should see a popup asking you to sign in.
At the moment, you cannot use Windows authentication. Whatever username and password you add here, you will always end up with an unauthorize response.
Plus, your local computer is not part of the domain.
Let’s see how to fix that.
Enable Windows authentication in Docker
The virtual machine vm-01 is part of the company domain, so we can pass a gMSA spec to run the container with Windows Authentication.
1. But first, what is a gMSA?
“Group managed service accounts (gMSAs) are domain accounts to help secure services. gMSAs can run on one server, or in a server farm, such as systems behind a network load balancing or Internet Information Services (IIS) server. After you configure your services to use a gMSA principal, account password management is handled by the Windows operating system (OS).”
https://learn.microsoft.com/en-us/azure/active-directory/architecture/service-accounts-group-managed
We will use a domain controller to create the gMSA.
2. Create a gMSA
Connect to the domain controller vm-dc-01 and execute the following commands:
1. Create a Key Distribution Services Root Key. This needs to be created only once. To validate that the root key is already created, you can run the command Get-KdsRootKey. The command returns no output if the get does not exist.
Add-KdsRootKey -EffectiveTime ((get-date).addhours(-10))
2. Create a group that is allowed to retrieve the gMSA and add vm-01 to it.
New-ADGroup -Name "gmsa group" -SamAccountName GmsaGroup -GroupScope DomainLocal -Description "Members of this group have access to the gMSA for containers."
Add-ADGroupMember -Identity "GmsaGroup" -Members "vm-01$"
3. Create the gMSA
New-ADServiceAccount -Name Gmsa -DNSHostName gmsa.mycompany.local -ServicePrincipalNames "host/gmsa", "host/gmsa.contoso.com" -PrincipalsAllowedToRetrieveManagedPassword GmsaGroup
4. Restart vm-01 to get its new group membership.
5. Run the following command to test that vm-01 can access to the gmsa. It should return True.
Test-ADServiceAccount gmsa
3. Create a GMSA spec file
Go to vm-01 to create a credential spec. Make sure Docker is running and run the following commands in a powershell terminal.
Get-WindowsCapability -Name Rsat.ActiveDirectory* -Online | Add-WindowsCapability -Online
Install-Module CredentialSpec
New-CredentialSpec -AccountName Gmsa
A new file C:\ProgramData\Docker\CredentialSpecs\mycompany_gmsa.json will be created.
4. Run the container with the GMSA spec file
From your local computer, push the docker image to the container registry. Again, do not forget to replace cryldkhz3mm5uwe with your container registry name.
az login
az acr login --name cryldkhz3mm5uwe
docker push cryldkhz3mm5uwe.azurecr.io/gmsa:1.0
From vm-01, run the docker command with the —security-opt parameter.
az login
az acr login --name cryldkhz3mm5uwe
docker pull cryldkhz3mm5uwe.azurecr.io/gmsa:1.0
docker run --security-opt "credentialspec=file://mycompany_gmsa.json" -p 8080:80 cryldkhz3mm5uwe.azurecr.io/gmsa:1.0
Open your browser and navigate to localhost:8080. Sign in with the user useradmin and the password.
You should see the Welcome page with your credentials at the top right corner.
Enable GMSA in Azure Kubernetes Services
With AKS, it is a bit different as we don’t want our node to be part of the domain. It would have been too complicated when adding new nodes to the cluster manually or with autoscaling.
This is why it is better to configure gMSA on the cluster without having to join the node to the domain.
To do so, we will use a standard user account and add the credentials to the Azure key vault.
AKS will use that account to access the gMSA.
1. Create a Standard domain account that can used the GMSA.
New-ADUser -Name "GmsaUser" -AccountPassword (ConvertTo-SecureString -AsPlainText "p@ssw0rd" -Force) -Enabled 1
2. Add this user account to the GmsaGroup.
Add-ADGroupMember -Identity "GmsaGroup" -Members "GmsaUser"
3. Add the credentials to Azure Key Vault. Replace kv-yldkhz3mm5uwe with your key vault name. Retrieve the name of your key vault from the Azure portal.
Notice the format of the value. It is domain\username:password
az keyvault secret set --vault-name kv-yldkhz3mm5uwe --name "GMSADomainUserCred" --value "mycompany.local\GmsaUser:p@ssw0rd"
4. Retrieve the principal ID of the kubelet identity with this command:
az identity show -n id-kubelet-01 -g rg-winauth-01 --query principalId
5. Create a file gmsa-spec.yaml. Replace the ObjectId in PluginInput with the kubelet principal ID. Replace SecretUri with the secret URI in key vault.
This yaml file is created based on the gmsa spec JSON file: C:\ProgramData\Docker\CredentialSpecs\mycompany_gmsa.json
apiVersion: windows.k8s.io/v1
kind: GMSACredentialSpec
metadata:
name: aks-gmsa-spec
credspec:
ActiveDirectoryConfig:
GroupManagedServiceAccounts:
- Name: Gmsa
Scope: mycompany
- Name: Gmsa
Scope: mycompany.local
HostAccountConfig:
PluginGUID: "{CCC2A336-D7F3-4818-A213-272B7924213E}"
PortableCcgVersion: "1"
PluginInput: "ObjectId=ccbcf7e6-9a9a-434e-8164-5b365312a41e;SecretUri=https://kv-yldkhz3mm5uwe.vault.azure.net/secrets/GMSADomainUserCred"
CmsPlugins:
- ActiveDirectory
DomainJoinConfig:
DnsName: mycompany.local
DnsTreeName: mycompany.local
Guid: c81feb2b-bd34-4568-9ebc-81e8a8ea1b51
MachineAccountName: Gmsa
NetBiosName: mycompany
Sid: S-1-5-21-3810415138-1110817698-543942465
6. Create a file gmsa-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: aks-gmsa-role
rules:
- apiGroups: ["windows.k8s.io"]
resources: ["gmsacredentialspecs"]
verbs: ["use"]
resourceNames: ["aks-gmsa-spec"]
7. Create a file gmsa-role-binding.yaml.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: allow-default-svc-account-read-on-aks-gmsa-spec
namespace: default
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: ClusterRole
name: aks-gmsa-role
apiGroup: rbac.authorization.k8s.io
8. Create a file deployment.yaml. Replace the container registry name with yours in the image fied.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: gmsa
name: gmsa
spec:
replicas: 1
selector:
matchLabels:
app: gmsa
template:
metadata:
labels:
app: gmsa
spec:
containers:
- name: gmsa
image: cryldkhz3mm5uwe.azurecr.io/gmsa:1.0
ports:
- containerPort: 80
livenessProbe:
initialDelaySeconds: 20
timeoutSeconds: 15
httpGet:
port: 80
path: /health
readinessProbe:
initialDelaySeconds: 20
timeoutSeconds: 15
httpGet:
port: 80
path: /health
resources:
limits:
memory: "512Mi"
cpu: "1000m"
requests:
memory: "256Mi"
cpu: "500m"
securityContext:
windowsOptions:
gmsaCredentialSpecName: aks-gmsa-spec
nodeSelector:
kubernetes.io/os: windows
9. Deploy those file to AKS.
kubectl apply -f gmsa-spec.yaml
kubectl apply -f gmsa-role.yaml
kubectl apply -f gmsa-role-binding.yaml
kubectl apply -f deployment.yaml
10. Retrieve the IP of the pod to access the application with the command:
kubectl get pod -o wide
11. From vm-01, navigate to the application with the IP address and sign in with useradmin.
You should be able to view the welcome page.
Important AKS configuration
GMSA has to be enabled on AKS. When creating your infrastructure, make sure to do the following.
In the main.bicep file, I added this block l.374.
The kubelet identity needs to be able to fetch the secret from the key vault. Since our key vault uses RBAC, I gave the role Key Vault Secret User to the kubelet identity l.302. If you are not using RBAC, create a new access policy.
Check the documentation for a reference of all Azure buit-in roles.
Conclusion
In conclusion, the decision to run Windows containers in Kubernetes is not without its challenges. The size of Windows containers tends to be significantly larger than their Linux counterparts, which can impact resource utilization and scalability.
However, there are situations where Windows containers are essential, particularly when migrating legacy applications that rely on Windows-specific features like Windows authentication.
Navigating Windows authentication in AKS can get a bit tricky and it has been quite a journey for me.
I hope this article will make it easier for you.
The code source is available on github: https://github.com/rceraline/devops/tree/main/2024-01-aks-windows-authentication
References
https://learn.microsoft.com/en-us/azure/aks/use-group-managed-service-accounts
Troubleshooting
For basic troubleshooting use this documentation.
https://learn.microsoft.com/en-us/azure/aks/use-group-managed-service-accounts#troubleshooting
For more advance troubleshooting, you can collect more logs to understand what is happening. RDP on the windows node and execute this script (already installed on the node).
C:\k\debug\collect-windows-logs.ps1
The script collects many logs. The following ones might help you with GMSA:
- CustomDataSetupScript.log
- Microsoft-Windows-Containers-CCG%4Admin.evtx
- Microsoft-AKSGMSAPlugin%4Admin.evtx
Once the script executed, retrieve the zip file generated by opening a windows explorer to the remote address \\10.0.2.33\c$\k\debug.
10.0.2.33 being the ip address of the node. Replace it with yours.
To RDP on a windows node, follow this documentation.
https://learn.microsoft.com/en-us/azure/aks/rdp?tabs=azure-cli
