Introduction After learning Docker, Kubernetes, Helm, Azure and Azure DevOps separately, I wanted to put everything together into one practical project. The goal was not to build a complicated application. The goal was to understand what happens when application code moves from source control into a real deployment environment. The final flow I wanted to understand was: Developer | v GitHub Repository | v Azure DevOps | v CI/CD Pipeline | +----------------------+ | | v v Build + Test Docker Build | v Azure Container Registry | v Helm | v AKS | v DEV | v Running Application The application itself was intentionally simple: Frontend Backend PostgreSQL database The infrastructure around the application was where most of the learning happened. I wanted to understand: Azure Resource Groups Azure Container Registry Azure Kubernetes Service Azure networking Private networking PostgreSQL Flexible Server Private DNS Docker Kubernetes namespaces Deployments Services ConfigMaps Secrets Helm Azure DevOps Azure service connections Workload Identity Federation Azure RBAC Kubernetes RBAC CI/CD Troubleshooting For this project I focused on DEV first. UAT, PROD and approval gates will be added later. 1. The Final Architecture The DEV environment ended up looking roughly like this: GitHub | v Azure DevOps | v Build + Test + Docker | v Azure Container Registry | v Helm | v Azure Kubernetes Service | +------------+------------+ | | v v frontend namespace backend namespace | | v v Frontend Deployment Backend Deployment | | v v LoadBalancer Service ClusterIP Service | | v | Public IP / Browser | v PostgreSQL Flexible Server | v Private Network | v Private DNS Zone The important design decision was that PostgreSQL was not installed inside AKS. Instead, I used Azure Database for PostgreSQL Flexible Server. That means AKS is responsible for running application workloads while Azure manages the database service. 2. The Project The project repository is: k8app-azure-devops Repository: https://github.com/Desmondgoldsmith/k8app-azure-devops The repository contains the frontend, backend and Kubernetes/Helm configuration. A simplified structure is: k8app-azure-devops/ │ ├── backend/ │ ├── app/ │ └── backend.dockerfile │ ├── frontend/ │ └── frontend.dockerfile │ ├── helm/ │ └── k8app/ │ ├── Chart.yaml │ ├── values.yaml │ └── templates/ │ ├── frontend-deployment.yaml │ ├── frontend-service.yaml │ ├── backend-deployment.yaml │ ├── backend-service.yaml │ ├── backend-configmap.yaml │ └── backend-secret.yaml │ └── azure-pipelines.yml The exact application is less important than the infrastructure around it. 3. Azure Subscription and Region I used an Azure for Students subscription. The region I used was: South Africa North I chose the region partly because my student subscription had restrictions on which Azure regions I could use. For example, some regions were unavailable under the subscription's policy. Before creating resources, it is worth checking which regions your subscription allows. You can do that from the Azure Portal when creating a resource, or with: az account list-locations -o table 4. Create the Azure Resource Group The first Azure resource I created was the Resource Group. I used: Resource Group: devops-aks-lab-rg-sa Region: South Africa North Why create a Resource Group? A Resource Group is a logical container for related Azure resources. Instead of having the project's resources scattered across Azure, I could put the main resources into one place: devops-aks-lab-rg-sa | +-- ACR +-- AKS +-- PostgreSQL +-- Networking resources This also makes permissions, management and cleanup easier. Azure Portal Go to: Azure Portal ↓ Resource groups ↓ Create Then select: Subscription: Azure for Students Resource group: devops-aks-lab-rg-sa Region: South Africa North Click: Review + create ↓ Create Azure CLI I could also create it using: az login Then: az group create \ --name devops-aks-lab-rg-sa \ --location southafricanorth Verify: az group show \ --name devops-aks-lab-rg-sa \ -o table 5. Create Azure Container Registry The next component was Azure Container Registry. I created: ACR name: dessydevopsacr The registry login server is: dessydevopsacr.azurecr.io Why do we need ACR? The CI pipeline builds Docker images. Those images need somewhere to live before Kubernetes can deploy them. So ACR becomes the central image repository: Azure DevOps | | Docker image v Azure Container Registry | | Docker image v AKS For this project I had two images: dessydevopsacr.azurecr.io/k8app-backend dessydevopsacr.azurecr.io/k8app-frontend Azure Portal Go to: Azure Portal ↓ Create a resource ↓ Search: Container Registry ↓ Create Under Basics, select: Subscription: Azure for Students Resource group: devops-aks-lab-rg-sa Registry name: dessydevopsacr Location: South Africa North SKU: Basic For this project I kept the ACR admin account disabled. Then: Review + create ↓ Create Azure CLI Equivalent CLI: az acr create \ --resource-group devops-aks-lab-rg-sa \ --name dessydevopsacr \ --location southafricanorth \ --sku Basic \ --admin-enabled false Verify: az acr show \ --name dessydevopsacr \ --resource-group devops-aks-lab-rg-sa \ --query "{name:name,loginServer:loginServer,sku:sku.name}" \ -o table Expected login server: dessydevopsacr.azurecr.io Why disable the ACR admin account? ACR can provide a static username and password through its admin account. I did not want the CI/CD pipeline to depend on that type of long-lived credential. Instead, Azure DevOps uses a service connection. That gives us a cleaner authentication model: Azure DevOps | v Service Connection | v Azure / ACR 6. Create the Virtual Network Because PostgreSQL was going to use private networking, networking needed to be considered before creating the database. The basic idea was to have separate network areas for different workloads. Conceptually: Virtual Network | +-- AKS subnet | +-- PostgreSQL subnet The PostgreSQL subnet we used was: postgres-subnet 10.0.2.0/24 A /24 gives a defined block of private addresses for the database subnet. The important reason for separating the database into its own subnet is that the database has a different purpose and security boundary from the application workloads. Important note about the IP plan The project definitely used: postgres-subnet: 10.0.2.0/24 The AKS cluster also used: Pod CIDR: 10.244.0.0/16 Service CIDR: 10.240.0.0/16 These are not the same thing as the Azure VNet subnet. The exact Azure VNet/AKS subnet CIDR is not preserved in my project notes, so I am intentionally not inventing it here. When reproducing the project, the network should be designed so the Azure VNet contains non-overlapping subnet ranges, for example: VNet | +-- AKS subnet | +-- PostgreSQL subnet | +-- 10.0.2.0/24 The important rule is: Do not create overlapping CIDR ranges. For example, your VNet cannot have two different subnets that both contain the same addresses. 7. Create the PostgreSQL Subnet In the Azure Portal, I created the subnet from the Virtual Network. Go to: Azure Portal ↓ Virtual networks ↓ Select your project VNet ↓ Subnets ↓ + Subnet Create: Subnet name: postgres-subnet Subnet address range: 10.0.2.0/24 Save the subnet. The purpose is to reserve a dedicated private network range for PostgreSQL. 8. Why PostgreSQL Was Private I deliberately did not expose PostgreSQL to the public internet. Instead: AKS | | Private network v PostgreSQL rather than: Internet | v Public PostgreSQL endpoint This is closer to how a production architecture would normally be designed. The application workloads communicate with the database through private networking. 9. Create the Private DNS Zone This was one of the concepts that took some time to understand. Creating a private IP for PostgreSQL is not enough. Applications normally connect to PostgreSQL using a hostname such as: dessy-k8app-postgres.postgres.database.azure.com The hostname needs to resolve to the private IP address. That is where Private DNS comes in. I created: private.postgres.database.azure.com What does Private DNS do? Think of DNS as a phonebook. The application knows: dessy-k8app-postgres.postgres.database.azure.com but the network ultimately needs an IP address. Private DNS provides the mapping: Hostname | v Private DNS | v Private IP In this project, the PostgreSQL private IP resolved to: 10.0.2.4 So the flow became: AKS Pod | | asks DNS: | "Where is | dessy-k8app-postgres.postgres.database.azure.com?" | v Private DNS Zone | v 10.0.2.4 | v PostgreSQL Without DNS resolution, an application could know the hostname but would not know where to send the network traffic. 10. Create the Private DNS Zone in Azure Portal Go to: Azure Portal ↓ Create a resource ↓ Search: Private DNS zones ↓ Create Set: Resource group: devops-aks-lab-rg-sa Name: private.postgres.database.azure.com Click: Review + create ↓ Create After creating it, the DNS zone needs to be linked to the VNet. Go to: Private DNS zone ↓ Virtual network links ↓ + Add Select the VNet used by the application/database. This allows resources in the VNet to use that private DNS zone. 11. Create Azure Database for PostgreSQL Flexible Server I created: PostgreSQL server: dessy-k8app-postgres Region: South Africa North PostgreSQL: 16 SKU: Standard_B1ms Database: app The hostname is: dessy-k8app-postgres.postgres.database.azure.com The server uses private networking. Why use PostgreSQL Flexible Server? Instead of running PostgreSQL inside Kubernetes: AKS | +-- PostgreSQL Pod I used: AKS | | private network v Azure PostgreSQL Flexible Server This separates application compute from database infrastructure. AKS can concentrate on running: frontend backend while Azure manages the PostgreSQL service. 12. Create PostgreSQL in the Azure Portal Go to: Azure Portal ↓ Create a resource ↓ Search: Azure Database for PostgreSQL Flexible Server ↓ Create Under Basics, configure: Subscription: Azure for Students Resource group: devops-aks-lab-rg-sa Server name: dessy-k8app-postgres Region: South Africa North PostgreSQL version: 16 Workload type: Development / appropriate low-cost option Compute: Standard_B1ms Configure the administrator: Administrator username: k8appadmin Administrator password: <your password> Do not publish the password. 13. Configure PostgreSQL Networking During PostgreSQL creation, select private access rather than public access. The database needs to use the VNet and the dedicated subnet: Virtual network: <project VNet> Subnet: postgres-subnet The private DNS configuration should use: private.postgres.database.azure.com The important relationship is: PostgreSQL | v postgres-subnet 10.0.2.0/24 | v Private DNS private.postgres.database.azure.com Azure then creates the private networking relationship. 14. Create the Application Database After creating the server, I created the database: app From the PostgreSQL resource, go to: PostgreSQL Flexible Server ↓ Databases ↓ + Add Create: Database name: app The application will later connect using: Server: dessy-k8app-postgres.postgres.database.azure.com Database: app User: k8appadmin The password is kept secret. 15. Verify PostgreSQL Networking From AKS Creating the database is not enough. I wanted to answer a more important question: Can a workload running inside AKS actually reach PostgreSQL? I tested this from inside Kubernetes. First I checked DNS resolution. kubectl run psql-test \ --rm -it \ --image=postgres:16 \ --restart=Never \ -- \ getent hosts dessy-k8app-postgres.postgres.database.azure.com The hostname resolved to the private PostgreSQL IP. In our case: 10.0.2.4 This proved that the private DNS path was working. 16. Test the PostgreSQL Connection I then used a temporary PostgreSQL client pod. kubectl run psql-test \ --rm -it \ --image=postgres:16 \ --restart=Never \ -- \ psql "host=dessy-k8app-postgres.postgres.database.azure.com port=5432 dbname=app user=k8appadmin sslmode=require" When prompted, I entered the database password. The connection succeeded over TLS. That proved this path was working: AKS Pod | v Private DNS | v Private IP | v PostgreSQL Flexible Server This was an important test because it separated database/networking problems from application problems. 17. Create the AKS Cluster The next major component was Azure Kubernetes Service. I created: AKS: dessy-aks-cluster Resource Group: devops-aks-lab-rg-sa Region: South Africa North The cluster used: Azure CNI Overlay Pod CIDR: 10.244.0.0/16 Service CIDR: 10.240.0.0/16 Load Balancer: Standard I also enabled Microsoft Entra integration and Kubernetes RBAC. 18. Why Azure CNI Overlay? AKS networking can be configured in different ways. For this project I used Azure CNI Overlay. One important distinction is that these CIDRs: 10.244.0.0/16 10.240.0.0/16 are Kubernetes networking ranges. They are different from the Azure VNet subnet ranges. I found this distinction important when learning AKS networking because it is easy to look at an IP such as: 10.244.x.x and assume it is an Azure VNet IP. It is not. The simplified model is: Azure VNet | +-- Azure subnet | +-- AKS nodes | +-- Kubernetes pod networking | +-- Kubernetes service networking 19. Create AKS in Azure Portal Go to: Azure Portal ↓ Create a resource ↓ Search: Kubernetes Service ↓ Create Under Basics, configure: Subscription: Azure for Students Resource group: devops-aks-lab-rg-sa Cluster preset: appropriate development option Kubernetes cluster name: dessy-aks-cluster Region: South Africa North Configure the node pool with one node for this learning environment. 20. Configure AKS Identity Under the identity/security configuration, use: Managed identity rather than manually managing long-lived credentials. This allows Azure resources to work with managed identities. 21. Configure Microsoft Entra ID and Kubernetes RBAC I enabled Microsoft Entra integration. The cluster uses Microsoft Entra ID for identity and Kubernetes RBAC for authorization. This became an important lesson later. Authentication and authorization are not the same thing. Think about them as: Microsoft Entra ID | | Who are you? v Authentication | v Kubernetes RBAC | | What are you allowed to do? v Authorization 22. Configure AKS Networking For the cluster networking, I used: Network plugin: Azure CNI Network plugin mode: Overlay Pod CIDR: 10.244.0.0/16 Service CIDR: 10.240.0.0/16 I also used a Standard Load Balancer. This is important because the frontend Kubernetes Service was later configured as: type: LoadBalancer Azure then provisions an Azure Load Balancer/public IP for that service. 23. Enable OIDC and Workload Identity I enabled: OIDC issuer Workload identity These capabilities are useful for modern identity-based authentication between Kubernetes workloads and Azure services. They also make it possible to avoid embedding long-lived Azure credentials inside applications. 24. Connect kubectl to AKS After the cluster was created, I retrieved its credentials: az aks get-credentials \ --resource-group devops-aks-lab-rg-sa \ --name dessy-aks-cluster \ --overwrite-existing Then: kubectl get nodes I expected the node to show: Ready This confirmed that my local machine could communicate with the cluster. 25. AKS Identity and ACR There are several identities involved in this project. This was one of the most important things I learned. The AKS cluster has a kubelet managed identity. I checked it with: az aks show \ --resource-group devops-aks-lab-rg-sa \ --name dessy-aks-cluster \ --query identityProfile.kubeletidentity.clientId \ -o tsv The kubelet identity needs permission to pull images from ACR. The permission required is: AcrPull The flow is: AKS Kubelet Identity | | AcrPull v Azure Container Registry | v Docker Image 26. Give AKS AcrPull Permission In Azure Portal, go to: Container Registry ↓ dessydevopsacr ↓ Access control (IAM) ↓ Role assignments Add a role assignment. Select: Role: AcrPull Assign it to the AKS kubelet managed identity. After this: AKS | | AcrPull v ACR | v k8app-backend image k8app-frontend image The important distinction is: Azure DevOps pushes the image. AKS pulls the image. These are two different operations and can use two different identities. 27. Understanding the Three Identity Layers This project involved three different concepts. OIDC / Workload Identity This answers: Who are you? For Azure DevOps: Azure DevOps | v OIDC / Workload Identity Federation | v Microsoft Entra ID Azure RBAC This answers: What can you do in Azure? For example: Azure DevOps identity | v Contributor | v Resource Group Kubernetes RBAC This answers: What can you do inside Kubernetes? For example: Azure DevOps identity | v Kubernetes RoleBinding | v edit | +-- frontend namespace | +-- backend namespace The simplified mental model is: OIDC ↓ Who are you? Azure RBAC ↓ What can you do in Azure? Kubernetes RBAC ↓ What can you do in Kubernetes? 28. Create the Azure DevOps Project I created an Azure DevOps project: DessyTest-1 Go to: Azure DevOps ↓ New Project Create: Project name: DessyTest-1 The GitHub repository remains the source repository. Azure DevOps is being used for the CI/CD pipeline. The conceptual relationship is: GitHub | v Azure DevOps | v Pipeline 29. Connect GitHub to Azure DevOps Inside Azure DevOps, create/configure the GitHub connection. Go to: Azure DevOps ↓ Project Settings ↓ Service connections Create the GitHub connection and authorize the repository. The purpose is to allow Azure DevOps to retrieve the repository source code. 30. Create the ACR Service Connection Azure DevOps needs a way to authenticate to ACR. Go to: Azure DevOps ↓ Project Settings ↓ Service connections ↓ New service connection Select: Docker Registry Then configure it for Azure Container Registry. I created the service connection: dessydevopsacr This connection is used by the Docker task in the pipeline. Conceptually: Azure DevOps | v ACR Service Connection | v Azure Container Registry 31. Create the AKS Azure Service Connection The deployment stage needs Azure access so it can obtain AKS credentials. I created: dessy-aks-deploy Go to: Azure DevOps ↓ Project Settings ↓ Service connections ↓ New service connection Select the Azure Resource Manager connection type. Use: Workload Identity Federation rather than a long-lived client secret. This gives: Azure DevOps | v OIDC / Workload Identity Federation | v Microsoft Entra ID | v Azure 32. Give the Azure DevOps Identity Azure Permissions The service principal used by the deployment service connection needs permission to work with the Azure resources. For this learning environment, I assigned: Contributor at the resource group scope: devops-aks-lab-rg-sa In Azure Portal: Resource Group ↓ devops-aks-lab-rg-sa ↓ Access control (IAM) ↓ Add role assignment Select: Contributor Then select the service principal used by: dessy-aks-deploy This was intentionally broad for the learning environment. A production environment should use more restrictive permissions where practical. 33. Kubernetes RBAC for Azure DevOps Getting Azure permissions was not enough. At one point the pipeline could authenticate to Azure but Kubernetes still rejected some operations. That taught me an important lesson: Azure authentication and Kubernetes authorization are separate problems. The Azure DevOps service principal was: <AZURE_DEVOPS_SERVICE_PRINCIPAL_OBJECT_ID> I created a RoleBinding in the: frontend namespace and another in: backend The binding gives the identity the Kubernetes: edit ClusterRole. Example: apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: azure-devops-deployer namespace: backend subjects: - kind: User name: <AZURE_DEVOPS_SERVICE_PRINCIPAL_OBJECT_ID> apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: edit apiGroup: rbac.authorization.k8s.io The frontend binding is the same except for: namespace: frontend 34. Test Kubernetes RBAC I tested whether the Azure DevOps identity could create deployments. For backend: kubectl auth can-i create deployments \ --as=<AZURE_DEVOPS_SERVICE_PRINCIPAL_OBJECT_ID> \ -n backend For frontend: kubectl auth can-i create deployments \ --as=<AZURE_DEVOPS_SERVICE_PRINCIPAL_OBJECT_ID> \ -n frontend Both returned: yes I deliberately did not give the pipeline: cluster-admin because that would give it unnecessarily broad access to the whole cluster. 35. Create the Kubernetes Namespaces I separated the frontend and backend workloads. The namespaces were: frontend backend Create them: kubectl create namespace frontend kubectl create namespace backend Verify: kubectl get namespaces The reason for separate namespaces is organization and isolation. Instead of: default | +-- frontend +-- backend +-- everything else we have: frontend namespace | +-- frontend Deployment +-- frontend Service backend namespace | +-- backend Deployment +-- backend Service +-- ConfigMap +-- Secret It also gives us a clean foundation for namespace-level RBAC. 36. Why Kubernetes Deployment? A Deployment describes how Kubernetes should run an application. For example: kind: Deployment tells Kubernetes: I want this application running. It controls things such as: Number of replicas Container image Container ports Pod labels Updates For example: replicas: 1 means we currently want one pod. 37. Why Kubernetes Service? Pods are temporary. Their IP addresses can change. A Kubernetes Service gives applications a stable way to communicate with a group of Pods. The model is: Pod | | temporary IP v Service | | stable name/IP v Other application There are two different Services in this project. Frontend: LoadBalancer Backend: ClusterIP 38. Frontend Service The frontend Service is: apiVersion: v1 kind: Service metadata: name: k8app-frontend namespace: frontend spec: type: LoadBalancer selector: app: k8app-frontend ports: - port: 80 targetPort: 80 The important part is: type: LoadBalancer This tells AKS that the service should be externally reachable. The flow is: Browser | v Azure Load Balancer | v k8app-frontend Service | v Frontend Pod 39. Backend Service The backend Service is: apiVersion: v1 kind: Service metadata: name: k8app-backend namespace: backend spec: type: ClusterIP selector: app: k8app-backend ports: - port: 80 targetPort: 80 The important part is: type: ClusterIP ClusterIP means the service is intended to be reachable internally within Kubernetes. Conceptually: Frontend / internal caller | v Backend ClusterIP | v Backend Pod 40. Create the Helm Chart Instead of maintaining raw Kubernetes YAML for every environment, I used Helm. The chart is: helm/k8app Create the directory structure: helm/ └── k8app/ ├── Chart.yaml ├── values.yaml └── templates/ 41. Create Chart.yaml The chart definition is: apiVersion: v2 name: k8app description: Helm chart for the K8App frontend and backend type: application version: 0.1.0 appVersion: "1.0.0" The important concept is that Helm packages our Kubernetes configuration into one deployable unit. Instead of manually doing: kubectl apply -f ... kubectl apply -f ... kubectl apply -f ... we can do: helm upgrade --install ... 42. Create values.yaml The values file contains environment-specific values. frontend: replicas: 1 image: repository: dessydevopsacr.azurecr.io/k8app-frontend tag: "" backend: replicas: 1 image: repository: dessydevopsacr.io/k8app-backend tag: "" config: projectName: "k8app" postgresServer: "dessy-k8app-postgres.postgres.database.azure.com" postgresDb: "app" secret: postgresUser: "k8appadmin" postgresPassword: "" The correct backend registry is: dessydevopsacr.azurecr.io so the backend repository should be: repository: dessydevopsacr.azurecr.io/k8app-backend The final version is therefore: frontend: replicas: 1 image: repository: dessydevopsacr.azurecr.io/k8app-frontend tag: "" backend: replicas: 1 image: repository: dessydevopsacr.azurecr.io/k8app-backend tag: "" config: projectName: "k8app" postgresServer: "dessy-k8app-postgres.postgres.database.azure.com" postgresDb: "app" secret: postgresUser: "k8appadmin" postgresPassword: "" Notice that the password is blank. That is intentional. The pipeline supplies it securely during deployment. 43. Why Helm Values Exist The values file lets us separate: Application template from: Environment values For example: image: repository: ... tag: "" The chart doesn't need to know which pipeline build is being deployed. The pipeline can provide: --set backend.image.tag=$(Build.BuildId) and: --set frontend.image.tag=$(Build.BuildId) So: Pipeline Build ID | v Docker image tag | v Helm values | v Kubernetes Deployment This creates a traceable relationship between a pipeline run and the deployed image. 44. Frontend Deployment The frontend Deployment is: apiVersion: apps/v1 kind: Deployment metadata: name: k8app-frontend namespace: frontend spec: replicas: {{ .Values.frontend.replicas }} selector: matchLabels: app: k8app-frontend template: metadata: labels: app: k8app-frontend spec: containers: - name: frontend image: "{{ .Values.frontend.image.repository }}:{{ .Values.frontend.image.tag }}" ports: - containerPort: 80 The important Helm expression is: {{ .Values.frontend.image.tag }} Helm replaces this with the actual value during deployment. For example, if the pipeline passes: --set frontend.image.tag=21 the resulting Kubernetes image becomes: dessydevopsacr.azurecr.io/k8app-frontend:21 45. Backend Deployment The backend Deployment contains the container and environment configuration. apiVersion: apps/v1 kind: Deployment metadata: name: k8app-backend namespace: backend spec: replicas: {{ .Values.backend.replicas }} selector: matchLabels: app: k8app-backend template: metadata: labels: app: k8app-backend spec: containers: - name: backend image: "{{ .Values.backend.image.repository }}:{{ .Values.backend.image.tag }}" ports: - containerPort: 80 env: - name: PROJECT_NAME valueFrom: configMapKeyRef: name: k8app-backend-config key: PROJECT_NAME - name: POSTGRES_SERVER valueFrom: configMapKeyRef: name: k8app-backend-config key: POSTGRES_SERVER - name: POSTGRES_DB valueFrom: configMapKeyRef: name: k8app-backend-config key: POSTGRES_DB - name: POSTGRES_USER valueFrom: secretKeyRef: name: k8app-backend-secret key: POSTGRES_USER - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: name: k8app-backend-secret key: POSTGRES_PASSWORD This is where ConfigMaps and Secrets become important. 46. What Is a ConfigMap? A ConfigMap stores non-sensitive configuration. For this application, things such as: PROJECT_NAME POSTGRES_SERVER POSTGRES_DB do not need to be treated as passwords. So I created: k8app-backend-config 47. Create the ConfigMap The Helm template is: apiVersion: v1 kind: ConfigMap metadata: name: k8app-backend-config namespace: backend data: PROJECT_NAME: {{ .Values.backend.config.projectName | quote }} POSTGRES_SERVER: {{ .Values.backend.config.postgresServer | quote }} POSTGRES_DB: {{ .Values.backend.config.postgresDb | quote }} The resulting Kubernetes object looks conceptually like: k8app-backend-config | +-- PROJECT_NAME=k8app +-- POSTGRES_SERVER=dessy-k8app-postgres.postgres.database.azure.com +-- POSTGRES_DB=app You can inspect it with: kubectl get configmap k8app-backend-config -n backend Or: kubectl describe configmap k8app-backend-config -n backend 48. What Is a Kubernetes Secret? A Secret is intended for sensitive configuration. In our case: POSTGRES_USER POSTGRES_PASSWORD The password should not be stored directly in: Deployment YAML values.yaml Git Instead, I created: k8app-backend-secret 49. Create the Kubernetes Secret The Helm template is: apiVersion: v1 kind: Secret metadata: name: k8app-backend-secret namespace: backend type: Opaque stringData: POSTGRES_USER: {{ .Values.backend.secret.postgresUser | quote }} POSTGRES_PASSWORD: {{ .Values.backend.secret.postgresPassword | quote }} The password is supplied at deployment time. I did not commit the actual password to Git. 50. How configMapKeyRef Works This part initially looked confusing: valueFrom: configMapKeyRef: name: k8app-backend-config key: POSTGRES_SERVER But it is actually simple. It means: Go to the ConfigMap named k8app-backend-config, find the key called POSTGRES_SERVER, and inject its value into the container as an environment variable named POSTGRES_SERVER. So: ConfigMap k8app-backend-config | +-- POSTGRES_SERVER | v Backend container | v POSTGRES_SERVER The same thing happens for: PROJECT_NAME POSTGRES_DB 51. How secretKeyRef Works This: valueFrom: secretKeyRef: name: k8app-backend-secret key: POSTGRES_PASSWORD means: Go to the Secret named k8app-backend-secret, find the POSTGRES_PASSWORD key, and inject its value into the container as the POSTGRES_PASSWORD environment variable. So: Kubernetes Secret | +-- POSTGRES_PASSWORD | v Backend container | v POSTGRES_PASSWORD This keeps sensitive configuration separate from normal configuration. 52. Why the Backend Initially Failed Before the ConfigMap and Secret were configured correctly, the backend container crashed. The backend expected environment variables such as: PROJECT_NAME POSTGRES_SERVER POSTGRES_USER POSTGRES_PASSWORD POSTGRES_DB The application's configuration code used environment-based configuration. Without those variables, the application could not construct its database configuration. This is an example of why logs are important. Instead of guessing, I checked: kubectl logs <backend-pod> -n backend The logs showed that required configuration values were missing. That led to: Backend Crash | v Check Pod logs | v Missing environment variables | v Create ConfigMap + Secret | v Reference them from Deployment | v Backend starts 53. Verify the Backend Configuration Check the ConfigMap: kubectl get configmap -n backend Check the Secret: kubectl get secret -n backend Check the Deployment: kubectl get deployment k8app-backend -n backend Check the Pod: kubectl get pods -n backend If the pod is running: 1/1 Running then the application container has successfully started. 54. Verify the Backend Logs Run: kubectl logs deployment/k8app-backend -n backend If something fails, I normally investigate in this order: kubectl get pods | v kubectl describe pod | v kubectl logs | v Check ConfigMap / Secret | v Check Service | v Check database/network This is much better than randomly changing YAML. 55. Verify the Database From the Backend Namespace Because the backend needs PostgreSQL, I verified that the backend environment contained the correct configuration. The database host is: dessy-k8app-postgres.postgres.database.azure.com The database is: app The username is: k8appadmin The password remains secret. 56. Validate the Helm Chart Before Deploying One of the most useful Helm commands is: helm template test-release ./helm/k8app This does not deploy anything. It renders the Helm templates into Kubernetes YAML. This allows us to catch template problems before sending anything to AKS. For example: helm template test-release ./helm/k8app If there is a problem such as: nil pointer evaluating interface {}.create it usually means the template is trying to access a value that doesn't exist in values.yaml. This is one reason I use: helm template before: helm upgrade 57. Deploy With Helm The main deployment command is: helm upgrade --install k8app helm/k8app \ --namespace backend \ --set backend.image.tag=21 \ --set frontend.image.tag=21 \ --wait \ --timeout 5m The number: 21 was the image tag for that pipeline/build. In the real pipeline, we don't manually type this. Azure DevOps provides: $(Build.BuildId) So the pipeline can automatically deploy the image produced by that build. 58. Why Helm Release Is in the Backend Namespace One slightly unusual part of this design is that the Helm release is installed using: --namespace backend while the chart itself contains resources in both: frontend backend Therefore: helm list -n backend shows: k8app while: helm list -n frontend does not. That does not mean the frontend wasn't deployed. It means the Helm release itself is stored/tracked in the backend namespace while the chart creates resources in both namespaces. 59. Verify the Helm Release Run: helm list -n backend Expected: NAME NAMESPACE REVISION STATUS CHART k8app backend 3 deployed k8app-0.1.0 You can also inspect: helm status k8app -n backend 60. Verify Kubernetes Deployments Check frontend: kubectl get deployments -n frontend Check backend: kubectl get deployments -n backend Expected conceptually: frontend k8app-frontend backend k8app-backend 61. Verify Pods Run: kubectl get pods -n frontend and: kubectl get pods -n backend Healthy pods should show: 1/1 Running For example: k8app-frontend-xxxxxxxxxx 1/1 Running and: k8app-backend-xxxxxxxxxx 1/1 Running 62. Verify Services Frontend: kubectl get service -n frontend Backend: kubectl get service -n backend The frontend should be: LoadBalancer while the backend should be: ClusterIP 63. The LoadBalancer Problem Initially, the frontend Service showed: EXTERNAL-IP <pending> At first it was tempting to think Kubernetes was broken. Instead, I used: kubectl describe service k8app-frontend -n frontend The important event was: PublicIPCountLimitReached The subscription had reached the regional public IP limit. The message indicated that the subscription could not create more than three public IP addresses in that region. This was a good example of why: kubectl describe is useful. kubectl get tells us what is happening. kubectl describe often helps explain why. 64. Troubleshooting the Public IP Limit I checked existing AKS resources. One older AKS cluster was: rest-go-aks I checked its state: az aks show \ --resource-group devops-aks-lab-rg-sa \ --name rest-go-aks \ --query "{name:name,powerState:powerState.code}" \ -o table It showed: Name PowerState ------------ ---------- rest-go-aks Stopped The old cluster was no longer needed for this project. I removed it: az aks delete \ --resource-group devops-aks-lab-rg-sa \ --name rest-go-aks \ --yes \ --no-wait Then I checked the public IP resources: az network public-ip list \ --query "[?location=='southafricanorth'].{Name:name,ResourceGroup:resourceGroup,IP:ipAddress}" \ -o table The old resources were gone. The frontend LoadBalancer was then able to obtain a public IP. 65. Final Frontend Public IP The frontend Service eventually received: 4.253.76.190 The browser application became accessible through: http://4.253.76.190 The final flow was therefore: Browser | v 4.253.76.190 | v Azure Load Balancer | v k8app-frontend Service | v Frontend Pod 66. Important Lesson From the LoadBalancer Failure The Kubernetes Service itself was not necessarily the problem. The chain was: Service | v Azure Load Balancer | v Azure Public IP The Azure subscription had reached its public IP quota. So the correct troubleshooting process was: EXTERNAL-IP pending | v kubectl describe service | v PublicIPCountLimitReached | v Check Azure resources | v Find unused AKS cluster | v Delete unused cluster | v LoadBalancer gets public IP This was one of the most useful troubleshooting lessons in the project. 67. Azure DevOps Variable Group The database password should not be placed directly in the pipeline YAML. So I created a variable group: k8app-dev-secrets Inside it: POSTGRES_PASSWORD The variable was marked as secret. 68. Create the Variable Group in Azure DevOps Go to: Azure DevOps ↓ Pipelines ↓ Library ↓ + Variable group Create: Variable group name: k8app-dev-secrets Add: Name: POSTGRES_PASSWORD Value: <your PostgreSQL password> Enable: Keep this value secret Save the variable group. 69. Why the Pipeline Uses - group The pipeline contains: variables: - group: k8app-dev-secrets This means: Load the variables stored inside the Azure DevOps variable group named k8app-dev-secrets into this pipeline. Later, the pipeline can reference: $(POSTGRES_PASSWORD) So the flow is: Azure DevOps Library | v k8app-dev-secrets | +-- POSTGRES_PASSWORD | v Azure DevOps Pipeline | v Helm | v Kubernetes Secret | v Backend Pod The actual password should never be committed to Git. 70. The Azure DevOps Pipeline The pipeline builds both images and then deploys them to DEV. The important variables are: variables: - group: k8app-dev-secrets - name: dockerRegistryServiceConnection value: 'a41b9e21-7348-4152-b6ca-249e8c78aae9' - name: containerRegistry value: 'dessydevopsacr.azurecr.io' - name: vmImageName value: 'ubuntu-latest' - name: tag value: '$(Build.BuildId)' The important part is: tag: '$(Build.BuildId)' This means every pipeline run can produce a unique image tag. For example: Build ID = 21 produces: dessydevopsacr.azurecr.io/k8app-backend:21 dessydevopsacr.io/k8app-frontend:21 The correct registry hostname is: dessydevopsacr.azurecr.io 71. Build Backend Image The backend job uses Docker@2. Conceptually: - task: Docker@2 inputs: command: buildAndPush containerRegistry: $(dockerRegistryServiceConnection) repository: k8app-backend dockerfile: backend/backend.dockerfile buildContext: backend tags: | $(tag) The important pieces are: Dockerfile: backend/backend.dockerfile Build context: backend Repository: k8app-backend Tag: Build.BuildId The result goes to: dessydevopsacr.azurecr.io/k8app-backend:<BuildID> 72. Build Frontend Image The frontend job does the same thing. - task: Docker@2 inputs: command: buildAndPush containerRegistry: $(dockerRegistryServiceConnection) repository: k8app-frontend dockerfile: frontend/frontend.dockerfile buildContext: frontend tags: | $(tag) The result is: dessydevopsacr.azurecr.io/k8app-frontend:<BuildID> 73. Deploy Stage After the Build stage succeeds, the pipeline runs: DeployDEV The deployment uses: dessy-aks-deploy as its Azure service connection. The pipeline obtains AKS credentials: az aks get-credentials \ --resource-group devops-aks-lab-rg-sa \ --name dessy-aks-cluster \ --overwrite-existing Then Helm performs the deployment. 74. Helm Deployment From Azure DevOps The deployment command is: helm upgrade --install k8app helm/k8app \ --namespace backend \ --set frontend.image.tag=$(Build.BuildId) \ --set backend.image.tag=$(Build.BuildId) \ --set backend.secret.postgresPassword="$(POSTGRES_PASSWORD)" \ --wait \ --timeout 5m Let's break this down. helm upgrade --install helm upgrade --install means: If the release exists, upgrade it. If it doesn't exist, install it. This makes the pipeline reusable. Release name k8app This is the Helm release name. Chart helm/k8app This tells Helm where the chart is. Namespace --namespace backend This is where the Helm release is tracked. Frontend image --set frontend.image.tag=$(Build.BuildId) The pipeline automatically passes the current build ID into Helm. Backend image --set backend.image.tag=$(Build.BuildId) The backend receives the same build ID. That means one pipeline run corresponds to the same application version: Build 21 | +-- frontend:21 | +-- backend:21 PostgreSQL password --set backend.secret.postgresPassword="$(POSTGRES_PASSWORD)" The password comes from the Azure DevOps variable group. It is then passed into the Helm chart and rendered into the Kubernetes Secret. 75. The Complete Image Flow This is the relationship I wanted to understand: Azure DevOps Build | | Build.BuildId = 21 | +--------------------+ | | v v frontend:21 backend:21 | | +---------+----------+ | v ACR | v Helm | v AKS This gives us traceability between: Pipeline run ↓ Docker image ↓ Helm release ↓ Kubernetes deployment ↓ Running application 76. Final Pipeline Structure The pipeline has two main stages: stages: - stage: Build displayName: Build and Push Images - stage: DeployDEV displayName: Deploy to DEV dependsOn: Build The relationship is: Build | | success v DeployDEV The deployment cannot start until the build stage has succeeded. 77. Verify the Deployment From the Pipeline After Helm deployment, I verify: kubectl get deployments -n frontend kubectl get deployments -n backend kubectl get services -n frontend kubectl get services -n backend I also use: kubectl get pods -n frontend kubectl get pods -n backend The objective is: Frontend Pod Running Backend Pod Running Frontend Service LoadBalancer Backend Service ClusterIP 78. Useful Kubernetes Debugging Commands When a deployment fails, these are some of the commands I use most often. List Pods kubectl get pods -n frontend kubectl get pods -n backend Check Pod Details kubectl describe pod <pod-name> -n backend This is useful for: Scheduling problems Image pull errors Environment problems Mount problems Events Check Logs kubectl logs <pod-name> -n backend For the previous crashed container: kubectl logs <pod-name> -n backend --previous Check Deployment kubectl describe deployment k8app-backend -n backend Check Services kubectl get services -n backend kubectl describe service k8app-frontend -n frontend Check ConfigMaps kubectl get configmaps -n backend kubectl describe configmap k8app-backend-config -n backend Check Secrets kubectl get secrets -n backend Do not casually print secret values into logs or screenshots. 79. Useful Helm Debugging Commands Render the chart: helm template test-release helm/k8app Render with debug: helm template test-release helm/k8app --debug List releases: helm list -n backend Check release: helm status k8app -n backend Check release history: helm history k8app -n backend Upgrade: helm upgrade k8app helm/k8app \ --namespace backend \ --set backend.image.tag=21 \ --set frontend.image.tag=21 \ --wait \ --timeout 5m 80. Why helm template Is So Useful One of the biggest Helm lessons for me was that Helm is essentially a templating and release-management layer around Kubernetes manifests. For example: image: "{{ .Values.backend.image.repository }}:{{ .Values.backend.image.tag }}" is not valid final Kubernetes configuration until Helm renders it. Running: helm template lets me see what Kubernetes will actually receive. The debugging process becomes: Helm template | v Rendered YAML | v Check values | v Fix template | v Deploy 81. Verify the ACR Images From Azure CLI: az acr repository list \ --name dessydevopsacr \ -o table You should see repositories such as: k8app-backend k8app-frontend List tags for the backend: az acr repository show-tags \ --name dessydevopsacr \ --repository k8app-backend \ -o table Frontend: az acr repository show-tags \ --name dessydevopsacr \ --repository k8app-frontend \ -o table This helps verify that the CI stage actually pushed the images. 82. Verify Which Image Kubernetes Is Running You can inspect the Deployment: kubectl get deployment k8app-backend \ -n backend \ -o jsonpath='{.spec.template.spec.containers[0].image}' And frontend: kubectl get deployment k8app-frontend \ -n frontend \ -o jsonpath='{.spec.template.spec.containers[0].image}' This should show the expected Build ID. For example: dessydevopsacr.azurecr.io/k8app-backend:21 This is useful when troubleshooting: Did the pipeline build the correct image, or is AKS still running an older version? 83. Verify the Application After the deployment succeeds: kubectl get service k8app-frontend -n frontend Get the external IP. Then open the IP in a browser. For this deployment, the frontend became accessible through: http://4.253.76.190 The application loaded successfully. That gave me the final checkpoint: GitHub ↓ Azure DevOps ↓ Build ↓ Docker ↓ ACR ↓ Helm ↓ AKS ↓ Frontend ↓ Running Application 84. The Complete DEV Environment At the end, the important Azure resources were: Resource Group devops-aks-lab-rg-sa ACR dessydevopsacr AKS dessy-aks-cluster PostgreSQL dessy-k8app-postgres PostgreSQL subnet postgres-subnet 10.0.2.0/24 Private DNS private.postgres.database.azure.com The Kubernetes namespaces were: frontend backend The Helm release was: k8app The container images were: dessydevopsacr.azurecr.io/k8app-frontend:<BuildID> dessydevopsacr.azurecr.io/k8app-backend:<BuildID> 85. A Simple Way to Think About the Whole System After building the project, this is the mental model I use: SOURCE CODE | v Azure DevOps | | Build v Docker Image | v ACR | | Pull v AKS | +--------------------+ | | v v Frontend Backend | | LoadBalancer ClusterIP | v PostgreSQL Flexible Server | v Private Network | v Private DNS Each component has one main responsibility. Component Responsibility GitHub Source code Azure DevOps CI/CD orchestration Docker Package applications ACR Store container images Helm Package/deploy Kubernetes resources AKS Run containers Kubernetes Service Provide stable networking ConfigMap Non-secret configuration Secret Sensitive configuration PostgreSQL Flexible Server Managed database Private DNS Resolve private database hostname Azure RBAC Azure authorization Kubernetes RBAC Kubernetes authorization 86. Important Lessons I Learned Lesson 1 — Authentication and authorization are different I initially thought: If Azure DevOps can authenticate to Azure, it should automatically be able to deploy to Kubernetes. That is not necessarily true. We had: OIDC ↓ Azure authentication then: Azure RBAC ↓ Azure permissions and separately: Kubernetes RBAC ↓ Kubernetes permissions Lesson 2 — ACR push and AKS pull are different operations Azure DevOps needs permission to: PUSH images into ACR. AKS needs permission to: PULL images from ACR. Therefore: Azure DevOps | | Push v ACR | | Pull v AKS Lesson 3 — Private networking requires DNS It is not enough to have: Private IP The application normally uses: hostname Therefore: Hostname ↓ Private DNS ↓ Private IP ↓ Database The Private DNS zone was: private.postgres.database.azure.com Lesson 4 — ConfigMap and Secret solve different problems ConfigMap: Normal configuration Secret: Sensitive configuration So: ConfigMap | +-- PROJECT_NAME +-- POSTGRES_SERVER +-- POSTGRES_DB while: Secret | +-- POSTGRES_USER +-- POSTGRES_PASSWORD Lesson 5 — Kubernetes Pods are not permanent endpoints Pods can be recreated. Therefore applications should normally communicate through Services rather than hardcoding Pod IPs. Pod ↓ Service ↓ Stable endpoint Lesson 6 — kubectl describe is extremely useful When something is: Pending or: Failed don't immediately change configuration. First investigate. For example: kubectl describe service k8app-frontend -n frontend revealed: PublicIPCountLimitReached That saved a lot of unnecessary Kubernetes troubleshooting. 87. Troubleshooting Cheat Sheet Pod is CrashLoopBackOff Start with: kubectl logs <pod> -n <namespace> Then: kubectl describe pod <pod> -n <namespace> Look for: missing environment variables bad configuration application startup failure database connection failure Pod is Pending Run: kubectl describe pod <pod> -n <namespace> Look at: Events Possible causes include: insufficient resources scheduling problem node problem ImagePullBackOff Check: kubectl describe pod <pod> -n <namespace> Verify: Image name Image tag ACR repository ACR permissions AKS AcrPull role Service EXTERNAL-IP is Pending Run: kubectl describe service <service> -n <namespace> Check the events. Don't assume the Kubernetes Service itself is broken. The problem could be Azure infrastructure or quota. Helm deployment fails First run: helm template test-release helm/k8app Then: helm upgrade --install ... If necessary: helm template test-release helm/k8app --debug Backend cannot connect to PostgreSQL Check: 1. DNS 2. Private IP 3. Network connectivity 4. PostgreSQL server 5. Database name 6. Username 7. Password 8. TLS configuration For DNS: kubectl run psql-test \ --rm -it \ --image=postgres:16 \ --restart=Never \ -- \ getent hosts dessy-k8app-postgres.postgres.database.azure.com 88. Security Notes There are a few things I would improve before calling this a production-ready implementation. First, never commit: POSTGRES_PASSWORD to Git. Never put it in: values.yaml or: Deployment YAML The real password should also never appear in: screenshots Dev.to article GitHub repository pipeline logs For a production implementation, I would also consider: Azure Key Vault External Secrets Workload Identity More restrictive RBAC Private ACR Private AKS networking Network policies TLS Ingress Health probes Database migrations The current project is a learning environment designed to understand the complete flow. 89. What I Would Improve Next The current DEV environment works, but there are several areas I want to improve. 1. Database migrations The PostgreSQL database currently has no application tables. The next step is to introduce migrations so the deployment process can create/update the schema safely. 2. Better secret management The Azure DevOps variable group works for the learning environment. A more production-oriented design would use: Azure Key Vault | v Workload Identity / Secret integration | v AKS 3. Ingress The frontend currently uses: LoadBalancer A production environment could instead use an ingress architecture: Internet | v WAF / Application Gateway | v Ingress | +---- frontend | +---- backend This also provides a cleaner way to route traffic between frontend and backend. 4. Health probes The Deployments should eventually have: livenessProbe readinessProbe This allows Kubernetes to understand whether an application is: alive and whether it is: ready to receive traffic 5. Pipeline error handling The deployment script should explicitly fail when an important command fails. For example, I would eventually add: set -euo pipefail This prevents a later command from accidentally hiding an earlier deployment failure. 6. Environment-specific Helm values Eventually I want: values-dev.yaml values-uat.yaml values-prod.yaml Then: DEV ↓ UAT ↓ Approval ↓ PROD 90. What Comes Next: UAT and PROD For now, I deliberately stopped at DEV. The next architecture will be: Build | v DEV | v UAT | v Approval Gate | v PROD The idea is that the same application artifact should move through environments rather than rebuilding different artifacts for each environment. That gives us: Build once | v Promote the same version | +---- DEV | +---- UAT | +---- PROD This is the direction I want to take the Azure DevOps pipeline next. 91. Final DEV Checklist Before considering the DEV environment complete, I verify: [✓] Resource Group created [✓] Azure Container Registry created [✓] AKS created [✓] PostgreSQL Flexible Server created [✓] PostgreSQL private subnet created [✓] Private DNS configured [✓] AKS can resolve PostgreSQL hostname [✓] AKS can connect to PostgreSQL [✓] ACR contains frontend image [✓] ACR contains backend image [✓] AKS has AcrPull permission [✓] frontend namespace created [✓] backend namespace created [✓] ConfigMap created [✓] Kubernetes Secret created [✓] Backend Deployment running [✓] Frontend Deployment running [✓] Backend Service is ClusterIP [✓] Frontend Service is LoadBalancer [✓] Helm chart renders successfully [✓] Helm release deployed [✓] Azure DevOps build succeeds [✓] Azure DevOps deploys to AKS [✓] Frontend receives public IP [✓] Application accessible from browser Conclusion This project started as a way for me to practice Azure DevOps and Kubernetes. It became much more useful once I stopped looking at the individual commands and started understanding how the components fit together. The final DEV deployment looks like: GitHub | v Azure DevOps | v Build + Test | v Docker Build | v ACR | v Helm | v AKS / \ / \ v v Frontend Backend | | LoadBalancer ClusterIP | v Private Network | v PostgreSQL Flexible Server The most important thing I learned was that DevOps is not simply about writing YAML. When something fails, I need to understand: What failed? | v Which component owns the failure? | v How does that component communicate with the other components? | v What identity is being used? | v What permission is required? | v What evidence can I collect? | v What should I change? For example: Backend failed | v Check logs | v Missing environment variables | v ConfigMap + Secret | v Deployment references them | v Backend starts And: Frontend LoadBalancer pending | v kubectl describe service | v PublicIPCountLimitReached | v Check Azure resources | v Find unused AKS cluster | v Remove unused resource | v LoadBalancer receives public IP Those troubleshooting experiences were more valuable than simply getting a green pipeline on the first attempt. The final goal is not just: "I know the commands." The goal is: I understand what each component does, why it exists, how the components communicate, how identities and permissions work, and how to troubleshoot the environment when something breaks. That is the foundation I now have for moving from DEV to UAT, approval gates and PROD.