DEV Community

iapilgrim
iapilgrim

Posted on

Building an AKS + ACI Virtual Node Lab (Step-by-Step)

#aks #aci #fargate

In this tutorial, weโ€™ll build a hybrid Kubernetes architecture using:

  • Azure Kubernetes Service (AKS)
  • Azure Container Instances (ACI)
  • AKS Virtual Nodes (serverless burst capacity)

By the end, youโ€™ll have:

  • A working AKS cluster
  • A delegated subnet for ACI
  • Virtual Node enabled
  • A pod successfully running on ACI
  • A troubleshooting checklist for common failures

๐Ÿ— Architecture Overview 

AKS Cluster
โ”‚
โ”œโ”€โ”€ VM Node Pool (steady workloads)
โ”‚   โ””โ”€โ”€ baseline-app
โ”‚
โ””โ”€โ”€ Virtual Node (ACI-backed)
    โ””โ”€โ”€ aci-test
Enter fullscreen mode Exit fullscreen mode

Key behavior:

  • VM-backed pods โ†’ 10.240.x.x
  • ACI-backed pods โ†’ 10.3.x.x (delegated subnet)

That IP difference proves true network separation.

๐Ÿงฐ Prerequisites

  • Azure CLI logged in
  • Subscription with permissions
  • Kubernetes CLI (kubectl)
  • Region that supports ACI

๐Ÿ›  Step 1 โ€” Register Required Providers

This step is commonly missed and causes ProviderFailed.

az provider register --namespace Microsoft.ContainerInstance
az provider register --namespace Microsoft.ContainerService
az provider register --namespace Microsoft.Network
Enter fullscreen mode Exit fullscreen mode

Verify:

az provider show --namespace Microsoft.ContainerInstance --query registrationState -o tsv
Enter fullscreen mode Exit fullscreen mode

Should return:

Registered
Enter fullscreen mode Exit fullscreen mode

๐Ÿ›  Step 2 โ€” Create VNet + Delegated Subnet

ACI requires subnet delegation.

RG=rg-aks-agic-demo

LOCATION=eastus2

az network vnet create \
  -g $RG \
  -n aks-vnet \
  --address-prefix 10.0.0.0/8 \
  --subnet-name aks-subnet \
  --subnet-prefix 10.240.0.0/16

az network vnet subnet create \
  --resource-group $RG \
  --vnet-name $VNET_NAME \
  --name aci-subnet \
  --address-prefix 10.3.0.0/16 \
  --delegations Microsoft.ContainerInstance/containerGroups
Enter fullscreen mode Exit fullscreen mode

๐Ÿ›  Step 3 โ€” Enable Virtual Node Add-On

This is the key step many forget.

az aks enable-addons \
  -g $RG \
  -n aks-aci-cluster \
  --addons virtual-node \
  --subnet-name aci-subnet
Enter fullscreen mode Exit fullscreen mode

Verify:

kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

You should see something like:

kubectl get nodes
NAME                                STATUS   ROLES    AGE     VERSION
aks-nodepool1-38580963-vmss000000   Ready    <none>   4h16m   v1.33.6
aks-nodepool1-38580963-vmss000001   Ready    <none>   4h16m   v1.33.6
virtual-node-aci-linux              Ready    agent    27m     v1.25.0-vk-azure-aci-1.6.2
Enter fullscreen mode Exit fullscreen mode

๐Ÿ›  Step 5 โ€” Deploy a Pod to Virtual Node

Create:

apiVersion: v1
kind: Pod
metadata:
  name: aci-test
spec:
  containers:
  - name: aci-test
    image: mcr.microsoft.com/azuredocs/aci-helloworld
  nodeSelector:
    kubernetes.io/role: agent
  tolerations:
  - key: virtual-kubelet.io/provider
    operator: Exists
Enter fullscreen mode Exit fullscreen mode

Apply:

kubectl apply -f aci-test.yaml
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl get pods -o wide
Enter fullscreen mode Exit fullscreen mode

Expected:

aci-test   1/1 Running   10.3.x.x   virtual-node-aci-linux
Enter fullscreen mode Exit fullscreen mode

Logs:

kubectl logs -f aci-test
Enter fullscreen mode Exit fullscreen mode

๐Ÿงช Validation Checklist

Check Expected Result
kubectl get nodes Shows virtual-node-aci-linux
Pod IP In ACI subnet range
Pod NODE virtual-node-aci-linux
Logs stream Successfully

๐Ÿงฏ Troubleshooting Guide

โŒ Pod Stuck in ProviderFailed

Cause 1: Provider Not Registered

Fix:

az provider register --namespace Microsoft.ContainerInstance
Enter fullscreen mode Exit fullscreen mode

โŒ Pod Stuck in Pending

Cause 2: Missing Toleration

Virtual node is tainted.

Add:

tolerations:
- key: virtual-kubelet.io/provider
  operator: Exists
Enter fullscreen mode Exit fullscreen mode

โŒ Error: Subnet Not Delegated

Check:

az network vnet subnet show \
  -g $RG \
  --vnet-name aks-vnet \
  -n aci-subnet \
  --query delegations
Enter fullscreen mode Exit fullscreen mode

Must show:

Microsoft.ContainerInstance/containerGroups
Enter fullscreen mode Exit fullscreen mode

โŒ Virtual Node Not Appearing

Check addon:

az aks show -g $RG -n aks-aci-cluster --query addonProfiles.virtualNode
Enter fullscreen mode Exit fullscreen mode

If disabled:

az aks enable-addons \
  -g $RG \
  -n aks-aci-cluster \
  --addons virtual-node \
  --subnet-name aci-subnet
Enter fullscreen mode Exit fullscreen mode

โŒ IP Not in ACI Subnet

If pod IP looks like VM subnet:

  • NodeSelector incorrect
  • Toleration missing
  • Pod scheduled to VM pool

Check:

kubectl describe pod aci-test
Enter fullscreen mode Exit fullscreen mode

Look at Node: field.

๐Ÿ’ก When to Use Virtual Nodes

โœ… Burst scaling
โœ… Cost optimization
โœ… Multi-tenant sandbox workloads
โœ… Untrusted job execution
โœ… CI/CD runners

Avoid for:

โŒ DaemonSets
โŒ Privileged containers
โŒ Stateful workloads

๐Ÿ’ฐ Cost Model Insight

  • VM nodes โ†’ Always billed
  • ACI โ†’ Per second billing
  • Ideal for unpredictable workloads

๐Ÿงน Cleanup 

az group delete -n $RG --yes --no-wait
Enter fullscreen mode Exit fullscreen mode

๐Ÿ Final Result

You now have a hybrid AKS architecture:

  • Stable workloads on VM nodes
  • Serverless burst capacity via ACI
  • Isolated networking
  • Production-ready troubleshooting knowledge

Top comments (0)