DEV Community: DevOpsInterview.Cloud

Terraform Mono-Repo Design for 20 AWS Accounts

DevOpsInterview.Cloud — Sun, 01 Nov 2026 09:20:02 +0000

"Your platform team manages twenty AWS accounts: dev, staging, prod and a handful of product-team sandboxes. All of the infrastructure is Terraform. Structure a single repository so that one engineer can safely deploy to one account without touching another. Walk me through the design." The prompt is deliberately open-ended. A strong answer is not about knowing a CLI flag; it shows how you reason about isolation, blast radius and team safety at a scale where a copy-paste mistake reaches production. There is also a tempting first answer, "use workspaces, one per account", and the follow-up questions probe whether you know where that holds and where it breaks: for deployments that need separate credentials and access controls, CLI workspaces alone are unsuitable.

The video version walks the same design if you prefer to watch it:

Why "one folder, twenty workspaces" fails

Be precise about what workspaces share, because the imprecise version ("they share state") is common and wrong. Each CLI workspace does get its own state file. With the S3 backend the default workspace is stored at the configured key, and every other workspace is stored in the same bucket under the env:/ prefix, followed by the workspace name and then the key. A corrupted sandbox state does not corrupt prod.

What every workspace shares is everything around the state: the same backend configuration and bucket, the same provider blocks, the same variable declarations, and whatever credentials that shared configuration resolves to. You can vary values with terraform.workspace conditionals, but those conditionals live in the same root module, so a mistake in the shared parts reaches every environment. HashiCorp's own documentation says it plainly: workspaces within a working directory use the same backend, so they are not a suitable isolation mechanism when deployments need separate credentials and access controls. Add the fact that terraform workspace select is mutable state in the operator's shell, easy to forget in a second terminal during an incident, and you have the real problem.

The design therefore has to deliver three things:

State isolation per account. A locked, corrupted or mid-migration state in account A must not block or affect account B.
Independent authentication per account. Each account's Terraform runs with credentials scoped to that account, obtained by role assumption rather than long-lived keys.
CI that scopes itself. A pull request that touches one account plans and applies against that account, not all twenty.

Repo layout: one root module per account

infra-repo/
├── modules/
│   ├── vpc/
│   ├── eks/
│   └── iam-baseline/
├── accounts/
│   ├── prod-111111111111/
│   │   ├── backend.tf
│   │   ├── providers.tf
│   │   ├── main.tf
│   │   └── terraform.tfvars
│   ├── staging-222222222222/
│   └── sandbox-eng-333333333333/
└── .github/workflows/terraform.yml

Two top-level directories carry the design. modules/ holds reusable child modules: the VPC, the EKS cluster, the IAM baseline every account receives. They declare resources, variables, outputs and their required_providers, but they contain no backend block and no provider configuration blocks; the calling root module supplies those. accounts/ holds one subdirectory per AWS account, named with a human-readable label plus the account ID so nobody has to remember which twelve-digit number is prod. Each account directory is a complete, independent Terraform root module with its own backend, its own provider configuration, its own tfvars and its own calls into the shared modules.

One detail to say out loud, because a common variant of this layout gets it wrong: there is no shared globals/variables.tf. Terraform only loads the .tf files in the root module's own directory, so a variable declared in a sibling directory is never read. Values that genuinely apply to every account belong in a small module that returns them as outputs, or in a common tfvars file that each pipeline passes with -var-file.

Backend and provider per account

# accounts/prod-111111111111/backend.tf
terraform {
  required_version = ">= 1.10, < 2.0"   # S3 native locking arrived in 1.10

  backend "s3" {
    bucket       = "tfstate-prod-111111111111"
    key          = "prod/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
    # assume_role is supplied at init time from one of the two files below
  }
}

# accounts/prod-111111111111/backend.plan.hcl   (read state, hold the lock)
assume_role = { role_arn = "arn:aws:iam::111111111111:role/TerraformPlanRole" }

# accounts/prod-111111111111/backend.apply.hcl  (read and write state)
assume_role = { role_arn = "arn:aws:iam::111111111111:role/TerraformDeployRole" }

# accounts/prod-111111111111/providers.tf
variable "terraform_role_name" {
  type        = string
  description = "TerraformPlanRole for pull-request plans, TerraformDeployRole for applies from main"
}

provider "aws" {
  region              = "us-east-1"
  allowed_account_ids = ["111111111111"]

  assume_role {
    role_arn     = "arn:aws:iam::111111111111:role/${var.terraform_role_name}"
    session_name = "terraform"
  }
}

The credential model is one chain with two rungs, and pull-request plans and applies from main use different roles at every rung. In CI, the workflow authenticates via OpenID Connect to one of two roles in a central tooling account: GitHubActionsPlan for pull-request runs and GitHubActionsApply for applies from main. Both trust policies restrict the token to this repository, and the apply role's additionally to the protected environment or branch its job runs under. Each account then carries a matching pair. TerraformPlanRole trusts the central plan role and has read-only infrastructure access plus read and lock access to state; TerraformDeployRole trusts only the central apply role and can change infrastructure and write state. The backend and the provider each assume the account role for the run, because the backend inherits nothing from the provider: without its own assume_role (an object attribute in the backend, unlike the nested block in the provider, and supplied here through a -backend-config file because backend blocks cannot read variables), terraform init would try to read the state bucket with the caller's ambient credentials, which have no rights in the target account. Humans do not get a third path. Applies are CI-only, and an engineer who needs a local plan uses an IAM Identity Center permission set that each TerraformPlanRole trust policy lists alongside the central plan role, so a laptop can plan but never apply.

The split exists because terraform init and terraform plan execute whatever HCL the pull request contains, including the providers it downloads, so pull-request code inherits the plan job's credentials. Those credentials must be unable to change infrastructure or overwrite state. What a plan still needs, and therefore what pull-request code can do, is read state and hold the lock; if the repository accepts pull requests from people who must not read state, gate the plan job behind an approval as well. Four more points here are interview material.

The state lives inside the account it describes. The prod bucket is owned by the prod account, so IAM policies in the prod account, together with the bucket policy, decide who can read or write the prod state. Nothing outside the account is granted S3 access directly; the central roles only get sts:AssumeRole into each account.
Locking no longer needs DynamoDB. use_lockfile = true uses S3 conditional writes to hold a .tflock object next to the state. Terraform 1.10 introduced it and 1.11 made it generally available; DynamoDB-based locking via dynamodb_table is deprecated. The S3 permissions are small: s3:ListBucket on the bucket, s3:GetObject and s3:PutObject on the state object, and s3:GetObject, s3:PutObject and s3:DeleteObject on the .tflock object. Terraform never deletes the state file, so the plan role gets that whole set minus s3:PutObject on the state object. If you mention DynamoDB locking in an interview, present it as the previous approach.
allowed_account_ids is the wrong-account guard. The provider refuses to run if the credentials resolve to any other account. It costs one line and catches the pasted-from-staging role ARN before a plan runs.
Be honest about what a directory is. The directory is the configuration and state boundary: it makes the target explicit and removes the hidden workspace-selection step. It is not a security boundary. A pull request can contain any HCL its author likes, including a provider block that names another account's role, so the ceiling on what a run can do is set by IAM: which roles the central identity is trusted to assume, and what those roles are allowed to do.

CI: plan and apply only the accounts a PR touched

Running plans against all twenty accounts on every pull request is slow and noisy, and a plan failure in an account the PR never touched is a false red build. Changed-path detection fixes that. A naive version has three bugs, so the excerpt below is longer than the one you would sketch on a whiteboard.

# .github/workflows/terraform.yml (excerpt)
on:
  pull_request:
    paths: ["accounts/**", "modules/**"]

permissions:
  id-token: write   # required for OIDC
  contents: read

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      accounts: ${{ steps.changed.outputs.accounts }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # the default depth of 1 has no base branch to diff against
      - id: changed
        shell: bash
        run: |
          CHANGED=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD")
          if grep -q '^modules/' <<<"$CHANGED"; then
            # a shared module changed: every account that consumes it needs a plan
            ACCOUNTS=$(ls -d accounts/*/ | cut -d/ -f2)
          else
            ACCOUNTS=$({ grep -E '^accounts/[^/]+/' <<<"$CHANGED" || true; } | cut -d/ -f2 | sort -u)
          fi
          # only roots that still exist can be planned; removing an account is a
          # separate teardown workflow, not a directory deletion
          ACCOUNTS=$(for a in $ACCOUNTS; do if [ -d "accounts/$a" ]; then echo "$a"; fi; done)
          JSON=$(printf '%s\n' $ACCOUNTS | jq -R . | jq -sc 'map(select(length > 0))')
          echo "accounts=$JSON" >> "$GITHUB_OUTPUT"

  plan:
    needs: detect-changes
    if: needs.detect-changes.outputs.accounts != '[]'   # an empty matrix fails the job
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false   # one account's failed plan must not cancel the others
      matrix:
        account: ${{ fromJson(needs.detect-changes.outputs.accounts) }}
    env:
      TF_VAR_terraform_role_name: TerraformPlanRole   # read-only chain for PR code
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.16.2   # the default is latest; pin and bump deliberately
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::999999999999:role/GitHubActionsPlan
          aws-region: us-east-1
      - run: terraform init -input=false -backend-config=backend.plan.hcl && terraform plan -input=false
        working-directory: accounts/${{ matrix.account }}

  # Apply lives in a separate workflow triggered on push to main. It runs inside a
  # protected environment, assumes GitHubActionsApply, sets TerraformDeployRole and
  # inits with backend.apply.hcl.

The three fixes:

Shallow checkouts have nothing to diff against. actions/checkout fetches a single commit by default, so a diff against the base branch fails. Fetch the history and diff against the PR's base ref rather than a hardcoded main.
A module change touches no account directory. With relative module sources, editing modules/vpc changes files only under modules/, so naive path filtering plans nothing and the breaking change surfaces on the next unrelated PR to each account. Fan out to every account when modules/ changes, or maintain an explicit module-to-consumers map. Atlantis handles this with per-project when_modified globs that can reference module paths.
An empty matrix is a failed job. GitHub Actions refuses to expand a matrix whose vector has no values, so a PR that only touched a stray file under accounts/ would fail the plan job without the if guard.

Two smaller reproducibility points belong in the same breath: pin the Terraform version in CI, since setup-terraform installs the latest release when none is given, and commit each root's .terraform.lock.hcl with provider version constraints so a plan on Tuesday uses the same provider builds as the apply on Wednesday.

Approval tiers are configuration, not code. Put the apply job behind GitHub Environments: no reviewers for sandbox accounts, required reviewers plus a deployment-branch rule restricting deploys to main for staging and production. Check what your GitHub plan offers for private repositories before promising this in an interview, since required reviewers are not available on every plan. Atlantis covers the same ground with per-project apply_requirements such as approved and mergeable, and it plans only the projects whose files a PR modified, which is the changed-path logic above without the shell script.

Versioning shared modules inside the same repo

A natural follow-up: the modules live in the repo, so how do twenty accounts adopt a module change at twenty different speeds?

# Option A: relative path. Every account tracks the module at the current commit.
module "vpc" {
  source = "../../modules/vpc"
  cidr   = "10.0.0.0/16"
}

# Option B: pinned tag. Each account upgrades when its owners choose to.
module "vpc" {
  source = "git::https://github.com/org/infra-repo.git//modules/vpc?ref=vpc-v1.4.2"
  cidr   = "10.0.0.0/16"
}

Relative paths are simple and refactor-friendly, and they are the right default while the module set is young. Their cost is coupling: a breaking change to the VPC module becomes part of every account's configuration the moment it merges, and the CI fan-out above is what turns that from a surprise into twenty visible plans. Pinned tags decouple accounts from module changes: sandbox can move to the new tag the day it is cut while prod waits. Renovate's Terraform manager understands git-tag module sources and can open the upgrade PRs, with one caveat: all modules in one repo share a single tag namespace, so per-module prefixes such as vpc-v1.4.2 and eks-v2.0.0 need a Renovate rule that maps each module to its own tag pattern, or you version the whole module set together, or you publish modules to a registry. The costs are real too: terraform init now clones the repository once per pinned module instead of reading a local directory, the runner needs Git credentials for a repo it already has checked out, and someone owns tagging and release notes. Pinned tags for prod-tier accounts and relative paths for the sandboxes is a defensible middle ground.

The pushback: "why not just workspaces?"

If this lands after you have explained the layout, answer in terms of failure modes rather than taste. Workspaces share one backend configuration and one set of provider blocks; per-workspace conditionals can vary the values, but they are not a credential or access-control boundary, and a mistake in the shared parts reaches every environment. Workspace selection is mutable state in the operator's shell (or the TF_WORKSPACE variable), which is exactly the kind of thing that goes wrong under pressure. Directories remove that hidden step: the configuration in front of you names the account it targets, and allowed_account_ids plus IAM stop a wrong-account run from succeeding.

Then give workspaces their due. Short-lived, near-identical environments inside a single account, such as a per-branch preview stack torn down when the PR closes, are what workspaces are good at. Account-level isolation across twenty accounts with different credentials is not.

The gotcha: drift across twenty accounts goes unnoticed

With this many accounts, someone will change something in the console. A security engineer tightens a security group during an incident; a developer attaches an IAM policy to unblock a deploy. State and reality now disagree, and Terraform will not tell you until an unrelated PR triggers a plan for that account and the diff includes changes nobody proposed. The CLI does not schedule drift checks for you (HCP Terraform's health assessments do), so with plain Terraform you schedule them.

#!/usr/bin/env bash
# nightly-drift-check.sh: report tracked objects whose live values differ from state
set -u
export TF_VAR_terraform_role_name=TerraformPlanRole   # read-only chain is enough
ROOT=$(pwd)
mkdir -p "$ROOT/drift-logs"
drifted=()
errors=()
for dir in accounts/*/; do
  account=$(basename "$dir")
  (
    cd "$dir" || exit 1
    terraform init -input=false -no-color -backend-config=backend.plan.hcl >/dev/null &&
      terraform plan -refresh-only -detailed-exitcode -input=false -no-color \
        >"$ROOT/drift-logs/$account.log" 2>&1
  )
  case $? in
    0) echo "clean   $account" ;;
    2) echo "DRIFT   $account"; drifted+=("$account") ;;
    *) echo "ERROR   $account (init or plan failed)"; errors+=("$account") ;;
  esac
done
[ ${#drifted[@]} -eq 0 ] || printf 'drifted: %s\n' "${drifted[*]}"
[ ${#errors[@]} -eq 0 ] || { printf 'failed: %s\n' "${errors[*]}"; exit 1; }
[ ${#drifted[@]} -eq 0 ] || exit 2

Three details matter. First, -detailed-exitcode returns 0 for no changes, 1 for an error and 2 for changes present. On a plain terraform plan, exit 2 also fires for configuration that merged but was never applied, which is not drift; -refresh-only restricts the comparison to state versus what the providers currently report. Read the log rather than paging on the exit code alone, since output-only changes and provider normalisation can show up there too. Second, be clear about the limit: a refresh compares objects Terraform already tracks, so a resource somebody created by hand outside Terraform is invisible to it. Third, a failed cd, init or plan must be reported as an error rather than counted as clean, which is what the chaining and the separate error list are for. And because this is a read-only path, it runs on the plan chain; the deploy roles are never involved.

Bootstrapping account twenty-one

The last operational question: how does a brand-new account get its first Terraform run? The state bucket has to exist before terraform init can use it, and the account roles have to exist before anything can assume them, and you want both created by code. There are two workable approaches.

The organisation-level answer: a new member account already contains a role the management account can assume, OrganizationAccountAccessRole for accounts created through AWS Organizations and AWSControlTowerExecution for accounts enrolled through Control Tower. A CloudFormation StackSet with service-managed permissions, or a Control Tower customization, then creates the state bucket and the Terraform role in every new account uniformly. Terraform never has to solve its own chicken-and-egg problem because the bootstrap happens before Terraform arrives.

The Terraform-only answer: a tiny bootstrap root module inside the account directory, run once with the local backend using the organization access role, that creates the bucket and the two roles, and is then migrated into that bucket under its own state key so it never collides with the account root's state.

# accounts/sandbox-data-444444444444/bootstrap/main.tf
# Run once with the local backend. Then replace it with the s3 backend using
# key = "bootstrap/terraform.tfstate" and run:
#   terraform init -migrate-state -backend-config=backend.bootstrap.hcl
# where that file sets assume_role to the same OrganizationAccountAccessRole,
# because the backend authenticates separately from the provider below.
terraform {
  backend "local" {}
}

provider "aws" {
  region = "us-east-1"
  assume_role {
    role_arn = "arn:aws:iam::444444444444:role/OrganizationAccountAccessRole"
  }
}

resource "aws_s3_bucket" "tfstate" {
  bucket = "tfstate-sandbox-data-444444444444"
}

resource "aws_s3_bucket_versioning" "tfstate" {
  bucket = aws_s3_bucket.tfstate.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_public_access_block" "tfstate" {
  bucket                  = aws_s3_bucket.tfstate.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# Plus aws_iam_role and policy resources for TerraformPlanRole and
# TerraformDeployRole, each trusting its central CI role, so the account
# root can run without this access role.

Enable versioning on every state bucket: HashiCorp recommends it because it lets you recover a state object that was deleted or corrupted, although it does not roll back the infrastructure an apply changed. With S3 native locking there is no DynamoDB table to create, which is one fewer thing to end up subtly different between account seventeen and account eighteen. Whichever route you pick, write it down in the repo. The failure mode is not the bootstrap itself but discovering a year later that three accounts were bootstrapped by hand and nobody remembers how.

The answer in one pass

Directories, not workspaces: one root module per account with its own backend, provider configuration and hardcoded role ARN, guarded by allowed_account_ids.
State lives in a bucket inside the account it describes, locked with use_lockfile, and the backend assumes the per-account role itself: the plan role for pull requests, the deploy role for applies.
CI authenticates via OIDC to a central plan or apply role and assumes the matching per-account role, so pull-request code can read infrastructure and state but change neither; it pins Terraform, diffs against the PR base with full history, fans out when modules/ changes, guards against an empty matrix, and gates applies with environment protection or Atlantis apply requirements.
Shared modules are pinned by tag where accounts need to upgrade on their own schedule, with Renovate opening the PRs.
Drift detection is a scheduled plan -refresh-only -detailed-exitcode across every account, alerting on exit 2 and treating errors separately.
New accounts are bootstrapped uniformly, by a StackSet or Control Tower customization, or by a documented local-backend bootstrap root followed by a state migration.

Land that and the natural follow-up is Terragrunt, which exists to remove the repeated backend and provider configuration this design puts in twenty folders and adds dependency ordering between roots; be ready to explain what it buys and what it costs.

Multi-account layout is one of the design questions Terraform interviews go deep on, alongside state internals, module design, drift and day-2 operations. Infrastructure as Code: Terraform & OpenTofu works through all of it with 50 senior-level interview questions, worked answers and the production reasoning behind them. It is one of the five books in the DevOps Interview Mastery bundle, and every purchase includes the free Interview-Day Playbook with a day-of checklist and behavioural answer frameworks.

Prefer to watch or listen? Answers like these are walked through on YouTube and on the podcast, on Spotify and Amazon Music.

Originally published at devopsinterview.cloud.

Locking Down Kubernetes RBAC in a Multi-Tenant Cluster

DevOpsInterview.Cloud — Sat, 31 Oct 2026 09:20:03 +0000

A new product team is onboarding onto your shared cluster. Their developers need to deploy to their own namespace, their CI pipeline needs to roll deployments and run database migrations, and the security auditor wants proof that no tenant identity can touch another team's workloads. Design it. This scenario shows up in interviews for the same reason it shows up in real platform work: it cannot be answered by reciting object names. It tests whether you can turn four RBAC primitives into a working permission model, and whether you know the handful of places where a copy-pasted ClusterRoleBinding hands someone the whole cluster.

The video version walks the same design if you prefer to watch it:

The four objects, and where identities actually come from

RBAC in Kubernetes is four kinds of object. A Role is a list of allowed verbs on resources, scoped to one namespace. A ClusterRole is a cluster-scoped object with the same rule format; its rules can target cluster-scoped resources like nodes and PersistentVolumes as well as namespaced ones, and where those rules apply depends on how it is bound. A RoleBinding grants a Role (or, importantly, a ClusterRole) to subjects, with effect limited to the RoleBinding's namespace; the subjects need not live in that namespace (users and groups are not namespaced at all, and a ServiceAccount subject can be referenced from any namespace). A ClusterRoleBinding grants a ClusterRole to subjects across the entire cluster.

The subjects are users, groups, and ServiceAccounts, and the distinction matters more than most candidates realise. Users and groups are not Kubernetes objects at all: an authenticator maps each presented credential to a username and group memberships, from the subject fields of a client certificate or from the claims in an OIDC token issued by your identity provider. ServiceAccounts are the identities Kubernetes itself creates and manages, and they are what pods and CI pipelines use to talk to the API server.

The pattern most candidates miss: ClusterRole plus RoleBinding

A RoleBinding can reference a ClusterRole, which grants that ClusterRole's permissions only inside the RoleBinding's namespace. That turns ClusterRoles into reusable permission templates. You define a developer role once and stamp it into each team's namespace with a per-namespace binding, so there is no duplicated YAML drifting apart between teams:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: namespace-developer
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "patch", "update"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]

The discipline that goes with the pattern: a ClusterRoleBinding only for identities that genuinely need cluster-wide access, such as cluster operators and a small set of controllers. Binding the built-in cluster-admin ClusterRole cluster-wide bypasses everything else in this post, and every one of those bindings is something you will have to explain to an auditor.

Wiring up the team namespace

For the new team, create their namespace and one RoleBinding in it that points the reusable ClusterRole at their group:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-dev-binding
  namespace: team-payments
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: namespace-developer
subjects:
- kind: Group
  name: payments-devs
  apiGroup: rbac.authorization.k8s.io

The group name matches whatever group claim your identity provider puts in the token. Joiners and leavers are then handled in the IdP, with no Kubernetes changes at all, and the binding itself is a small, reviewable file in Git. Everyone in payments-devs can deploy and read pods in team-payments, and nothing in this binding grants anything anywhere else. RBAC is additive across bindings, though: a second binding somewhere else can widen what the group can do, which is why the audit section below exists.

ServiceAccounts and the default-account trap

Every pod that does not specify a ServiceAccount runs as the default one in its namespace, and by default that account's token is mounted into the pod whether the workload needs API access or not. Under RBAC the default ServiceAccount starts with nothing beyond the basic discovery permissions granted to all authenticated principals, but the moment someone grants it more (usually to unblock one workload in a hurry), every pod in the namespace silently inherits the grant. Two rules prevent that:

Every workload that needs API access gets its own dedicated ServiceAccount.
The default ServiceAccount gets automountServiceAccountToken: false, so pods that never asked for API access do not carry a credential. A pod can still set the same field on its own spec, which takes precedence, so this is a default rather than an enforcement mechanism; enforcement of what pods may do comes from the admission layer covered below.

For the team's CI pipeline, that looks like:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-ci
  namespace: team-payments
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-deployer
  namespace: team-payments
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "patch", "update"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["create", "get", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-ci-binding
  namespace: team-payments
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ci-deployer
subjects:
- kind: ServiceAccount
  name: payments-ci
  namespace: team-payments

The Job rule exists because this pipeline runs its database migrations as Kubernetes Jobs; if yours does not, drop it. Also give the pipeline short-lived credentials, either time-limited TokenRequest tokens (kubectl create token) or a federated external identity, rather than a long-lived ServiceAccount token Secret, which Kubernetes itself discourages.

Assume the CI credential leaks eventually, and be precise about the blast radius. Patching deployments and creating Jobs are both workload creation, which Kubernetes documents as a privilege-escalation path in its own right: the stolen credential can run an arbitrary image, mount any Secret in the namespace, and run pods as any ServiceAccount in the namespace. The credential's direct API permissions stop at team-payments, but what a malicious workload reaches from there depends on the other layers: pod networking, cloud metadata endpoints and workload identity, node hardening, and whatever the namespace itself holds. Containment therefore rests on keeping high-privilege ServiceAccounts and shared secrets out of tenant namespaces, on the network and admission controls covered below, and on admission policies if you need to restrict which images or ServiceAccounts a Job may use.

Auditing a cluster you inherited

The built-in tooling answers most audit questions. kubectl auth can-i checks a single permission, --list dumps everything the current identity can do (with the caveat that the list can be incomplete depending on the authorizers in play, which is worth stating rather than trusting it blindly), and --as impersonates another identity, provided your own identity has impersonation rights:

# Everything the CI ServiceAccount can do in its namespace
$ kubectl auth can-i --list \
  --as=system:serviceaccount:team-payments:payments-ci \
  -n team-payments

# Spot-check cross-namespace access: should come back "no"
$ kubectl auth can-i get pods \
  --as=system:serviceaccount:team-payments:payments-ci \
  -n team-checkout

# Every subject bound to cluster-admin
$ kubectl get clusterrolebindings -o json | jq '.items[]
  | select(.roleRef.name=="cluster-admin")
  | {name: .metadata.name, subjects: .subjects}'

Two caveats before treating those answers as proof. First, --as with a ServiceAccount name does not automatically include the groups that account would carry in real requests, such as system:serviceaccounts and system:serviceaccounts:team-payments; permissions granted to those groups only show up if you add them explicitly with --as-group. Second, a no from can-i proves exactly one verb-resource-namespace combination. Stronger RBAC evidence for an auditor is an enumeration of every binding that references the identity and its groups, with can-i spot checks on top; even that is a point-in-time view of RBAC alone, so it sits alongside the authorizer configuration, the real group claims coming from the IdP, and the audit log rather than standing in for them.

The cluster-admin query is usually the fastest way to find trouble in an inherited cluster. Expect the built-in binding for the system:masters group plus a small, explainable set of operator or break-glass identities; a long list of humans and pipeline accounts is a finding. For a fuller picture, rakkess renders an access matrix per subject, and rbac-tool (from Alcide, now part of Rapid7) can visualise bindings and flag risky grants across the cluster.

The bind and escalate verbs

Kubernetes has escalation-prevention rules built into the RBAC API: you can only create or update a role binding if you already hold every permission in the referenced role at the binding's scope, or if you hold the bind verb on that role. Likewise, creating or updating a role that contains permissions you do not yourself hold requires the escalate verb. Both verbs target the roles and clusterroles resources, and both are deliberate bypasses of the safety rails.

That makes them a quiet escalation path when combined with write access to the corresponding objects. A subject who can create ClusterRoleBindings and holds unrestricted bind on ClusterRoles can grant cluster-admin cluster-wide to any identity, including their own; the same combination with namespaced RoleBindings hands over everything in that namespace. A subject with escalate plus write access on roles can author permissions they were never granted. There is a legitimate use: bind constrained with resourceNames to a specific, limited role is how you delegate that one role safely. Unrestricted bind and escalate, and wildcards generally, are the red flags, so scan for them and then read what the scan finds instead of treating every hit as an incident:

# ClusterRoles with wildcard resources
$ kubectl get clusterroles -o json | jq '.items[]
  | select(.rules[]?.resources[]? == "*")
  | .metadata.name'

# ClusterRoles granting bind or escalate
$ kubectl get clusterroles -o json | jq '.items[]
  | select(.rules[]?.verbs[]? == "bind" or .rules[]?.verbs[]? == "escalate")
  | .metadata.name'

These queries cover ClusterRoles only, and the same problems hide in namespaced Roles, in wildcard verbs and apiGroups, and in non-resource URLs, so a real audit repeats the pattern across those too.

Aggregation is the other way permissions grow without anyone editing a role. The rules of the built-in admin, edit and view ClusterRoles are aggregation-controlled: a controller populates them from every ClusterRole carrying a label like rbac.authorization.k8s.io/aggregate-to-edit: "true". Operators and CRD-based add-ons routinely ship ClusterRoles with those labels, which means installing one can silently expand what edit grants in every namespace that binds it. After installing an operator, diff the contents of the aggregated roles against what they held before.

Where RBAC stops

RBAC governs the Kubernetes API server and nothing else. It does not restrict traffic between pods, it does not stop a privileged container from reaching the node, and it has no opinion about what runs inside a container. For this scenario's threat model, tenant isolation rests on three layers: RBAC for API access, NetworkPolicies for pod-to-pod traffic, and Pod Security Admission (or a policy engine like OPA Gatekeeper or Kyverno) to stop privileged pods from being created in the first place. Stricter tenancy requirements add more: resource quotas, storage isolation, node isolation or sandboxed runtimes, and at the far end dedicated clusters per tenant. The mistake to avoid is treating RBAC as the whole security story and skipping the rest.

Making least privilege the default

Three habits turn all of the above from a one-off cleanup into a steady state:

Bake it into provisioning. Namespace creation should apply the default-ServiceAccount lockdown automatically, so new workloads start safe and opt in to API access:

$ kubectl patch serviceaccount default \
  -n team-payments \
  -p '{"automountServiceAccountToken": false}'

Keep RBAC in Git. Applying bindings through a reviewed pipeline means every permission change has a commit and an approver behind it. Be careful not to oversell this in an interview: a GitOps reconciler manages the resources it is told about, and an out-of-band ClusterRoleBinding that was never in Git will not be automatically reverted, and may not even be reported, unless you have configured drift or orphan detection to cover it. Git gives you provenance; catching out-of-band changes is the next habit's job.
Alert on ClusterRoleBinding writes. Watch updates and patches as well as creations, since adding a subject to an existing binding grants just as much as creating a new one. This requires API server auditing to be configured with a policy that captures RBAC writes; managed providers can expose the audit stream through their logging services (often behind an explicit control-plane logging toggle), and self-managed API servers log nothing until you set a policy. Once the events flow, tools such as Falco can consume them via the Kubernetes audit event source; note that Falco's shipped rule set covers specific cases like bindings to cluster-admin, so alerting on all ClusterRoleBinding writes means a small custom rule. These writes should be rare, deliberate and reviewed; an unexpected one is either a mistake or an attack, and both deserve a page.

The model to present

ClusterRoles as reusable permission templates, granted per namespace via RoleBindings.
Subjects are IdP groups for humans, dedicated ServiceAccounts for workloads; joiners and leavers are an IdP change, not a cluster change.
automountServiceAccountToken: false on default ServiceAccounts; every API-consuming workload gets its own account with minimal verbs.
Audit by enumerating bindings per subject, with kubectl auth can-i --list and impersonation (including --as-group) as spot checks; enumerate cluster-admin bindings and hunt for unrestricted bind, escalate and wildcards in both ClusterRoles and Roles.
Watch aggregated ClusterRoles after installing operators.
RBAC plus NetworkPolicy plus Pod Security Admission, since each layer covers what the others cannot. And because workload creation is itself an escalation path, keep high-privilege ServiceAccounts and shared secrets out of tenant namespaces.

Walking those six points through the multi-tenant scenario covers the design question, the audit question and the trade-off question in one pass, and it names the residual risks explicitly.

RBAC is one chapter of the Kubernetes story real interviews go deep on, alongside networking, scheduling, debugging and scaling. Containers & Orchestration covers that full arc with 52 worked questions, follow-up chains and production war stories. It is one of the five books in the DevOps Interview Mastery bundle, and every purchase includes the free Interview-Day Playbook with a day-of checklist and behavioural answer frameworks.

Prefer to watch or listen? Answers like these are walked through on YouTube and on the podcast, on Spotify and Amazon Music.

Originally published at devopsinterview.cloud.

Debugging a Kubernetes Production Incident, Step by Step

DevOpsInterview.Cloud — Fri, 30 Oct 2026 09:20:03 +0000

It is 3AM. PagerDuty goes off. The payment service is down in production, and the first thing you type into the terminal says more about your seniority than anything on your CV. "Walk me through how you would debug it" is one of the most common senior DevOps and SRE interview questions precisely because it cannot be memorised: interviewers are listening for a repeatable mental model, not a list of commands.

This post walks one realistic incident end to end, the way you would narrate it in an interview or actually run it during an outage. If you prefer watching it as a live terminal walkthrough, the video version covers the same incident:

The incident

Your team deployed a new version of the payment service 45 minutes ago. Three replicas, but only one pod is serving traffic. Users are hitting 503s. You have kubectl access to the prod cluster, no dashboards pre-loaded, and no runbook for this specific failure. Go.

Step 1: start wide, not deep

The first rule of production debugging: start at the cluster level, not the pod level. Engineers who jump straight into the logs of the first broken pod they see routinely spend 20 minutes on the wrong pod.

$ kubectl get pods -n prod
NAME                       READY   STATUS             RESTARTS   AGE
payment-7d9f8b6c4-xk2lp    1/1     Running            0          44m
payment-7d9f8b6c4-r8tnv    0/1     CrashLoopBackOff   7          44m
payment-7d9f8b6c4-9wqmz    0/1     CrashLoopBackOff   3          44m

This one view already tells a story. The restart counts mean the containers are coming up and dying repeatedly, which is a different failure from a pod that never scheduled at all. And each status points down a different road: ImagePullBackOff means the image tag does not exist or registry credentials are wrong; Pending means scheduling or resource starvation; CrashLoopBackOff means the container starts and then exits. Know which road you are on before you start walking.

Step 2: read the exit code before reading a single log line

kubectl describe pod is where the real story lives, and most people scroll straight past the most valuable field in its output: Last State. It shows what happened on the previous run, before the restart.

$ kubectl describe pod payment-7d9f8b6c4-r8tnv -n prod
...
Last State:     Terminated
  Reason:       Error
  Exit Code:    1
  Finished:     Mon, 03 Aug 2026 03:12:46 +0000
Restart Count:  7
Events:
  Warning  BackOff   2m (x6 over 10m)   kubelet   Back-off restarting failed container

Exit code 1 is a generic application failure: the process itself decided to exit. Exit code 137 means the process was killed with SIGKILL, and the usual suspects are the OOM killer or a forced termination after the grace period expired; it is the Reason: OOMKilled field and the accompanying events, not the number alone, that confirm memory. The distinction matters because the two problems in our incident need completely different fixes, and the exit code narrows the search dramatically before you have read a single log line.

Step 3: pull logs from the container that already died

Here is the first trap. If you run kubectl logs against a crash-looping pod, you often get the logs of the freshly restarted container, which has not failed yet, so the output looks clean. You need the previous, dead container:

$ kubectl logs payment-7d9f8b6c4-r8tnv -n prod --previous | grep -iE "error|fatal|panic"
ERROR failed to acquire db connection: dial tcp 10.0.1.55:5432: connect: connection refused
ERROR max retries exceeded, shutting down
panic: runtime error: invalid memory address or nil pointer dereference
main.initDBPool(0xc0001a4000)

That is the smoking gun for the crashing pod: the new release changed the database connection pool code and it cannot reach the database it was configured for. Two other places people forget to look:

Init containers. They run before the app container and can fail quietly. kubectl logs <pod> -c <init-container-name> shows them.
All replicas at once. With ten replicas you rarely know which pod to watch. stern streams colour-coded logs from every pod matching a selector (stern payment -n prod), so the misbehaving replica identifies itself.

Step 4: resource forensics for the OOMKilled pod

The second broken pod is dying with Reason: OOMKilled and exit code 137. The decisive evidence is in kubectl describe: the termination reason next to the limits block. kubectl top pod -n prod adds corroboration from live usage, with two caveats worth saying out loud in an interview: it needs the Metrics Server installed, and its readings lag by up to a minute, so a brief startup spike (exactly our failure mode) may never appear in it.

$ kubectl describe pod payment-7d9f8b6c4-9wqmz -n prod | grep -A4 Limits
    Limits:
      memory:  256Mi
    Requests:
      memory:  128Mi

The new release loads a larger in-memory cache on startup, crosses 256Mi, and the kernel kills it. The triage question interviewers want you to ask out loud: is this a legitimate new requirement (raise the limit) or a leak introduced by this build (roll back)? Raising limits blindly is the classic wrong answer; it masks leaks until the node itself runs out of allocatable memory, at which point the blast radius is every pod on that node instead of one deployment.

Step 5: confirm the timeline, then stop the bleeding

Before acting, verify that the deploy actually caused this:

$ kubectl get events -n prod --sort-by=.metadata.creationTimestamp
$ kubectl rollout history deployment/payment -n prod
REVISION  CHANGE-CAUSE
7         <none>
8         <none>

A detail that reads as real experience: CHANGE-CAUSE is empty unless your deploy tooling sets the kubernetes.io/change-cause annotation, so in most clusters you correlate revisions with images via kubectl rollout history deployment/payment --revision=8 instead. Either way, revision 8 went out 44 minutes ago, exactly when the failures started. kubectl rollout undo deployment/payment -n prod starts the rollback to the previous revision; watch it complete with kubectl rollout status, since the actual recovery time depends on image pulls and readiness probes. The framing matters in interviews: rollback is a deliberate decision to restore service while the fix lands, not a panic button. Say it that way.

The gotcha that quietly eats debugging time: stale ConfigMap values

Environment variables injected from ConfigMaps or Secrets are baked in at pod startup. Updating the ConfigMap does nothing to running pods; they keep the old values until they restart. The failure mode is maddening because every individual check looks correct:

$ kubectl get configmap payment-config -n prod -o jsonpath='{.data.DB_HOST}'
10.0.1.99        # the new, correct value

$ kubectl exec payment-7d9f8b6c4-xk2lp -n prod -- env | grep DB_HOST
DB_HOST=10.0.1.55   # what the running pod actually has

Checking what the running process actually sees, rather than what the manifest says it should see, resolves an outsized share of "but the config is right" incidents. (Volume-mounted ConfigMaps normally do update in place after a delay, with one exception: subPath mounts never receive updates. And env vars never do, while apps rarely re-read files without a restart anyway.)

Ephemeral containers: debugging images that have no shell

Production images are increasingly distroless or heavily slimmed: no shell, no curl, nothing to kubectl exec into. The old workaround was rebuilding the image with debug tools and redeploying, and hoping the bug still reproduced. Ephemeral containers (stable since Kubernetes 1.25) end that:

$ kubectl debug -it payment-7d9f8b6c4-xk2lp -n prod --image=busybox --target=payment
/ # wget -qO- localhost:8080/health
{"status":"ok","db":"disconnected","version":"1.8.0"}

kubectl debug injects a temporary busybox container into the running pod. It always shares the pod's network namespace, so local ports and connectivity are testable immediately; seeing the target's processes via --target additionally requires process namespace sharing support from the container runtime, which is the norm on current containerd but worth stating as an assumption. Either way, you get real tools inside a shell-less pod without touching the application container or rebuilding the image.

The playbook

Cluster-wide view first: kubectl get pods and read statuses plus restart counts.
Exit codes and termination reasons from describe choose the debugging path: 1 is an app-level failure, 137 is SIGKILL, and Reason: OOMKilled confirms memory.
kubectl logs --previous for containers that already died; remember init containers.
Compare limits against live usage before deciding raise-versus-rollback.
Verify the timeline with events and rollout history, then roll back deliberately.
Check what running pods actually have in their environment, not what the ConfigMap says.
kubectl debug with ephemeral containers when there is no shell to exec into.

Interviewers are not scoring the commands. They are scoring whether a structured model shows up under pressure, because that is exactly what 3AM tests.

If you want this depth across the whole container stack, with 52 worked Kubernetes questions, follow-up chains and production war stories, that is what Containers & Orchestration covers. It is one of the five books in the DevOps Interview Mastery bundle, and every purchase includes the free Interview-Day Playbook with a day-of checklist and behavioural answer frameworks.

Prefer to watch or listen? Answers like these are walked through on YouTube and on the podcast, on Spotify and Amazon Music.

Originally published at devopsinterview.cloud.

How to Shrink a 2GB Docker Image with Multi-Stage Builds

DevOpsInterview.Cloud — Thu, 29 Oct 2026 09:20:03 +0000

"Your image is 2GB. Production pulls are timing out, registry costs are climbing, and security keeps flagging the attack surface. Fix it." This scenario shows up constantly in DevOps interviews because it tests three things at once: whether you understand what is actually inside an image, whether you can use multi-stage builds correctly (most candidates get the dependency handling subtly wrong), and whether you know the real trade-offs between base images instead of reciting "use Alpine".

The video version walks the same optimization live if you prefer to watch it:

Why images balloon to 2GB in the first place

The root cause is treating a container like a virtual machine. A typical bloated Node.js image looks like this: a full Debian-based node base image (over 1GB by itself), node_modules installed with dev dependencies included (webpack, jest, eslint and friends shipping to production), the entire source tree copied in including .git, and often a compiler toolchain someone added to make one native module build. The application code is 10MB; everything else is baggage.

The baggage is not just size. Every package you ship is attack surface, every CVE in the toolchain is a finding in your security scan, and every node that does not already have the layers cached pulls all of it across the network.

Multi-stage builds, done correctly

The idea: use one stage with the full toolchain to build, then copy only the finished artifacts into a minimal runtime stage. The build stage is discarded; only the final stage ships.

The classic mistake is dependency handling. You cannot npm ci --omit=dev in the build stage, because building usually needs the dev dependencies (TypeScript, bundlers). And you should not ship the build stage's node_modules, because it contains them. The clean pattern uses three stages:

# syntax=docker/dockerfile:1

# Stage 1: build with the full toolchain
FROM node:22-alpine AS builder
WORKDIR /app
# Native modules? Alpine's node image ships no compiler toolchain. Any stage
# that runs npm ci over native deps (this one AND deps below) needs, BEFORE it:
# RUN apk add --no-cache python3 make g++
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Stage 2: production dependencies only
FROM node:22-alpine AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev

# Stage 3: minimal runtime
FROM node:22-alpine AS runtime
WORKDIR /app
RUN addgroup -S app && adduser -S app -G app
COPY --from=deps /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
COPY package.json ./
USER app
EXPOSE 3000
CMD ["node", "dist/server.js"]

Details interviewers probe on:

npm ci, not npm install. It installs exactly what the lockfile says, fails loudly on drift, and is faster in CI. npm install can silently mutate the lockfile.
COPY package*.json ./ before COPY . . so the dependency layer is cached and only rebuilds when the lockfile changes, not on every source edit.
A non-root user. Root containers are among the first things any security review flags; creating a dedicated user costs three lines.
Pinned base versions. node:22-alpine beats node:latest, but be precise in interviews: a version tag is still mutable and will pick up new patch builds. Only pinning by digest (node@sha256:...) makes the base truly immutable.

Two supporting pieces do a surprising amount of work. A .dockerignore excluding node_modules, .git, tests and docs keeps the build context small and stops COPY . . from dragging junk into layers. And docker history <image> (or the excellent dive tool) shows you exactly which instruction created which megabytes, which is how you find the accidental 400MB layer.

The honest numbers

This is where many tutorials oversell, and where being precise wins interviews. First, define the measurement: the sizes below are uncompressed, what docker images shows locally. Registries store compressed layers, so Docker Hub numbers look roughly two to three times smaller for the same image; comparing one article's compressed number against another's uncompressed one is how "50MB Node images" get invented. For our Node.js example:

Approach	Realistic size (uncompressed)
Full `node:22` base, dev deps, source tree	~2GB
Multi-stage with `node:22-alpine` runtime	150 to 250MB
Multi-stage with distroless Node runtime	similar, slightly leaner surface
Go or Rust static binary on `scratch`/distroless	10 to 50MB

A Node.js image bottoms out around 150MB because the runtime itself weighs that much before your code arrives; a minimal Python Alpine image starts smaller but still lands well above the static-binary class once real dependencies are installed. The 90 percent-plus reductions are real. When a team genuinely needs the sub-50MB class, the answer is a compiled language producing a static binary, where the final stage can be FROM scratch plus CA certificates.

Alpine vs distroless vs scratch

Alpine (~5MB base): has a shell and the apk package manager, so you can still debug interactively. The trade-off people forget: it uses musl libc instead of glibc, which occasionally breaks native modules and prebuilt binaries. If your app depends on native code, test on Alpine before committing to it.
Distroless (Google's images): your language runtime, CA certificates, tzdata, and nothing else. No shell, no package manager, dramatically reduced attack surface, and noticeably quieter CVE scans. The debugging story needs a plan: use kubectl debug with ephemeral containers, or the :debug image variants which include busybox.
Scratch: literally empty. Only works for static binaries, and you must copy in CA certificates yourself if the app makes TLS calls. The leanest and most secure option when it fits.

A defensible production default: Alpine for interpreted languages where the debugging ergonomics matter, distroless where security posture is the priority, scratch for static binaries.

What this buys you beyond disk

The size reduction is the headline, but the second-order effects are what you should lead with in an interview answer: cold pulls on fresh nodes drop from minutes to seconds, which is exactly when it matters (autoscaling events, node replacement, disaster recovery). CVE scan reports shrink from hundreds of findings to a handful, because the vulnerable packages simply are not there. And registry storage and egress costs fall roughly in proportion to the image, across every environment that stores a copy.

Recap

Multi-stage builds: fat builder stage, separate production-deps stage, minimal runtime stage.
npm ci with a lockfile, dependency layers before source layers, .dockerignore always.
Alpine, distroless, or scratch, chosen by debugging needs, security posture and whether the binary is static.
Non-root user and pinned base versions, every time.
Quote honest numbers, and say whether they are compressed or uncompressed: 90 percent-plus reductions are achievable, but a Node runtime image bottoms out around 150MB uncompressed.

Container questions in real interviews go several follow-ups deeper than any one article: image internals, layer caching strategy, registry architecture, then straight into Kubernetes. The DevOps Interview Mastery bundle picks up from there: five books and 252 worked questions across Kubernetes, cloud, Terraform, CI/CD and SRE, with follow-up chains and production war stories. Every purchase includes the free Interview-Day Playbook with a day-of checklist and behavioural answer frameworks.

Prefer to watch or listen? Answers like these are walked through on YouTube and on the podcast, on Spotify and Amazon Music.

Originally published at devopsinterview.cloud.

20 Terraform Interview Questions from Real Senior DevOps Interviews

DevOpsInterview.Cloud — Wed, 28 Oct 2026 09:20:03 +0000

Terraform interviews have a distinctive shape: almost every hard question is secretly a question about state. Providers, modules and HCL syntax are easy to learn on the job; what interviewers probe is whether you understand what Terraform believes about the world, where that belief lives, and what happens when it diverges from reality. Here are the questions that come up in real senior screens, with the answers that survive follow-ups.

State: where every interview goes first

1. What is the state file, and why does Terraform need one at all?

State maps your configuration's resource addresses to real provider object IDs, and caches attributes. Without it, Terraform could not know that aws_instance.web is instance i-0abc123, so it could not plan updates or deletes. The follow-up is usually "couldn't Terraform just query the cloud instead?" Partially, and that is what import and refresh do, but tags and names are not reliable identity, and many resources have no queryable natural key. State is the identity layer.

2. Your team of six runs Terraform from laptops with local state. What goes wrong and how do you fix it?

Everything: divergent copies of the truth, overwritten changes, secrets in state sitting unencrypted on six laptops, no locking, no history. The fix is a remote backend (S3 with DynamoDB locking, or Azure Storage with blob leases, or Terraform Cloud/HCP), encryption at rest, and CI as the only place apply runs. Mentioning "humans get plan, only the pipeline gets apply" is the sentence that marks you senior.

3. What does state locking protect against, and what does it NOT protect against?

It prevents two concurrent operations from mutating the same state and corrupting it. It does not prevent logical conflicts: two workspaces or two stacks managing the same real resource will still fight each other, lock or no lock, because locking is per state file, not per cloud resource.

4. Someone deleted a resource in the console. What do plan and apply do now?

Terraform refreshes state, notices the object is gone, and plans to recreate it (configuration still declares it). The reverse case is the better follow-up: someone changed an attribute in the console. Terraform plans to revert it back to the configuration. If the console change was correct, you update the configuration to match, and this is the moment to say the word drift and describe running scheduled terraform plan -detailed-exitcode in CI to detect it before it surprises anyone.

5. How do you remove a resource from Terraform management without destroying it?

terraform state rm for the surgical version, or a removed block (Terraform 1.7+/OpenTofu) for the declarative, code-reviewed version. Bonus points for knowing the reverse direction too: import blocks (1.5+) let you adopt existing infrastructure with a plannable, reviewable diff, replacing the old imperative terraform import one-liner.

6. What is in the state file that makes it sensitive?

Every attribute of every resource, which includes database passwords, private keys and tokens that providers return. Treat state like a secrets store: encrypt it, restrict who can read the backend, and never commit it. If asked how to reduce the exposure: ephemeral resources and write-only attributes (recent Terraform/OpenTofu releases) keep certain secrets out of state entirely.

Modules, structure and environments

7. How do you structure Terraform for dev, staging and prod?

There are two defensible answers and interviewers accept either if you defend it: separate root modules per environment composing shared child modules (maximum isolation, some duplication), or one root module with per-environment tfvars and separate state (less duplication, riskier blast radius). Workspaces alone as the environment mechanism is the answer that gets challenged: workspaces share the same backend and code path, which makes prod isolation weak.

8. What makes a good module, and what is over-modularisation?

A good module wraps a real architectural decision (your org's notion of "a service", "a VPC") with a small, opinionated variable surface. Over-modularisation is wrapping single resources in modules that add nothing but indirection, forcing every change through two files and a version bump. If you have inherited a repo with 40 one-resource modules, say so; the scar tissue is convincing.

9. How do you share values between two Terraform stacks?

terraform_remote_state data source reads another stack's outputs (tight coupling, read access to their entire state), or the publishing stack writes to SSM Parameter Store/Secrets Manager and consumers read via data sources (looser coupling, better access control). Naming the trade-off is the point of the question.

10. count vs for_each: when does count hurt you?

count identifies instances by index, so removing one element from the middle renumbers everything after it, and Terraform plans to destroy and recreate resources that did not logically change. for_each identifies by key, so removals are surgical. Rule of thumb worth saying out loud: count for "N identical things", for_each for "these named things".

Execution, CI/CD and day-2 operations

11. Walk me through what terraform plan actually does.

Reads configuration, refreshes state against real provider APIs, builds a dependency graph, and computes the diff between desired and actual, producing a plan file. Two details that impress: the plan can be saved and applied verbatim later (plan -out then apply plan.tfpl), which is how CI guarantees what was reviewed is what runs; and refresh is why plan needs credentials even when nothing changed.

12. How should Terraform run in CI/CD?

Plan on pull request with the plan output posted for review; apply only on merge, from a locked-down runner with short-lived cloud credentials (OIDC federation, not static keys); state in a remote backend the runner alone can write. Add policy checks (OPA, Sentinel, or checkov/tfsec) between plan and apply. If you mention posting the plan as a PR comment so reviewers see the blast radius, you are describing a real pipeline, and it shows.

13. What does -target do and why is it dangerous as a habit?

It narrows the operation to selected resources and their dependencies. As a one-off escape hatch during an incident it is fine; as a habit it produces partial applies, so state stops reflecting the full configuration and the next full plan becomes a minefield of surprise changes.

14. A plan shows a destroy/recreate on a production database. What are your options?

First, understand why: some attribute changes force replacement. Then, in preference order: avoid the change if it is cosmetic; check whether the provider supports in-place update via a different attribute; use lifecycle { prevent_destroy = true } as the safety net that fails the plan; and for renames or refactors, moved blocks tell Terraform the resource is the same object at a new address so no replacement happens at all. Knowing moved blocks is the difference between a rename and an outage.

15. How do you upgrade a provider major version safely?

Pin versions explicitly, read the upgrade guide's breaking changes, upgrade in a non-prod stack first, run plan and inspect for unexpected diffs, then roll forward stack by stack. The keyword the interviewer is waiting for is the lock file (.terraform.lock.hcl), committed, so every machine and runner resolves identical provider builds.

The 2026 questions

16. Terraform vs OpenTofu: what actually changed and what would you pick?

Terraform moved to the BUSL licence in 2023; OpenTofu is the Linux Foundation fork that stayed MPL open source and now ships its own features (state encryption arrived there first). For most internal use the licence change has no practical effect, and the tools remain drop-in compatible for mainstream workflows. A grown-up answer names the real decision inputs: legal comfort with BUSL, appetite for community vs vendor governance, and which ecosystem your platform tooling targets.

17. Terraform vs Ansible: why both?

Different jobs: Terraform declares infrastructure with a lifecycle (create, update, destroy, tracked in state); Ansible executes procedural configuration on existing machines without a state model. The classic pairing is Terraform to create the instances and Ansible (or cloud-init, or baked images) to configure them. Saying "we replaced Ansible with immutable images and user data" is also a fine answer if you can defend it.

18. How do you test Terraform?

Layered: fmt and validate as table stakes; static analysis (tflint, checkov) for policy; terraform test (native since 1.6) for module contract tests; and ephemeral-environment integration tests for the expensive truths. Honest seniors also say: most real confidence comes from small blast radii and reviewable plans, not test suites.

19. What are data sources and when do they beat hardcoding?

Data sources query existing infrastructure at plan time: the current AMI, the VPC built by another team, an IP range. They beat hardcoding whenever the value has an authoritative source that can change; they lose when they create hidden cross-team coupling that a pinned, versioned input would make explicit.

20. Describe a Terraform incident you caused or inherited.

Every senior has one: the state file someone deleted, the -target habit that hid a landmine, the for_each refactor that planned 200 replacements, the provider upgrade that changed defaults. Interviewers ask because the answer cannot be memorised. Prepare yours as a story: context, mistake, blast radius, fix, and the guardrail you added afterwards.

Preparing properly

If these questions felt comfortable, you are interview-ready on Terraform basics and then some. If several answers surprised you, the gap is usually state internals and day-2 operations, which is exactly where interviews concentrate. Infrastructure as Code: Terraform & OpenTofu works through all of this with 50 senior-level interview questions, worked answers and the production reasoning behind them, and it ships as part of the DevOps Interview Mastery bundle alongside books on Kubernetes, cloud platforms, CI/CD and SRE.

Prefer to watch or listen? Answers like these are walked through on YouTube and on the podcast, on Spotify and Amazon Music.

Originally published at devopsinterview.cloud.

25 Kubernetes Interview Questions You Will Actually Be Asked in 2026

DevOpsInterview.Cloud — Tue, 27 Oct 2026 09:20:03 +0000

Most Kubernetes interview lists are padded with definitions nobody asks anymore. Nobody senior is going to ask you "what is a pod" and stop there. What they will do is start with a simple question and drill into it until they find the edge of your knowledge. This list is organised the way real interviews go: each section starts with the opening question, then shows the follow-ups that separate a memorised answer from real experience.

Fundamentals (asked at every level, drilled hard at senior level)

1. Why does Kubernetes use pods instead of scheduling containers directly?

Because some containers need to share a network namespace and volumes as a single unit. A pod gives its containers one IP, one localhost, and shared storage, which is what makes sidecar patterns (log shippers, service mesh proxies, init containers) possible. The follow-up is usually "give me a real case where you needed two containers in one pod." Have one ready.

2. A pod is stuck in Pending. Walk me through your debugging.

The answer they want is a sequence, not a guess:

kubectl describe pod <name>   # read Events first, always

Pending almost always means the scheduler cannot place it: insufficient CPU or memory on any node, a nodeSelector or affinity rule nothing satisfies, a taint without a matching toleration, or an unbound PersistentVolumeClaim. If the events show FailedScheduling, read the reason, it names the constraint. Saying "I would check the events before anything else" signals experience by itself.

3. What actually happens when you run kubectl apply?

kubectl sends the manifest to the API server, which validates it, runs admission controllers, and persists the object to etcd. Nothing is "created" yet. Controllers watching that resource type notice the new desired state and work to realise it: the Deployment controller creates a ReplicaSet, the ReplicaSet controller creates pods, the scheduler binds each pod to a node, and the kubelet on that node pulls images and starts containers. Interviewers love this question because it tests whether you understand Kubernetes as a set of reconciliation loops rather than a command executor.

4. Deployment vs StatefulSet vs DaemonSet. When is each one wrong?

A Deployment is wrong when pods need stable identities or ordered startup (databases, Kafka brokers). A StatefulSet is wrong for stateless services because you pay for ordering guarantees you do not need, and rolling updates are slower. A DaemonSet is wrong for anything that should not scale with the node count. The "when is it wrong" framing is increasingly common because it cannot be answered from a definition.

5. What is the difference between requests and limits, and what happens when each is exceeded?

Requests are what the scheduler reserves; limits are what the runtime enforces. Exceed a CPU limit and you get throttled. Exceed a memory limit and the kernel OOM-kills the container. The senior follow-up: "why might you set memory request equal to limit but leave CPU limit unset?" Because memory is incompressible (an OOM kill is data loss) while CPU throttling is survivable, many production teams set CPU requests only. Knowing this debate exists puts you ahead of most candidates.

Networking and services

6. How does a ClusterIP service actually route traffic?

There is no proxy process sitting in the data path. kube-proxy programs iptables or IPVS rules on every node that rewrite the service's virtual IP to a randomly selected backend pod IP. The EndpointSlice controller keeps the backend list in sync with ready pods. If you can say "the ClusterIP never appears on the wire, it is rewritten before the packet leaves the node," you have answered at a senior level.

7. A service returns connection refused intermittently. Where do you look?

Check whether the endpoints list matches your expectation first: kubectl get endpointslices for the service. Intermittent failures usually mean some backends are unhealthy but still listed (readiness probe missing or wrong), a pod is being terminated while still receiving traffic (no preStop hook or too short a grace period), or the targetPort does not match the container port on some pods after a partial rollout.

8. Why do readiness and liveness probes exist separately, and how do you misuse them?

Readiness controls traffic (a failing pod is removed from endpoints); liveness controls restarts. The classic misuse is pointing a liveness probe at a dependency check: when the database blips, every pod fails liveness simultaneously and Kubernetes restarts your entire fleet, turning a dependency slowdown into a full outage. Liveness should check "is this process wedged," nothing more. Interviewers ask this because the failure mode is so common in production.

9. Explain NetworkPolicy default behaviour.

Without any NetworkPolicy, all pod-to-pod traffic is allowed. The moment any policy selects a pod, that pod denies everything not explicitly allowed in the policy's direction (ingress or egress). Also worth saying: NetworkPolicy does nothing unless the CNI plugin enforces it. Applying policies on a cluster whose CNI ignores them gives you compliance theatre, not security.

Scheduling and reliability

10. How do taints and tolerations differ from node affinity?

Taints repel pods from nodes (node-driven); affinity attracts pods to nodes (pod-driven). A toleration only permits scheduling onto a tainted node, it does not require it, which is why dedicated node pools need both: a taint to keep everyone else off and an affinity rule to pull the right workload in.

11. What is a PodDisruptionBudget and when did one save you (or bite you)?

A PDB limits how many pods a voluntary disruption (node drain, cluster upgrade) can take down at once. The bite-you story interviewers recognise: a PDB of minAvailable equal to the replica count makes every drain hang forever, blocking cluster upgrades. If you have watched a managed-cluster upgrade stall for hours because of an over-strict PDB, tell that story.

12. Your cluster autoscaler keeps adding nodes but pods stay Pending. Why might that be?

Because the pending pods have constraints that no new node satisfies either: a nodeSelector for a label the node group does not carry, a pod anti-affinity rule that needs more distinct zones than exist, or a PVC bound to a zone-specific volume while the autoscaler adds nodes in another zone. The autoscaler simulates scheduling before scaling, but only against node groups it manages, so mismatched constraints loop forever.

Senior-level and war-story territory

13. A node goes NotReady. What happens to its pods, exactly and in what order?

Nothing immediately. The node controller marks the node NotReady after missing heartbeats (default 40 seconds), then waits the eviction grace (default 5 minutes) before marking pods for deletion. Pods on the dead node cannot confirm termination, so StatefulSet pods stay stuck to guard at-most-one semantics until the node object is deleted or force-deleted. Reciting the two timing windows and the StatefulSet caveat is exactly what "senior" sounds like.

14. How would you debug high p99 latency that appeared after moving a service to Kubernetes?

A strong answer names CPU throttling first: check container_cpu_cfs_throttled_periods_total. CFS quota enforcement causes multi-millisecond stalls even at low average utilisation, and it is the single most common Kubernetes-specific latency regression. Then DNS (ndots:5 causes multiple lookups per external name), then conntrack exhaustion, then noisy neighbours on oversubscribed nodes.

15. How do you run a database on Kubernetes, and should you?

The honest answer wins: prefer a managed database unless you have a reason not to, and if you must run it in-cluster, use a battle-tested operator (CloudNativePG, Strimzi) rather than a hand-rolled StatefulSet. Explain what the operator gives you: automated failover, backup orchestration, and safe rolling upgrades that plain StatefulSets do not provide. Interviewers ask this to test judgement, not YAML.

Quick-fire round

These get asked as warm-ups or filler. One or two sentences each is the right depth:

What is the difference between a ReplicaSet and a Deployment? The Deployment manages ReplicaSets to provide rollout and rollback; you almost never create a ReplicaSet directly.
How do rolling updates achieve zero downtime? New ReplicaSet scales up while the old scales down, gated by maxSurge/maxUnavailable and readiness probes. No readiness probe, no real zero downtime.
What are init containers for? Sequential setup that must finish before app containers start: waiting for dependencies, running migrations, fixing volume permissions.
ConfigMap vs Secret? Same mechanism, different intent; Secrets are base64-encoded (not encrypted) and support encryption at rest in etcd plus tighter RBAC.
What is a headless service? clusterIP: None; DNS returns the pod IPs directly instead of a virtual IP. Needed for StatefulSet stable network identities.
Namespace vs cluster-scoped resources? Nodes, PersistentVolumes, StorageClasses and CRDs are cluster-scoped; most workloads are namespaced. RBAC can be either (Role vs ClusterRole).
What does kubectl drain do? Cordons the node, then evicts pods respecting PodDisruptionBudgets. DaemonSet pods are skipped (they would just come back).
How does HPA decide to scale? Compares observed metric (CPU by default) against target across pods, scales toward the ratio, with a stabilisation window to prevent flapping.
CrashLoopBackOff: what is it and what is the first command? Container repeatedly exits; back-off delay doubles up to five minutes. First command: kubectl logs <pod> --previous to see why the last run died.
What is the Ingress vs Gateway API story in 2026? Ingress is frozen; Gateway API is the successor with typed routes and role separation. New platforms should default to Gateway API, and interviewers increasingly expect you to know it.

How to prepare beyond a question list

Reading a list makes questions familiar. It does not make your answers survive follow-ups, and follow-ups are where offers are won. For each topic above, be ready to tell one story from real usage: a probe that caused an outage, a drain that hung, a throttled service. If your experience is thin, build the stories in a homelab: break a cluster deliberately and fix it.

If you want 52 senior-level Kubernetes questions with worked answers, production war stories and follow-up chains, that is exactly what Containers & Orchestration covers. It is one of the five books in the DevOps Interview Mastery bundle, and every purchase includes the free Interview-Day Playbook with a day-of checklist and behavioural answer frameworks.

Prefer to watch or listen? Answers like these are walked through on YouTube and on the podcast, on Spotify and Amazon Music.

Originally published at devopsinterview.cloud.

The 60-Day DevOps Interview Preparation Roadmap (2026 Edition)

DevOpsInterview.Cloud — Mon, 26 Oct 2026 09:20:03 +0000

Sixty days is enough to prepare for DevOps interviews properly, but only if you spend the time on what interviews actually test. Most candidates fail not because they know too little, but because they spread two months evenly across everything and arrive shallow in the five areas every loop drills: Linux and networking fundamentals, one cloud done well, Kubernetes, IaC and CI/CD, and production judgement. Here is the plan I would give a friend.

The ground rules

Depth beats coverage. One cloud provider deeply beats three superficially. Interviewers follow up until they find the edge of your knowledge; make sure that edge is deep in the topics they care about, and be honest everywhere else.

Everything you study, you build. A concept you have not touched breaks down on the second follow-up question. Every week below has a build component. A laptop with Docker plus one cheap cloud account is enough; kind or k3d gives you Kubernetes for free.

Stories are the currency of senior interviews. Keep a running document of incidents, debugging sessions, and design decisions from your current and past work. Every study week, add one story the topic reminded you of. You will use these in behavioural rounds and in every "tell me about a time" follow-up.

Weeks 1-2: Linux, networking, and the fundamentals filter

Interviews for every DevOps level start with a fundamentals filter, and it kills more candidates than Kubernetes ever will.

Study: processes and signals, file descriptors and permissions, systemd units, memory (what the OOM killer does and why), DNS resolution end to end, TCP handshakes and TIME_WAIT, TLS at a working level, HTTP semantics including load-balancer health checks.

Build: debug on purpose. Fill a disk and find it with df and du. Kill a process with each signal and explain the difference. Trace a slow DNS lookup with dig. Watch a TLS handshake in curl -v.

The bar: you can narrate "what happens when I type a URL and press enter" for five unbroken minutes, from DNS to rendered response, and answer interruptions at any layer.

Weeks 3-4: one cloud, done properly

Pick the provider your target jobs mention most (in most markets that is AWS, but go where your applications point) and go deep: VPC networking (subnets, route tables, NAT, security groups vs NACLs), IAM (roles, policies, and why instance profiles beat access keys), compute and autoscaling, managed databases, object storage, load balancers, and cost awareness, which is a 2026 interview topic in its own right.

Build: a three-tier application from scratch, no console clicking allowed after day one; script or Terraform everything. Break it deliberately: wrong security group, missing route, over-restrictive IAM policy, and debug each.

The bar: given a networking symptom ("instance cannot reach the internet"), you list the four or five most likely causes in the order you would check them.

If your interviews span multiple clouds, Cloud Platforms (AWS, Azure & GCP) maps the equivalent services and the questions asked about each, which saves you re-deriving everything three times.

Weeks 5-6: Kubernetes, the main event

For most DevOps loops in 2026, Kubernetes is the longest technical conversation. Two weeks of focused work gets you conversational at depth.

Study, in this order: pods and the pod lifecycle, Deployments and rolling updates, Services and Ingress, ConfigMaps and Secrets, requests/limits and what happens when each is breached, probes and their misuse, StatefulSets, scheduling (taints, affinity, PodDisruptionBudgets), and the debugging workflows for Pending, CrashLoopBackOff, and NotReady nodes.

Build: run kind on your laptop. Deploy the app from weeks 3-4 onto it. Then run the breakage drills: bad image tag, memory limit too low, readiness probe pointing at the wrong port, a node drain that a PDB blocks. Each drill is a war story you now own.

The bar: handed kubectl describe output for a broken pod, you can identify the problem class in under a minute, and you can explain what the control plane does during a rolling update without notes. Our post on Kubernetes interview questions is a good self-test at the end of week 6.

Week 7: Infrastructure as Code

Study: Terraform state (what it is, why remote, locking, drift), plan/apply mechanics, modules and environment structure, import and moved blocks, and how Terraform runs safely in CI. State questions dominate real interviews; syntax questions barely appear. Test yourself against our Terraform interview questions list.

Build: put everything from weeks 3-6 under Terraform in a repo with plan-on-PR wired up, even if the "CI" is a GitHub Actions workflow on a personal repo.

The bar: you can answer "someone changed it in the console, what happens on the next plan?" instantly, and structure a dev/staging/prod layout on a whiteboard while naming its trade-offs.

Week 8: CI/CD and GitOps

Study: pipeline anatomy (build, test, scan, artifact, deploy), deployment strategies and when each fits (rolling, blue/green, canary), GitOps with Argo CD or Flux and what reconciliation buys you over push deploys, secrets in pipelines (OIDC federation over static keys), and rollback thinking: how fast, triggered by what signal.

Build: a pipeline that ships your app to your kind cluster via Argo CD. Then break a deploy and roll it back, and note the time it took; "our rollback was two minutes because Git revert re-synced" is a lovely sentence to say in an interview. For the full progressive-delivery picture, CI/CD & GitOps covers pipelines, Argo CD, and canary analysis at interview depth.

Weeks 9-10 (senior track): observability, reliability, security

If you are interviewing at senior or SRE level, these two weeks are what separate you.

Study: SLIs, SLOs and error budgets with real numbers, the three telemetry types and when each answers a question (metrics for "is it broken", traces for "where", logs for "why"), alerting philosophy (page on symptoms, not causes), incident response roles, and the security topics loops now include: least privilege, supply chain basics, and secrets management.

Build: put Prometheus and Grafana on your cluster, define one SLO for your app, and make one alert fire for the right reason. Run one game-day against yourself.

The bar: you can design an alerting strategy that would not page anyone at 3 a.m. for a cause that self-heals, and defend it. This is precisely the ground SRE & Production Operations covers, question by question.

The final two weeks: convert knowledge into offers

Mock out loud. Answering in your head feels fluent; answering out loud exposes every gap. Record yourself on the twenty most likely questions, or drill with a friend who interrupts with follow-ups.

Prepare eight stories. Incidents, migrations, conflicts, failures. Structure each as situation, action, result, and what changed afterwards. Senior loops are half stories.

Do reconnaissance per company. Read their engineering blog and job description; a loop at a Kubernetes-heavy shop and one at a serverless shop deserve different final-week emphasis.

Day-of logistics matter more than people admit. Question frameworks, whiteboard habits, what to ask the interviewer, salary conversation timing. We wrote the Interview-Day Playbook for exactly this, and it comes free with every book purchase.

The compressed version

If you have less than 60 days: weeks 1-2 fundamentals are non-negotiable, one cloud at depth beats everything else, Kubernetes gets whatever time remains, and Terraform state plus one prepared incident story covers a surprising fraction of the rest. The DevOps Interview Mastery bundle packages all five books (250+ questions with worked answers across cloud, containers, IaC, CI/CD and SRE) if you want the structured version of this entire roadmap in one place.

Prefer to watch or listen? Answers like these are walked through on YouTube and on the podcast, on Spotify and Amazon Music.

Originally published at devopsinterview.cloud.

AWS, Azure, or GCP: Which Cloud Certification Should You Get First in 2026?

DevOpsInterview.Cloud — Sun, 25 Oct 2026 09:20:03 +0000

The honest answer up front: for most people, in most markets, the first certification should be on whichever cloud the jobs you want actually use, and if you have no signal at all, AWS is the safe default. But "it depends" is useless without the decision framework, so here is the one that actually settles it, followed by what each path looks like in 2026.

The three questions that decide it

1. What do your target jobs run? Open twenty job listings you would genuinely apply for and count the cloud mentions. This single exercise outweighs every generic comparison article, including this one. Enterprise-heavy markets and Microsoft shops lean Azure. Startups, product companies, and most of the global market lean AWS. Data-heavy and Kubernetes-forward companies over-index on GCP relative to its market share.

2. What is already on your CV? If you have years of .NET, Active Directory or Microsoft 365 around you, Azure certifications compound what you have. If you come from data engineering, GCP's BigQuery-centred ecosystem makes your story coherent. No prior signal? Default to AWS for raw job volume.

3. Are you optimising for a first job or a better job? For breaking in, certification plus demonstrable projects is the combination that works; the cert alone rarely moves the needle. For moving up, depth on the cloud you already use professionally beats starting a second provider almost every time.

The market reality in 2026

Market share has been stable for years: AWS around 30 percent, Azure around 20-25 percent, GCP around 12-13 percent. Job listings roughly track it, with two wrinkles worth knowing. Azure's share of enterprise job listings is higher than its infrastructure share, because large regulated organisations standardised on Microsoft agreements. And multi-cloud stopped being a buzzword: a large fraction of senior listings now name two providers, which changes the game later in your career but should not change your first move. Learn one cloud properly first; the second one is dramatically easier because the concepts transfer and only the service names and sharp edges differ.

If you pick AWS

The certification: Solutions Architect Associate (SAA-C03) remains the highest-signal first cert in the industry. Cloud Practitioner is fine as a warm-up if you are completely new, but it does not carry hiring weight; do not stop there.

Why it works: largest job pool, richest learning ecosystem, and interviewers everywhere can calibrate against it. AWS-specific interview questions are the best documented of the three, which makes preparation efficient.

The trap: SAA is a breadth exam and passing it does not make you interview-proof. The gap between "certified" and "can debug a VPC routing problem live" is exactly what interviews probe.

If you pick Azure

The certification: AZ-104 (Azure Administrator) is the strong first move for ops-flavoured roles; AZ-900 only if you need a gentle on-ramp. For DevOps roles specifically, AZ-400 comes after AZ-104, not instead of it.

Why it works: in enterprise and government-adjacent markets, Azure demand is intense and often less competed. If your region or industry is Microsoft country, an Azure path gets interviews an AWS cert would not.

The trap: Azure's portal-first culture can leave you shallow on automation. Interviewers at good Azure shops probe for CLI, Bicep or Terraform fluency precisely because so many candidates lack it.

If you pick GCP

The certification: Associate Cloud Engineer first, then Professional Cloud Architect or Professional DevOps Engineer. Google's Professional certs are widely considered the hardest of the three families, and they carry real respect.

Why it works: smaller job pool but a better ratio of demand to qualified candidates in its niches: data platforms, ML infrastructure, and Kubernetes-native companies (GKE is still the reference managed Kubernetes). If your target list is full of those companies, GCP first is not contrarian, it is correct.

The trap: betting on GCP in a market with few GCP employers. Do the twenty-listings exercise before committing.

What certifications actually do (and do not do)

They get you past filters: recruiter searches, HR keyword screens, and the "does this person know the basics" doubt that kills career-changer CVs. They structure your learning, which matters more than people admit. What they do not do is pass interviews for you. Every interviewer has met impressively certified candidates who could not explain what happens when a health check fails. The candidates who convert certifications into offers pair them with projects they can defend in detail and with interview-specific preparation, because interview questions ("this instance cannot reach the internet, debug it") are a different genre from exam questions.

That interview layer, across all three clouds, is what Cloud Platforms (AWS, Azure & GCP) covers: 50 senior-level interview questions spanning the services, networking, IAM and architecture conversations that follow once the certification gets you in the room. If you are building the whole preparation stack (cloud, Kubernetes, Terraform, CI/CD, SRE), the DevOps Interview Mastery bundle includes it alongside the other four books. And for the study plan around it, our 60-day DevOps interview roadmap slots certification prep into weeks 3 and 4.

The bottom line

Count the jobs, weigh your background, and commit to one provider for six months. AWS SAA if you need a default. Azure AZ-104 if your market is enterprise Microsoft. GCP ACE if your targets are data or Kubernetes companies. Then spend the time saved on the thing certifications cannot give you: the ability to reason about production out loud.

Prefer to watch or listen? Answers like these are walked through on YouTube and on the podcast, on Spotify and Amazon Music.

Originally published at devopsinterview.cloud.