DEV Community: Aisalkyn Aidarova

2-2 The Midnight Payment Processing Incident — Kubernetes, PostgreSQL, Terraform, CI/CD, Grafana and PagerDuty

Aisalkyn Aidarova — Fri, 30 Oct 2026 02:04:16 +0000

Student hands-on guide · 3 classes × 3 hours · AWS EKS training environment

Your mission

You are a DevOps/SRE engineer supporting a banking payment-processing platform. At midnight, PagerDuty notifies the primary on-call engineer that payment transactions are failing.

You open Grafana and see increasing HTTP 503 errors. But when you inspect Kubernetes, the application Pods are all Running and Ready.

How can the application be healthy according to Kubernetes while customers cannot complete payments?

You will investigate the application, its PostgreSQL database, Kubernetes networking, deployment history, monitoring, and the infrastructure changes that could have caused the incident.

This is a realistic fictional production case study, supported by a real, runnable training lab. You must not claim that it is an actual incident at a named bank.

1. Incident ticket

INC-2026-1047

SEV-1 · Scenario

Payment Processing API — Elevated 5xx Error Rate

PagerDuty · Payments Production · 12:05 AM

Reported impact: Payment submissions failing across the customer-facing banking application.

Monitoring: HTTP 503 responses increasing, payment success rate declining, API Pods still Ready.

Initial hypothesis: Application dependency failure, database connectivity, or recent infrastructure change.

Assigned: Primary On-Call SRE

Escalation: Incident Commander, Backend Engineering, DBA, Cloud Platform, Release Engineering, Security.

In the hypothetical production architecture, the application runs on AWS EKS across three Availability Zones and connects to Amazon RDS PostgreSQL. Terraform manages security groups and networking; GitHub Actions handles application and infrastructure pipelines.

For the actual student lab, we use two Payment API replicas and PostgreSQL inside Kubernetes to keep costs and resource requirements low.

2. Production architecture and team responsibilities

Reference production architecture. The classroom lab replaces RDS and AWS security-group failure with an in-cluster database and NetworkPolicy.

Team	Responsibility during incident
Primary On-Call SRE	Acknowledge alert, assess impact, investigate, mitigate and communicate
Incident Commander	Coordinate teams, severity, updates and escalation
Backend/Application Team	Investigate application errors, connection pooling, configuration and code
Database Administrator	Check database availability, active connections, locks, CPU, I/O and query performance
Cloud/Platform Team	Investigate EKS, VPC, security groups, routing and network policies
Release/CI/CD Team	Check recent releases, GitHub Actions workflows and Terraform changes
Security Team	Review and approve necessary access-rule changes

A primary on-call engineer may initially perform several of these functions, especially on a small team. Do not wait for a meeting before checking the impact and following your incident-response runbook.

3. What you will build

The lab creates:

Namespace payments-incident-lab
PostgreSQL database with a simulated_payments table
Python Payment API with two replicas
Kubernetes Services and health probes
Prometheus metrics for requests, errors and latency
A controlled database connectivity incident
Recovery and verification procedures

The lab uses synthetic payments only. It does not charge cards or connect to real payment processors.

4. Requirements

You need a training EKS cluster, working kubectl, Docker with permission to push images to a registry, and enough capacity for three small Pods in addition to monitoring.

For the NetworkPolicy incident, the cluster must have NetworkPolicy enforcement enabled. Installing a NetworkPolicy resource without an enforcing CNI will not necessarily block traffic.

First, verify your environment

kubectl config current-context
kubectl get nodes -o wide
kubectl get namespaces
kubectl get pods -n monitoring
kubectl get storageclass

Do not perform this exercise in a real production namespace.

Part A — Create all application and Kubernetes files

Create the project folders:

mkdir -p payment-incident-lab/{app,k8s,scripts,grafana,terraform}
cd payment-incident-lab

File 1 — `app/requirements.txt`

flask==3.1.1
psycopg[binary]==3.2.9
prometheus-client==0.22.1

File 2 — `app/app.py`

This is the application students will troubleshoot. Notice that /readyz checks the application process, but deliberately does not check database connectivity. This allows Pods to remain Ready while payment requests fail.

import os        try:            with connect() as conn:                with conn.cursor() as cur:                    cur.execute("SELECT 1")                    cur.fetchone()            REQ.labels("/api/db-check", "200").inc()            return jsonify(database="reachable"), 200        except Exception as exc:            app.logger.error(                "database connectivity failed: %s",                type(exc).__name__,            )            DB_ERRORS.inc()            REQ.labels("/api/db-check", "503").inc()            return jsonify(error="database_unavailable"), 503@app.post("/api/payments")def payment():    with DURATION.labels("/api/payments").time():        data = request.get_json(silent=True) or {}        key = request.headers.get("Idempotency-Key", "").strip()        if (            not key            or len(key) > 128            or not isinstance(data.get("amount_cents"), int)            or isinstance(data.get("amount_cents"), bool)            or not 1 <= data["amount_cents"] <= 100000        ):            REQ.labels("/api/payments", "400").inc()            return jsonify(                error="Idempotency-Key and integer "                      "amount_cents 1..100000 required"            ), 400        try:            with connect() as conn:                with conn.cursor() as cur:                    cur.execute(                        """                        INSERT INTO simulated_payments                        (idempotency_key, amount_cents)                        VALUES (%s, %s)                        ON CONFLICT (idempotency_key)                        DO NOTHING                        """,                        (key, data["amount_cents"]),                    )                    cur.execute(                        """                        SELECT id, amount_cents, status                        FROM simulated_payments                        WHERE idempotency_key = %s                        """,                        (key,),                    )                    row = cur.fetchone()

File 3 — `app/Dockerfile`

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

RUN useradd --uid 10001 --create-home appuser

USER 10001

EXPOSE 8080

CMD ["python", "app.py"]

File 4 — `k8s/00-namespace.yaml`

apiVersion: v1
kind: Namespace
metadata:
  name: payments-incident-lab

File 5 — `k8s/01-postgres.yaml`

This database is disposable. Its storage uses emptyDir, so deleting or replacing the database Pod can destroy its data.

apiVersion: v1
kind: Secret
metadata:
  name: postgres-auth
  namespace: payments-incident-lab
type: Opaque
stringData:
  POSTGRES_PASSWORD: classroom-only-not-production
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-init
  namespace: payments-incident-lab
data:
  001-init.sql: |
    CREATE TABLE IF NOT EXISTS simulated_payments (
      id BIGSERIAL PRIMARY KEY,
      idempotency_key TEXT NOT NULL UNIQUE,
      amount_cents INTEGER NOT NULL
        CHECK (amount_cents > 0),
      status TEXT NOT NULL
        DEFAULT 'simulated_approved',
      created_at TIMESTAMPTZ NOT NULL DEFAULT now()
    );
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lab-postgres
  namespace: payments-incident-lab
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: lab-postgres
  template:
    metadata:
      labels:
        app: lab-postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: payments
            - name: POSTGRES_USER
              value: labuser
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-auth
                  key: POSTGRES_PASSWORD
          volumeMounts:
            - name: init
              mountPath: /docker-entrypoint-initdb.d
            - name: data
              mountPath: /var/lib/postgresql/data
          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - pg_isready -U labuser -d payments
            periodSeconds: 5
          resources:
            requests:
              cpu: 50m
              memory: 128Mi
            limits:
              memory: 512Mi
      volumes:
        - name: init
          configMap:
            name: postgres-init
        - name: data
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: lab-postgres
  namespace: payments-incident-lab
spec:
  selector:
    app: lab-postgres
  ports:
    - name: postgres
      port: 5432
      targetPort: 5432

File 6 — `k8s/02-payment-api.yaml`

Replace REPLACE_WITH_YOUR_IMAGE with the container image you build.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  namespace: payments-incident-lab
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: payment-api
  template:
    metadata:
      labels:
        app: payment-api
    spec:
      containers:
        - name: payment-api
          image: REPLACE_WITH_YOUR_IMAGE
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: DB_HOST
              value: lab-postgres
            - name: DB_NAME
              value: payments
            - name: DB_USER
              value: labuser
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-auth
                  key: POSTGRES_PASSWORD
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 10
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
  name: payment-api
  namespace: payments-incident-lab
  labels:
    app: payment-api
spec:
  selector:
    app: payment-api
  ports:
    - name: http
      port: 8080
      targetPort: http

File 7 — `k8s/03-networkpolicy-outage.yaml`

This is the controlled failure. It blocks new incoming connections to the PostgreSQL Pod from the API.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-postgres-ingress
  namespace: payments-incident-lab
spec:
  podSelector:
    matchLabels:
      app: lab-postgres
  policyTypes:
    - Ingress
  ingress: []

Important: This only works when your cluster enforces Kubernetes NetworkPolicy. On EKS, check the Amazon VPC CNI NetworkPolicy configuration before running this exercise.

File 8 — `k8s/04-servicemonitor-optional.yaml`

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payment-api
  namespace: monitoring
  labels:
    release: monitoring
spec:
  namespaceSelector:
    matchNames:
      - payments-incident-lab
  selector:
    matchLabels:
      app: payment-api
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

The release: monitoring label must match the selector of your Prometheus installation. This file requires the Prometheus Operator ServiceMonitor CRD.

Part B — Build, deploy and verify

Step 1. Build and push the Docker image

Run from inside payment-incident-lab.

Replace the Docker Hub username with yours.

export IMAGE='YOUR_DOCKERHUB_USERNAME/jumptotech-payment-api:v1'

docker buildx build \
  --platform linux/amd64 \
  -t "$IMAGE" \
  --push ./app

If your EKS worker nodes are ARM64, use linux/arm64 instead.

Step 2. Deploy PostgreSQL

kubectl apply -f k8s/00-namespace.yaml

kubectl apply -f k8s/01-postgres.yaml

kubectl rollout status deployment/lab-postgres \
  -n payments-incident-lab \
  --timeout=180s

Check:

kubectl get pods -n payments-incident-lab
kubectl get svc -n payments-incident-lab

The PostgreSQL Pod should show 1/1 Running.

Step 3. Deploy Payment API

sed "s|REPLACE_WITH_YOUR_IMAGE|$IMAGE|" \
  k8s/02-payment-api.yaml | kubectl apply -f -

kubectl rollout status deployment/payment-api \
  -n payments-incident-lab \
  --timeout=180s

kubectl get deployments -n payments-incident-lab
kubectl get pods -n payments-incident-lab -o wide

Expected baseline:

NAME           READY   UP-TO-DATE   AVAILABLE
lab-postgres   1/1     1            1
payment-api    2/2     2            2

Step 4. Verify the database table

kubectl exec -n payments-incident-lab \
  deployment/lab-postgres -- \
  psql -U labuser -d payments \
  -c '\dt'

You should see the simulated_payments table.

Check the number of synthetic payments:

kubectl exec -n payments-incident-lab \
  deployment/lab-postgres -- \
  psql -U labuser -d payments \
  -c 'SELECT COUNT(*) FROM simulated_payments;'

Step 5. Test the application

Create scripts/02-smoke-test.sh:

#!/usr/bin/env bash
set -euo pipefail

NS=payments-incident-lab

kubectl run "smoke-$(date +%s)" \
  -n "$NS" \
  --image=curlimages/curl:8.11.1 \
  --restart=Never \
  --rm -i \
  --command -- sh -c '
    echo "READINESS:"
    curl -sS -i --max-time 10 \
      http://payment-api:8080/readyz

    echo
    echo "DB CHECK:"
    curl -sS -i --max-time 10 \
      http://payment-api:8080/api/db-check

    echo
    echo "SYNTHETIC PAYMENT:"
    curl -sS -i --max-time 10 \
      -X POST \
      http://payment-api:8080/api/payments \
      -H "Content-Type: application/json" \
      -H "Idempotency-Key: classroom-$(date +%s)" \
      -d "{\"amount_cents\":1250}"
'

Run:

bash scripts/02-smoke-test.sh

Expected results:

Endpoint	Expected HTTP status
`/readyz`	200
`/api/db-check`	200
`/api/payments`	200

The temporary curl Pod may display an attach warning after completing its request. Focus on the actual HTTP response.

Part C — Configure Grafana and Prometheus

First verify the monitoring components:

kubectl get pods -n monitoring

kubectl get crd servicemonitors.monitoring.coreos.com

kubectl get prometheus -n monitoring

Inspect the ServiceMonitor selectors:

kubectl get prometheus -n monitoring -o yaml \
  | grep -A12 serviceMonitorSelector

If the ServiceMonitor label matches your Prometheus configuration:

kubectl apply -f k8s/04-servicemonitor-optional.yaml

kubectl get servicemonitor -n monitoring

In Grafana, select your Prometheus data source and create the following panels.

Panel 1 — Payment API request rate by HTTP status

sum by (status) (
  rate(payment_http_requests_total[5m])
)

This distinguishes successful requests from HTTP 400, 409 and 503 responses.

Panel 2 — HTTP 5xx error rate

sum(
  rate(payment_http_requests_total{status=~"5.."}[5m])
) or vector(0)

Panel 3 — Database errors over five minutes

sum(
  increase(payment_db_errors_total[5m])
) or vector(0)

Panel 4 — Available replicas

kube_deployment_status_replicas_available{
  namespace="payments-incident-lab",
  deployment="payment-api"
}

Panel 5 — Desired replicas

kube_deployment_spec_replicas{
  namespace="payments-incident-lab",
  deployment="payment-api"
}

Panel 6 — Pod readiness

sum by (condition) (
  kube_pod_status_ready{
    namespace="payments-incident-lab",
    pod=~"payment-api-.*"
  }
)

Panel 7 — Payment request p95 latency

histogram_quantile(
  0.95,
  sum by (le) (
    rate(
      payment_http_request_duration_seconds_bucket{
        endpoint="/api/payments"
      }[5m]
    )
  )
)

Generate test traffic before expecting request metrics to appear. Check that Prometheus is actually scraping the Payment API target. The metrics may remain absent if the ServiceMonitor selector is incorrect.

Key learning point

A dashboard can show:

Available Pods

2 / 2

Kubernetes sees healthy processes

Payment API

503

Database-dependent requests fail

This is not contradictory. Kubernetes readiness and application business functionality are different signals.

Part D — Create the incident

Class 2: Midnight outage simulation

Your instructor announces:

It is 12:05 AM. PagerDuty has triggered a critical alert. Customers cannot complete payment transactions. Grafana shows elevated HTTP 503 errors, but all Payment API Pods appear healthy. You are the primary on-call engineer. Investigate the incident and identify the failing dependency.

Step 1. Confirm the healthy baseline

kubectl get pods -n payments-incident-lab

bash scripts/02-smoke-test.sh

Do not continue until the database check and payment request return HTTP 200.

Step 2. Introduce the controlled failure

Create scripts/03-inject-outage.sh:

#!/usr/bin/env bash
set -euo pipefail

kubectl apply -f k8s/03-networkpolicy-outage.yaml

kubectl get networkpolicy \
  -n payments-incident-lab

printf '\nRun the smoke test and investigate.\n'

Run:

bash scripts/03-inject-outage.sh

Then:

bash scripts/02-smoke-test.sh

When NetworkPolicy enforcement is active, expected behavior is:

Component	State
Payment API Pods	Running, Ready
PostgreSQL Pod	Running, Ready
API readiness endpoint	HTTP 200
Database connectivity endpoint	HTTP 503
Synthetic payment request	HTTP 503
Application logs	Database connectivity errors

If requests still succeed, verify that NetworkPolicy enforcement is enabled. Do not assume the incident is active simply because kubectl apply succeeded.

Step 3. Investigate the incident

Start with the Kubernetes resources:

kubectl get pods -n payments-incident-lab -o wide

kubectl get deployments -n payments-incident-lab

kubectl get svc -n payments-incident-lab

kubectl get endpointslices -n payments-incident-lab

Inspect the API Deployment:

kubectl describe deployment payment-api \
  -n payments-incident-lab

Inspect the API Pods:

kubectl describe pods \
  -l app=payment-api \
  -n payments-incident-lab

Inspect application logs:

kubectl logs deployment/payment-api \
  -n payments-incident-lab \
  --tail=100 \
  --max-log-requests=10

Look for errors indicating the application cannot communicate with PostgreSQL.

Inspect the database:

kubectl get pods \
  -l app=lab-postgres \
  -n payments-incident-lab

kubectl logs deployment/lab-postgres \
  -n payments-incident-lab \
  --tail=100

Check the NetworkPolicies:

kubectl get networkpolicy \
  -n payments-incident-lab

kubectl describe networkpolicy block-postgres-ingress \
  -n payments-incident-lab

kubectl get networkpolicy \
  -n payments-incident-lab \
  -o yaml

Check rollout history:

kubectl rollout history deployment/payment-api \
  -n payments-incident-lab

kubectl rollout status deployment/payment-api \
  -n payments-incident-lab

Important: In this particular incident, the API Deployment may have had no recent rollout at all. The problem is a networking configuration change, not necessarily a new container version.

Student investigation script

Create scripts/04-investigate.sh:

#!/usr/bin/env bash
set -euo pipefail

NS=payments-incident-lab

kubectl get pods,deploy,svc,endpointslices -n "$NS"

kubectl describe deployment payment-api -n "$NS"

kubectl logs deployment/payment-api \
  -n "$NS" \
  --tail=60 \
  --max-log-requests=10

kubectl get networkpolicy -n "$NS" -o yaml

kubectl rollout history deployment/payment-api -n "$NS"

Run:

bash scripts/04-investigate.sh

Root-cause analysis

In this lab, the NetworkPolicy selects the PostgreSQL Pod and allows no incoming traffic. The application can still start and answer /readyz, but new PostgreSQL connections fail.

The root cause is therefore database connectivity blocked by Kubernetes NetworkPolicy, not a crashed application Pod.

A blind kubectl rollout undo of the Payment API is not an appropriate fix because the network policy would remain in effect.

Part E — Restore service

After identifying the policy and following the classroom change procedure, remove the offending rule.

kubectl delete networkpolicy block-postgres-ingress \
  -n payments-incident-lab

Check that the application is still available:

kubectl rollout status deployment/payment-api \
  -n payments-incident-lab \
  --timeout=120s

Run the payment tests again:

bash scripts/02-smoke-test.sh

Verify the database contains the successful synthetic payments:

kubectl exec -n payments-incident-lab \
  deployment/lab-postgres -- \
  psql -U labuser -d payments \
  -c 'SELECT id, amount_cents, status, created_at
      FROM simulated_payments
      ORDER BY id DESC
      LIMIT 10;'

Create scripts/05-recover.sh:

#!/usr/bin/env bash
set -euo pipefail

kubectl delete networkpolicy block-postgres-ingress \
  -n payments-incident-lab \
  --ignore-not-found

kubectl rollout status deployment/payment-api \
  -n payments-incident-lab \
  --timeout=120s

bash scripts/02-smoke-test.sh

Run:

bash scripts/05-recover.sh

Recovery is verified when new database checks and synthetic payment requests succeed. In a real payment system, the application and database teams would also need to verify transaction integrity, retries and reconciliation.

Part F — Terraform and AWS RDS investigation

Why Terraform matters

In a production environment, PostgreSQL may run on Amazon RDS rather than inside Kubernetes.

Suppose a GitHub Actions workflow applies a Terraform change that accidentally removes the security-group rule allowing the EKS application to reach RDS on TCP port 5432.

The symptoms could resemble our lab: Kubernetes Pods are Ready, but application requests fail because database connections cannot be established.

The two failure mechanisms are different:

Classroom lab	Production reference scenario
Kubernetes NetworkPolicy blocks PostgreSQL ingress	AWS security group blocks EKS-to-RDS traffic
PostgreSQL runs inside EKS	PostgreSQL runs on Amazon RDS
Fix the Kubernetes NetworkPolicy	Restore the correct AWS security-group rule
Investigate `kubectl get networkpolicy`	Investigate AWS VPC, security groups, CloudTrail and Terraform changes

File 9 — `terraform/EXAMPLE_ONLY.tf`

This is a teaching example. Do not run terraform apply on this file against an existing AWS environment. The correct security group and identity must first be established.

variable "rds_security_group_id" {
  type = string
}

variable "approved_eks_application_security_group_id" {
  type = string
}

resource "aws_vpc_security_group_ingress_rule" "postgres_from_eks" {
  security_group_id = var.rds_security_group_id

  referenced_security_group_id = (
    var.approved_eks_application_security_group_id
  )

  ip_protocol = "tcp"

  from_port = 5432
  to_port   = 5432

  description = "Approved EKS application traffic to PostgreSQL"
}

Terraform investigation commands

For an actual managed Terraform project, first identify the correct repository, workspace, state and pipeline.

terraform version

terraform workspace show

terraform state list

terraform plan

Review the plan and the last GitHub Actions run. Determine whether a rule was removed, replaced, or changed.

Do not assume that the latest Terraform deployment caused the incident merely because it happened recently. Correlate the change timestamp with application errors, RDS connection failures, AWS configuration history and network evidence.

CI/CD investigation

Students should be able to explain this workflow:

Developer creates pull request

GitHub Actions: test, build, security checks

Terraform fmt, validate, plan and review

Approved infrastructure change

Application deployment and post-deploy tests

Prometheus / Grafana validation and alerting

An important production lesson: a pipeline can finish successfully while the system is broken. Successful execution of terraform apply only proves Terraform applied the requested changes; it does not prove payment processing still works.

Part G — Three-class teaching schedule

Class	Duration	Activities
Class 1	3 hours	Architecture, team roles, Docker, PostgreSQL, EKS Deployment, Services, API testing, Grafana
Class 2	3 hours	PagerDuty ticket, incident injection, logs, database investigation, NetworkPolicy, Terraform and CI/CD analysis
Class 3	3 hours	Recovery, verification, incident report, prevention, mock interview and follow-up questions

Class 1 — Build the platform

Students must finish with a working Payment API, two healthy replicas, PostgreSQL connectivity, successful synthetic payments and visible metrics.

Instructor questions: What is the difference between a Deployment and a Service? Why do we use a Secret? What does a readiness probe check? How does the API connect to PostgreSQL? Why might database availability differ from Pod readiness?

Class 2 — Investigate the outage

Do not immediately tell students the root cause. Present the incident ticket and ask them to gather evidence.

Students should identify that the Pods are Ready, the database Pod is running, the API reports database connectivity errors, and a NetworkPolicy is blocking the connection.

Instructor questions: Is this an application crash? Why are there no container restarts? Why does the HTTP 503 error appear? Why would restarting the Pods not fix the problem? What evidence would connect this to a Terraform change in AWS?

Class 3 — Recover and explain

Students must explain the root cause, restore connectivity, validate synthetic payments, review Grafana, and present an incident report.

They should distinguish between mitigation (restoring service), root cause (identifying the failing rule), and prevention (adding change controls and connectivity tests).

Part H — Interview preparation

These are the questions students should practice aloud, with technically accurate answers.

1. Your Pods are Running and Ready, but customers receive HTTP 503. What do you investigate?

Sample answer:

I would first assess the customer impact using Grafana or another monitoring tool. I would check the HTTP 5xx error rate, latency, and affected endpoints. Then I would inspect the Kubernetes Deployments, Pods, Services and application logs. Since the Pods are Ready, I would not assume Kubernetes itself is healthy end to end. I would investigate downstream dependencies such as the database, networking, authentication and external APIs.

2. How can all Pods be Ready while payment processing is failing?

Kubernetes readiness only checks what the configured readiness probe verifies. If the probe checks only the application process, the Pod can remain Ready even when the database is unreachable. That's why I would also monitor transaction success rates and perform synthetic end-to-end checks.

3. What would you check first to determine whether the database is the problem?

I would examine the application logs for connection timeouts, authentication failures, connection-pool exhaustion and query errors. Then I would check database health and connectivity. I would also review database metrics such as CPU, memory, connections, I/O and wait events.

4. How would you prove that a Terraform change caused the outage?

I would correlate the incident start time with the Terraform pipeline execution. Then I would review the Terraform plan, the applied changes and AWS configuration history. If the change removed a required RDS security-group rule, I would verify that the affected application could no longer reach the database. I would use that evidence to establish causation rather than relying only on timing.

5. Would you immediately run `kubectl rollout undo`?

Not automatically. I would first determine whether the application deployment caused the failure. If the root cause is a network rule or database configuration, rolling back the application might not help. I would follow the incident runbook and select the safest mitigation supported by the evidence.

6. What is the difference between HTTP 502, 503 and 504?

Code	Meaning	Common example
502 Bad Gateway	Gateway received an invalid response from an upstream server	Reverse proxy cannot obtain a valid upstream response
503 Service Unavailable	Service cannot currently handle the request	Payment API deliberately returns 503 when database is unavailable
504 Gateway Timeout	Gateway did not receive an upstream response in time	ALB or proxy times out waiting for an application response

A database timeout does not automatically mean the customer receives 504. In this lab, the application catches the error and explicitly returns 503.

7. Who would you involve in a SEV-1 incident?

I would follow the company's incident-response procedure. Depending on the impact, I would involve the incident commander, application engineers, database team, platform engineers and release team. I would communicate the symptoms, current impact, evidence collected, and mitigation plan so the teams could work in parallel.

8. How would you investigate a delayed failure after a successful release?

I would compare the release timestamp with the first customer errors and infrastructure changes. I would investigate whether the failure was triggered by connection-pool exhaustion, database load, expired credentials, a scheduled job, network changes or traffic growth. I would also check whether the application had existing connections that continued working temporarily before new connections started failing.

9. What database metrics would you inspect?

I would review CPU utilization, available memory, connection counts, disk I/O, read and write latency, active sessions, slow queries, locks and wait events. I would use those metrics to distinguish between database overload, slow queries, connection exhaustion and network connectivity failures.

10. What is a database connection pool?

A connection pool maintains reusable database connections so an application doesn't need to establish a new connection for every request. If the pool is exhausted or connections become invalid, requests may wait, time out or fail. I would check active connections, pool utilization, wait time and timeout settings.

11. How would you restore service without creating additional risk?

I would first identify the failing dependency and the last known good configuration. I would coordinate the mitigation through the incident process, restore only the necessary access, and verify recovery using application requests, database checks, monitoring and synthetic transactions. I would avoid broad security-group changes or unnecessary restarts.

12. What would you do after recovery?

I would confirm that error rates and payment success metrics returned to baseline. Then I would document the incident timeline, impact, root cause, mitigation and prevention actions. I would recommend post-deployment database connectivity checks, better dependency monitoring, Terraform plan review and appropriate change approvals.

Part I — Student incident report

Every student must complete this report after Class 3.

Incident report template

Incident ID: INC-2026-1047

Severity: SEV-1 (training scenario)

Affected service: Payment Processing API

Detection: PagerDuty / Grafana

Start time: Record actual lab injection time

Recovery time: Record actual successful verification time

Customer impact: Synthetic payment requests failed in the lab; production customer impact is hypothetical

Observed symptoms: API Pods Ready, database-dependent requests returned HTTP 503

Root cause: PostgreSQL ingress blocked by Kubernetes NetworkPolicy

Mitigation: Removed the offending policy in the isolated lab

Verification: Database checks and synthetic payments returned HTTP 200

Prevention: Network change reviews, dependency checks, monitoring, and automated connectivity tests

Evidence: Attach command outputs, relevant logs, Grafana screenshots and the exact configuration change.

File 10 — `scripts/01-check.sh`

#!/usr/bin/env bash
set -euo pipefail

kubectl get nodes
kubectl get ns
kubectl get pods -n monitoring

kubectl get pods,deploy,svc \
  -n payments-incident-lab

kubectl get networkpolicy \
  -n payments-incident-lab

File 11 — `scripts/06-cleanup.sh`

Use only after the classes are finished. This deletes the entire training namespace and its synthetic database.

#!/usr/bin/env bash
set -euo pipefail

read -r -p \
  'Delete ENTIRE payments-incident-lab namespace? Type DELETE: ' \
  answer

if [[ "$answer" == "DELETE" ]]; then
  kubectl delete namespace payments-incident-lab
else
  echo "Canceled."
fi

To clean up the optional ServiceMonitor separately:

kubectl delete servicemonitor payment-api \
  -n monitoring \
  --ignore-not-found

Final student challenge

Your instructor will ask:

"It is 12:05 AM. Your Payment API has two healthy Pods, PostgreSQL is Running, and your application deployment has not changed. However, payment requests are returning HTTP 503. What are your first actions, how do you identify the root cause, who do you involve, and how do you restore service safely?"

You must be able to explain the investigation in your own words, demonstrate the relevant commands, identify the exact failed dependency, and verify the recovery.

Your goal is not to memorize commands. Your goal is to think and communicate like a DevOps/SRE engineer responding to a production incident.

JumpToTech DevOps Engineering · Practical Incident Response Training

JumpToTech DevOps Production Incident Lab #2 The Midnight Payment Processing Incident

Aisalkyn Aidarova — Fri, 30 Oct 2026 01:05:13 +0000

Student training handbook | 3 classes × 3 hours | AWS EKS · Terraform · GitHub Actions · PostgreSQL RDS · Prometheus · Grafana · PagerDuty

Students, welcome to your next production incident investigation.

In our previous incident, you investigated Kubernetes Pods that became unavailable after a deployment. This time, the challenge is different.

All Kubernetes Pods are healthy. The deployment succeeded. The application is reachable. But customers cannot complete payments.

This is the type of scenario that tests whether you understand the entire production system—not just Kubernetes commands.

You will work as an on-call DevOps/SRE engineer at a fictional enterprise payment platform. The scenario is designed to resemble incidents that could occur in large financial technology companies, banks, and e-commerce organizations. All company details, ticket numbers, figures, and incident events below are fictional training data.

By the end of three classes, you should be able to investigate a multi-system incident, identify an infrastructure change, coordinate with other engineering teams, restore service safely, explain database connectivity, and defend your decisions during a technical interview.

INC-2026-1047 — Payment Authorization Failures

SEV-1

Payment success rate

41%

Baseline: 99.95%

HTTP 5xx rate

37%

On payment API

Kubernetes Pods

12/12

Ready and Running

Database

Available

RDS instance is running

Incident clue: A Terraform deployment completed approximately 20 minutes before the payment failures began.

The critical interview question is:

“If all Kubernetes Pods are healthy and the RDS database is available, why are customers receiving payment failures?”

You are not allowed to answer, “I would restart the Pods,” without first explaining what evidence justifies that action.

Your three-class mission

Class	Duration	Your objective
Class 1 — Detection and triage	3 hours	Understand the ticket, production architecture, incident roles, and evidence collection
Class 2 — Deep troubleshooting	3 hours	Investigate Kubernetes, CI/CD, Terraform, networking, and PostgreSQL; identify the root cause
Class 3 — Recovery and interviews	3 hours	Restore service, verify transactions, complete the postmortem, and pass a mock technical interview

You will not be given the root cause at the beginning. Your instructor will reveal additional evidence as you progress.

1. The company and production environment

You work for NorthStar Payments, a fictional payment technology company processing transactions for online retailers.

NorthStar provides payment authorization, order management, transaction history, refunds, and customer account services.

The production system runs on AWS. The platform team manages infrastructure through Terraform, developers deploy applications through GitHub Actions, and SRE engineers monitor the platform using Prometheus, Grafana, CloudWatch, and PagerDuty.

Production architecture

Conceptual production architecture. AWS networking and service placement are simplified for training.

The system has these components:

Component	Purpose	Responsible team
Amazon EKS	Runs containerized services	Platform / SRE
Payment API	Receives and processes payment requests	Backend application team
Payment workers	Process asynchronous events	Backend application team
Amazon RDS PostgreSQL	Stores transaction and payment records	Database / DBA team
AWS Security Groups	Control permitted network connections	Cloud platform / security
Terraform	Provisions and changes AWS infrastructure	Cloud platform
GitHub Actions	Tests and deploys infrastructure and applications	DevOps / release engineering
Prometheus and Grafana	Collect and visualize service metrics	SRE / observability
CloudWatch	AWS infrastructure logs and metrics	Cloud platform / SRE
PagerDuty	Routes urgent alerts to on-call responders	SRE / incident management
Jira Service Management	Tracks incident tickets and investigation	Service management

Example workload sizes

For this case study, the production environment has 12 Payment API Pods, 6 background workers, 3 Availability Zones, and one Multi-AZ PostgreSQL RDS deployment. These are illustrative values, not actual figures from any bank.

The Payment API has a readiness endpoint at /ready, which checks whether the HTTP process is responsive. It does not check whether the application can reach PostgreSQL. That design choice will become important.

2. Your production engineering team

An enterprise incident is not normally handled by one engineer working alone. You may be the first person paged, but different teams own different systems.

Incident response team structure

Incident Commander

Coordinates response and decisions

Primary SRE

Triage, metrics, Kubernetes

Platform Engineer

Terraform, VPC, security groups

Backend Engineer

Payment API, application logs

Database Engineer

RDS, SQL, connection health

Release Engineer

CI/CD, change history

Security Engineer

Network access, policy review

Customer support and business stakeholders receive coordinated status updates through the incident communication process.

Who is responsible for what?

Primary on-call DevOps/SRE engineer — your role

You acknowledge the alert, establish the initial technical picture, identify impacted components, collect evidence, coordinate with the incident team, and execute approved operational actions within your access and authority.

Incident Commander

Owns coordination, severity management, escalation, communications, and ensuring someone is assigned to each workstream. The incident commander does not necessarily perform the technical fix.

Application engineer

Investigates payment request failures, database client errors, connection pool behavior, retry logic, and application code.

Cloud platform engineer

Investigates AWS networking, Terraform state and plans, security groups, IAM, EKS networking, and infrastructure changes. This engineer normally owns the corrective Terraform pull request.

Database engineer

Checks database availability, active connections, PostgreSQL errors, query performance, locks, replication or failover state, and database-side access settings.

Release engineer

Finds the GitHub Actions deployment run, commit SHA, pull request, approvers, change window, and any simultaneous releases.

Security engineer

Reviews whether a proposed security group change is permitted and ensures recovery does not introduce unnecessary network exposure.

Customer support and communications

Collect customer reports and distribute approved incident updates. They should not receive raw database credentials or sensitive customer records.

Interview question

“If you are the primary on-call engineer, are you personally responsible for fixing every component?”

A strong answer:

“I am responsible for responding to the incident, understanding the impact, and driving technical triage within my role. I would engage the correct service owners for specialized components such as RDS, networking, and application code. We work together under the incident response process to restore service safely.”

3. Where the production ticket comes from

In this scenario, Prometheus detects the elevated error rate and sends the alert to Alertmanager. Alertmanager routes it to PagerDuty. PagerDuty pages the primary on-call engineer. An incident ticket is then created in Jira Service Management through the company's configured integration or incident workflow.

PagerDuty — Production Alert

12:15 AM · Payment Platform · Primary On-Call

High Payment Failure Rate

Service: payment-api-prod

Environment: production

Cluster: northstar-prod-eks

Namespace: payments-prod

Trigger: HTTP 5xx rate exceeds the configured threshold for 5 minutes.

Customer payment authorization is degraded.

Example Jira incident ticket

Jira Service Management · INC-2026-1047

Payment authorization failures — customer impact

Status	Investigating
Assigned group	Production SRE
First detected	12:15 AM
Customer symptoms	Payment errors and timeouts
Related change	CHG-2026-318 — Terraform network update
Impacted service	Payment API

Description

Payment success rate has dropped from approximately 99.95% to 41%. Payment API 5xx errors have increased. Kubernetes reports all 12 API Pods as Ready. RDS instance status is Available. Investigate application dependencies, infrastructure changes, and customer impact.

Initial assignment: Primary on-call SRE.

Where else might you learn about the incident?

You may see the same incident in the PagerDuty mobile app, Slack incident channel, Jira queue, Grafana dashboard, CloudWatch alarms, or customer support escalation. Different companies use different integrations.

Student question: “What is the difference between an alert, an incident ticket, and a change ticket?”

An alert is a monitoring signal that requires attention.
An incident ticket tracks the unexpected service disruption and response.
A change ticket documents an intended production change, its approvals, timing, and rollback plan.

4. The incident timeline

Time	Event
11:20 PM	GitHub Actions infrastructure pipeline starts
11:28 PM	Terraform plan is reviewed and approved
11:40 PM	Terraform apply completes successfully
11:45 PM	Infrastructure pipeline reports success
12:00 AM	Payment API error rate begins increasing
12:10 AM	Payment success rate drops sharply
12:15 AM	PagerDuty pages the primary on-call engineer
12:17 AM	SRE acknowledges and begins triage
12:22 AM	Incident Commander engaged
12:30 AM	Database connectivity errors identified
12:40 AM	Recent infrastructure change becomes the leading suspect
12:50 AM	Recovery change is prepared and reviewed
1:05 AM	Connectivity restored; application recovery monitored
1:20 AM	Payment success and error metrics return to baseline
1:35 AM	Incident moves into observation and post-incident follow-up

These timestamps are the planned case-study narrative, not measured outcomes from the student lab.

5. The mystery: why healthy Pods cannot process payments

At first, you check Kubernetes:

kubectl get pods -n payments-prod

The output shows:

NAME                           READY   STATUS    RESTARTS
payment-api-7d8f5b6c9-a1b2c    1/1     Running   0
payment-api-7d8f5b6c9-d3e4f    1/1     Running   0
payment-api-7d8f5b6c9-g5h6i    1/1     Running   0
...
12 payment-api Pods Ready

Then you inspect the Deployment:

kubectl get deployment payment-api -n payments-prod

NAME          READY   UP-TO-DATE   AVAILABLE
payment-api   12/12   12           12

You inspect RDS in AWS and see:

DB instance: northstar-payments-prod
Status: Available
Engine: PostgreSQL
Port: 5432

Yet the payment API returns:

HTTP/1.1 503 Service Unavailable

Application logs contain:

ERROR payment authorization failed
ERROR could not establish database connection
ERROR connection timeout to postgres endpoint:5432

This is the central lesson of the incident:

Kubernetes Pod readiness, application dependency health, and successful business transactions are three different things.

A database can be operational while a particular application cannot connect to it.

Class 1 — Detect, assess, and investigate

3 hours · Students act as the primary on-call SRE

Class 1 schedule

Time	Activity
0:00–0:25	Read incident ticket, identify business impact
0:25–0:50	Understand team structure and incident communication
0:50–1:20	Investigate Kubernetes health
1:20–1:50	Investigate Grafana and HTTP errors
1:50–2:20	Check application logs and database connectivity
2:20–2:45	Identify next investigation steps
2:45–3:00	Mock interview and handover

Task 1 — Your first five minutes

You receive the PagerDuty alert. You are the primary on-call engineer.

Your interviewer asks:

“What would you do immediately?”

A model response:

“First, I would acknowledge the PagerDuty alert. Then I would open our monitoring dashboards to validate the impact, including payment success rate, HTTP 5xx errors, latency, and Kubernetes availability. I would determine whether the issue affects all customers or a subset, identify the impacted services, and follow our incident escalation procedure. I would not immediately restart Pods because the evidence does not yet show that restarting them would help.”

What you should ask the incident team

Which payment operations are failing: authorization, capture, refund, or all of them?
When did the errors begin?
Are all customers affected or only certain regions or merchants?
Did an application release, infrastructure change, or database maintenance occur?
Are failures isolated to one service or shared by multiple services?
Are transactions being rejected, timing out, or potentially duplicated?
Is there a known-good recovery procedure?

Important: During a payment incident, do not blindly retry failed transactions. A timeout does not necessarily mean a payment was never processed. The application and payment teams must verify transaction status and idempotency before retrying.

Task 2 — Kubernetes investigation

Run the following commands in the designated training namespace. In the production case study, the namespace is payments-prod.


# Confirm cluster context before running commands
kubectl config current-context

# Inspect deployments
kubectl get deployments -n payments-prod

# Inspect Pods and their nodes
kubectl get pods -n payments-prod -o wide

# Inspect Services
kubectl get svc -n payments-prod

# Inspect Service endpoints
kubectl get endpointslices -n payments-prod

# Inspect nodes
kubectl get nodes -o wide

# Inspect the payment Deployment
kubectl describe deployment payment-api -n payments-prod

# Inspect one payment Pod
kubectl describe pod POD_NAME -n payments-prod

# Review application logs
kubectl logs POD_NAME -n payments-prod --tail=100

What should you conclude?

If all Pods are Ready, you have evidence that Kubernetes considers them eligible to serve traffic. You do not yet have evidence that the payment workflow is functioning correctly.

Check the readiness probe:

kubectl get deployment payment-api \
  -n payments-prod -o yaml

Suppose you discover:

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5

The endpoint checks whether the application process can respond, but it does not test PostgreSQL connectivity.

Interview follow-up: “Should a readiness probe always connect to the database?”

No. Dependency-aware readiness checks can be useful, but they require careful design. If every Pod fails readiness during a shared database outage, the Service may lose all endpoints, potentially worsening recovery. Teams often separate lightweight process readiness, dependency health, and end-to-end business transaction monitoring.

Task 3 — Grafana investigation

Your instructor should show four monitoring panels.

Example incident dashboard

Available API replicas

12 / 12

Stable

Payment success rate

41%

Severely degraded

HTTP 5xx error rate

37%

Elevated

Database connection failures

Increasing

Application-side metric

PromQL examples

These queries assume that the required Kubernetes and application metrics are actually collected. Metric names such as http_requests_total and payment_transactions_total are example application instrumentation; they do not appear automatically from kube-state-metrics.

Available replicas

kube_deployment_status_replicas_available{
  namespace="payments-prod",
  deployment="payment-api"
}

Desired replicas

kube_deployment_spec_replicas{
  namespace="payments-prod",
  deployment="payment-api"
}

HTTP 5xx percentage

100 *
sum(rate(http_requests_total{
  service="payment-api",
  status=~"5.."
}[5m]))
/
clamp_min(
  sum(rate(http_requests_total{
    service="payment-api"
  }[5m])),
  0.001
)

Payment success percentage

100 *
sum(rate(payment_transactions_total{
  result="success"
}[5m]))
/
clamp_min(
  sum(rate(payment_transactions_total[5m])),
  0.001
)

95th percentile HTTP latency

histogram_quantile(
  0.95,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{
      service="payment-api"
    }[5m])
  )
)

For HTTP status-code metrics, instrument the application or collect appropriate ingress/load-balancer metrics. Prometheus cannot derive HTTP 503 counts from Pod readiness alone.

Interviewer asks

“Grafana shows all 12 Pods healthy but the payment success rate is 41%. Where do you investigate next?”

A strong response:

“I would investigate the application request path and its dependencies. I would review payment API logs, trace failed requests, check database connection metrics, and test connectivity from the application network. I would also review recent infrastructure and application changes.”

Task 4 — Application logs

Example logs:

00:11:42 INFO  POST /api/payments/authorize
00:11:42 ERROR database connection timeout
00:11:42 ERROR SQLSTATE 08001
00:11:42 WARN  payment authorization unavailable
00:11:42 INFO  response_status=503 duration_ms=5002

Explain to the instructor:

What does SQLSTATE 08001 suggest?
Does this prove the database server has crashed?
What is the difference between a connection timeout and an authentication failure?
Why might the application return 503 while its Pod remains Ready?
Which team should investigate the database, and which team should investigate the application?

A useful answer to question 2:

“No. A database connection failure can result from networking, security groups, routing, DNS, authentication, database capacity, or a database outage. I would test each layer before deciding which component is responsible.”

Class 1 communication exercise

Students should practice this update to the incident commander:

Incident update — Payment API

We have confirmed significant customer-facing payment failures. Payment success rate has dropped to approximately 41%, and HTTP 5xx errors are elevated.

All 12 Payment API Pods remain Ready, and the RDS instance reports Available. However, application logs show PostgreSQL connection timeouts.

We are investigating the application-to-database network path and reviewing recent infrastructure changes with the platform and database teams.

The root cause is not yet confirmed. Our priority is restoring payment authorization safely and preventing duplicate transactions.

Class 1 homework

Students should submit a short incident update, the commands they would use, their initial hypotheses, and an explanation of why healthy Pods do not guarantee healthy payment processing.

Class 2 — Terraform, CI/CD, networking, and database

3 hours · Deep technical investigation

Class 2 schedule

Time	Activity
0:00–0:25	Review Class 1 evidence
0:25–0:55	Trace the payment request to PostgreSQL
0:55–1:25	Investigate database connectivity and metrics
1:25–1:55	Review GitHub Actions and Terraform change history
1:55–2:25	Compare security group rules
2:25–2:45	Confirm the cause and propose recovery
2:45–3:00	Technical interview challenge

Task 1 — Understand the database connection

The payment application uses PostgreSQL to store payment transactions.

Example database tables:


CREATE TABLE payments (
    payment_id UUID PRIMARY KEY,
    merchant_id VARCHAR(50) NOT NULL,
    amount_cents BIGINT NOT NULL,
    currency CHAR(3) NOT NULL,
    status VARCHAR(20) NOT NULL,
    idempotency_key VARCHAR(100) UNIQUE NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE payment_events (
    event_id BIGSERIAL PRIMARY KEY,
    payment_id UUID NOT NULL REFERENCES payments(payment_id),
    event_type VARCHAR(50) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

Example payment statuses are PENDING, AUTHORIZED, DECLINED, and FAILED.

In a real payment platform, financial transaction data needs additional controls, auditing, access restrictions, and carefully designed state transitions. This simplified schema is for learning connectivity and SQL troubleshooting, not a complete payment ledger.

Normal payment request flow

Customer sends POST /api/payments/authorize.
ALB routes the request to the Payment API in EKS.
Payment API validates the request and checks its idempotency key.
Application opens or reuses a PostgreSQL connection.
Application reads or writes payment state and coordinates the external authorization workflow.
Application returns the appropriate response and emits metrics.

What happens if step 4 fails?

The application may be unable to read or persist the payment state. It must not blindly repeat external authorization requests or assume that a timed-out transaction did not succeed.

Task 2 — Test database connectivity

The database engineer confirms that RDS is Available. Your next question is whether the Payment API network can reach the database endpoint on TCP port 5432.

For the production case study, you would use approved access through the existing application Pod or an authorized diagnostic Pod in the same network context.


# Check the ServiceAccount and Pod configuration
kubectl get pod POD_NAME \
  -n payments-prod -o yaml

# Inspect relevant application logs
kubectl logs POD_NAME \
  -n payments-prod --tail=100

# Check database DNS resolution
getent hosts DB_ENDPOINT

# Check TCP connectivity
nc -vz -w 5 DB_ENDPOINT 5432

The last two commands must run in an authorized environment with equivalent network access to the affected application, not simply on an unrelated EC2 management host. A successful connection from a bastion does not prove that an EKS Pod can connect.

Expected investigation evidence:

DNS resolution: successful

TCP connection:
connect to DB_ENDPOINT port 5432:
Connection timed out

This suggests a network-path problem but does not yet identify the exact AWS rule responsible.

Interviewer follow-ups

“What if DNS fails?”

Investigate CoreDNS, VPC DNS configuration, DNS records, resolver access, and the endpoint being used.

“What if DNS works but TCP times out?”

Investigate security groups, Network ACLs, routing, NetworkPolicies, database endpoint reachability, and possible network packet drops.

“What if TCP succeeds but PostgreSQL authentication fails?”

Investigate database credentials, IAM database authentication if configured, Secrets Manager, PostgreSQL roles, SSL requirements, and authentication settings.

“What if TCP succeeds but SQL queries are slow?”

Investigate query plans, locks, active sessions, connection pools, CPU, I/O, and database wait events.

Task 3 — Inspect PostgreSQL health

The database team runs approved diagnostic queries:


-- Verify database responds
SELECT 1;

-- Count current connections
SELECT count(*) AS active_connections
FROM pg_stat_activity;

-- Inspect connection states
SELECT state, count(*)
FROM pg_stat_activity
GROUP BY state;

-- Check configured connection limit
SHOW max_connections;

-- Review sessions waiting on resources
SELECT
    pid,
    state,
    wait_event_type,
    wait_event
FROM pg_stat_activity
WHERE wait_event IS NOT NULL;

If the database is responsive and the connection count is low, but Payment API Pods cannot connect, the team should investigate the network path.

A low connection count alone does not prove a security group problem. It becomes useful evidence when combined with timeouts, recent changes, and network configuration.

AWS RDS checks

aws rds describe-db-instances \
  --db-instance-identifier northstar-payments-prod \
  --query 'DBInstances[0].{Status:DBInstanceStatus,Endpoint:Endpoint.Address,Port:Endpoint.Port,VpcSecurityGroups:VpcSecurityGroups}'

Check CloudWatch or RDS monitoring for database connections, CPU utilization, free memory, storage, and relevant logs. Enhanced Monitoring and Performance Insights/Database Insights may provide deeper diagnostics if configured.

Task 4 — Investigate the CI/CD pipeline

The release engineer discovers a GitHub Actions workflow:

GitHub Actions — Production Infrastructure

Succeeded

Workflow: terraform-production.yml · Completed 11:45 PM

Checkout repository
Terraform format and validate
Terraform plan
Production approval
Terraform apply

Interviewer: “If the Terraform pipeline succeeded, how could it have caused the incident?”

Answer:

“A successful Terraform apply means the requested infrastructure changes were applied. It does not prove that application connectivity or business functionality remained healthy. The pipeline may lack post-deployment integration tests or application-level validation.”

Production workflow example

Students should understand this pipeline structure:


name: Production Terraform

on:
  workflow_dispatch:

permissions:
  contents: read
  id-token: write

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # Configure approved AWS OIDC credentials
      # Install a pinned, approved Terraform version
      - run: terraform fmt -check
      - run: terraform init -input=false
      - run: terraform validate
      - run: terraform plan -out=tfplan

  # Production apply should be a separate approved job.
  # Use a protected environment, artifact integrity
  # checks, and the reviewed saved plan.

This is a teaching skeleton, not a complete production-ready workflow. In a real pipeline, the saved plan must be securely transferred between jobs, access controlled, and applied only after approval.

Questions students must answer

What is the difference between terraform plan and terraform apply?
What does Terraform state store?
Why use a remote backend and state locking?
Why should GitHub Actions authenticate to AWS using OIDC rather than long-lived access keys?
What does a protected GitHub environment accomplish?
Why is manual approval useful but not sufficient?
What post-apply checks would have detected the payment outage?
Why can an infrastructure change fail without any new application image?

Task 5 — The Terraform change

The platform engineer compares the previous and current security group configuration.

Before the change

The RDS security group allows PostgreSQL access from the EKS application security group.

resource "aws_vpc_security_group_ingress_rule" "postgres_from_eks" {
  security_group_id            = aws_security_group.rds.id
  referenced_security_group_id = aws_security_group.eks_application.id

  ip_protocol = "tcp"
  from_port   = 5432
  to_port     = 5432
}

After the change

A Terraform refactor accidentally replaces the application security group reference with an unrelated security group.

resource "aws_vpc_security_group_ingress_rule" "postgres_from_eks" {
  security_group_id            = aws_security_group.rds.id
  referenced_security_group_id = aws_security_group.management.id

  ip_protocol = "tcp"
  from_port   = 5432
  to_port     = 5432
}

Important: The code is illustrative. Actual EKS Pod network traffic may use node security groups or Pod security groups depending on the VPC CNI and security-groups-for-Pods configuration. The effective source security group must be verified in the actual architecture.

What happened?

The authorized source for PostgreSQL traffic was changed.

The database remained healthy, and the EKS Pods remained Ready, but new database connections from the application network were no longer permitted by the intended ingress rule.

Existing established connections may continue temporarily because AWS security groups are stateful and connection tracking can preserve some flows. As connections close, recycle, or time out, application failures may increase.

This explains why the outage can appear after the Terraform apply rather than exactly when the change is made.

Compare the infrastructure change

git log --oneline -10

git diff PREVIOUS_COMMIT CURRENT_COMMIT -- terraform/

terraform plan

For AWS-side verification:

aws ec2 describe-security-groups \
  --group-ids sg-EXAMPLE \
  --query 'SecurityGroups[*].IpPermissions'

Do not assume a security group ID from a screenshot or example. Identify the actual RDS security group and the effective source security group first.

Root cause evidence chain

Terraform changed the RDS ingress source security group.
The infrastructure change occurred before the payment errors.
Payment API logs show database connection timeouts.
RDS remains operational and responsive from an authorized network.
The application network cannot establish new TCP connections to port 5432.
Restoring the approved application security group rule restores connectivity and payment processing.

This combination of change history, network testing, application logs, and successful recovery provides much stronger causal evidence than timing alone.

Class 2 interview challenge

“How would you prove the Terraform change caused the payment failures rather than a PostgreSQL outage?”

Expected answer:

“I would compare the Terraform plan and AWS security group configuration with the last known-good state, verify that the database itself is responsive, and test connectivity from the affected application network. I would correlate the network failures with the infrastructure change and confirm that restoring the correct rule resolves the connectivity issue.”

Class 3 — Recovery, prevention, and interview mastery

3 hours · Students act as incident responders and interview candidates

Class 3 schedule

Time	Activity
0:00–0:30	Decide on mitigation and evaluate rollback risk
0:30–1:00	Restore database connectivity
1:00–1:30	Verify Kubernetes, database, and customer transactions
1:30–2:00	Write the incident report and preventive actions
2:00–2:40	Advanced technical interview
2:40–3:00	Final individual interview assessment

Task 1 — How do you restore production?

The platform engineer has identified the incorrect RDS security group ingress rule.

The incident commander asks:

“What is the safest and fastest way to restore service?”

The team should evaluate two possible recovery approaches.

Option A — Revert the incorrect Terraform change. Restore the previously approved application security group rule in code, review the plan, obtain the necessary approval, and apply the corrective configuration.

Option B — Emergency network change. If the approved emergency procedure allows an immediate AWS-side correction, an authorized engineer can restore the exact known-good rule and subsequently reconcile Terraform code and state. This must be documented and reviewed to avoid configuration drift.

Do not open PostgreSQL to 0.0.0.0/0, disable all security groups, or expose RDS publicly as a troubleshooting shortcut.

Example corrected Terraform configuration

resource "aws_vpc_security_group_ingress_rule" "postgres_from_eks" {
  security_group_id            = aws_security_group.rds.id
  referenced_security_group_id = aws_security_group.eks_application.id

  ip_protocol = "tcp"
  from_port   = 5432
  to_port     = 5432

  description = "PostgreSQL access from approved EKS application SG"
}

Interviewer asks

“Would you simply run terraform apply again?”

No. First inspect the planned changes. Applying the same incorrect configuration again would not solve the incident.

“Would you run terraform destroy?”

No. Destroying production infrastructure would be highly dangerous and unrelated to this targeted connectivity issue.

“Would you roll back the Kubernetes Deployment?”

Not automatically. If the root cause is an RDS security group rule, reverting the application image may have no effect. The corrective action should target the confirmed cause.

Task 2 — Verify recovery at every layer

After the approved rule is restored, verify the application network can connect to PostgreSQL.


# Verify application Pod health
kubectl get pods -n payments-prod

# Verify Deployment availability
kubectl get deployment payment-api -n payments-prod

# Verify Service endpoints
kubectl get endpointslices -n payments-prod

# Check recent application errors
kubectl logs deployment/payment-api \
  -n payments-prod --tail=100

# Confirm application rollout state
kubectl rollout status \
  deployment/payment-api \
  -n payments-prod

Then verify database connectivity using an approved diagnostic path, confirm the application can perform its expected queries, and run a controlled synthetic payment test using a dedicated nonfinancial test transaction or payment-provider sandbox.

Recovery criteria

Students must demonstrate all five checks

Payment API can establish PostgreSQL connections

Database queries succeed without exposing sensitive records

Customer-facing 5xx rate returns to normal

Payment success rate returns to expected baseline

No unintended duplicate payments or unresolved transaction states

0 of 5 recovery checks completed

Remember: A green Grafana dashboard alone does not establish that customer transactions are correct.

Task 3 — Write the post-incident report

Postmortem — INC-2026-1047

Incident summary

Payment authorization was degraded following a production Terraform networking change.

Customer impact

The illustrative payment success rate declined from 99.95% to 41%, with elevated HTTP 5xx errors and latency.

Technical root cause

An infrastructure refactor changed the source security group allowed to reach RDS PostgreSQL on TCP port 5432. Application Pods remained Ready because their readiness checks did not verify database-dependent business functionality.

Contributing factors

Terraform validation confirmed configuration syntax but did not test application-to-database connectivity.
The CI/CD pipeline lacked a post-apply payment workflow smoke test.
The alerting design detected payment failures but did not immediately identify the affected network dependency.
The readiness probe was insufficient as an indicator of end-to-end payment health.

Mitigation

The team restored the approved database ingress rule and verified application connectivity, transaction processing, and recovery metrics.

Preventive actions

Introduce security group policy checks, infrastructure change review, post-deployment connectivity testing, synthetic payment monitoring, and clearer incident runbooks.

Who owns the preventive actions?

Action	Owner	Verification
Restore and review security group configuration	Platform engineer	Terraform plan and AWS configuration
Add network policy tests to CI/CD	Platform / security	Automated policy checks
Add post-apply connectivity smoke test	DevOps / SRE	Successful synthetic check
Improve payment error metrics	Backend team	Grafana dashboard and alerts
Improve transaction reconciliation	Backend / database	Controlled recovery tests
Update incident runbook	SRE	Reviewed incident exercise
Review production change approvals	Release engineering	Change audit

Practical hands-on lab — Build the failure safely

Students, you will reproduce the application-to-database connectivity failure in your own training EKS cluster.

You will not modify an actual bank, customer database, or production security group.

The safest classroom version uses a PostgreSQL Pod inside a dedicated namespace and a Kubernetes NetworkPolicy to block connectivity. This reproduces the symptom and troubleshooting workflow without changing live AWS infrastructure.

For the Terraform portion, you will separately review an example security group change and its plan. An optional isolated AWS RDS exercise can be added later when your instructor has prepared a dedicated account, budget, and recovery procedure.

Lab architecture

Step 1 — Create an isolated namespace

kubectl create namespace payments-incident-lab

Step 2 — Deploy PostgreSQL

This database is temporary and contains no real customer data.


apiVersion: v1
kind: Secret
metadata:
  name: lab-postgres-secret
  namespace: payments-incident-lab
type: Opaque
stringData:
  POSTGRES_PASSWORD: "lab-only-change-me"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lab-postgres
  namespace: payments-incident-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: lab-postgres
  template:
    metadata:
      labels:
        app: lab-postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: payments
            - name: POSTGRES_USER
              value: labuser
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: lab-postgres-secret
                  key: POSTGRES_PASSWORD
          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - pg_isready -U labuser -d payments
            periodSeconds: 5
          resources:
            requests:
              cpu: 50m
              memory: 128Mi
            limits:
              memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
  name: lab-postgres
  namespace: payments-incident-lab
spec:
  selector:
    app: lab-postgres
  ports:
    - port: 5432
      targetPort: 5432

Save this as postgres-lab.yaml and apply:

kubectl apply -f postgres-lab.yaml

kubectl rollout status deployment/lab-postgres \
  -n payments-incident-lab

This lab uses ephemeral PostgreSQL storage. Data is lost when the database Pod is replaced. That is acceptable for a disposable connectivity exercise but not for a production database.

Step 3 — Verify normal connectivity

Use a temporary PostgreSQL client Pod.

kubectl run payment-db-client \
  -n payments-incident-lab \
  --image=postgres:16 \
  --restart=Never \
  --env="PGPASSWORD=lab-only-change-me" \
  --command -- sleep 3600

Wait until it is Running:

kubectl wait --for=condition=Ready \
  pod/payment-db-client \
  -n payments-incident-lab \
  --timeout=120s

Run a database query:

kubectl exec -n payments-incident-lab \
  payment-db-client -- \
  psql -h lab-postgres \
  -U labuser -d payments \
  -c "SELECT 1;"

Expected:

 ?column?
----------
        1
(1 row)

This proves that the client can reach PostgreSQL before the incident is introduced.

The example password is deliberately disposable. Never use real database credentials in shared class terminals or recorded commands.

Step 4 — Inject the incident

Apply a NetworkPolicy that selects the PostgreSQL Pod and allows no incoming connections.


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-postgres-ingress
  namespace: payments-incident-lab
spec:
  podSelector:
    matchLabels:
      app: lab-postgres
  policyTypes:
    - Ingress
  ingress: []

Save as block-postgres.yaml:

kubectl apply -f block-postgres.yaml

Prerequisite: The cluster must have a network policy implementation that enforces Kubernetes NetworkPolicy, such as a properly configured Amazon VPC CNI network policy agent or another compatible CNI. Otherwise, the policy may exist without actually blocking traffic.

Test again with a timeout:

kubectl exec -n payments-incident-lab \
  payment-db-client -- \
  sh -c 'PGCONNECT_TIMEOUT=5 psql -h lab-postgres -U labuser -d payments -c "SELECT 1;"'

Expected on a correctly enforcing cluster:

connection to server at "lab-postgres" failed:
Connection timed out

Students should now investigate why PostgreSQL is Running and Ready but the client cannot connect.

kubectl get pods -n payments-incident-lab

kubectl get svc -n payments-incident-lab

kubectl get networkpolicy -n payments-incident-lab

kubectl describe networkpolicy block-postgres-ingress \
  -n payments-incident-lab

Student challenge

You are not told that the instructor introduced a NetworkPolicy.

Your job is to identify the exact resource that changed and explain how it prevents database access.

Step 5 — Restore service

Once you have identified and documented the cause, remove the injected policy:

kubectl delete networkpolicy block-postgres-ingress \
  -n payments-incident-lab

Test again:

kubectl exec -n payments-incident-lab \
  payment-db-client -- \
  psql -h lab-postgres \
  -U labuser -d payments \
  -c "SELECT 1;"

Expected: SELECT 1 succeeds.

Clean up the client:

kubectl delete pod payment-db-client \
  -n payments-incident-lab

When the entire lab is complete, the instructor can delete the dedicated namespace.

Important difference between the lab and production

The production case study involves an AWS security group changed by Terraform. The hands-on exercise uses Kubernetes NetworkPolicy to reproduce blocked database traffic safely.

Both teach network connectivity troubleshooting, but they are different control planes. Students must be able to explain that distinction during an interview.

Also, this initial lab tests connectivity directly; it does not yet deploy a complete payment API, transaction simulator, or the custom payment metrics used in the fictional dashboard. Those would be additional components if the class needs a fully instrumented end-to-end payment workflow.

Advanced interview question bank

Practice the following questions without looking at the answers first.

Incident response

1 / 17

You receive a payment failure alert. What are your first five actions?

Show model answerFollow-up

PreviousNext

Final mock interview: Tell me about a critical production incident

Students, your final task is to explain the entire incident in approximately three minutes. This is a model answer for discussing the case study. Adapt it to your actual experience rather than claiming to have personally resolved an incident you only practiced.

Production Incident — Payment API Database Connectivity Failure

One complex production incident scenario I worked through involved a customer-facing payment processing application running on AWS EKS with PostgreSQL on Amazon RDS.

The environment used Terraform for infrastructure management, GitHub Actions for CI/CD, and Prometheus, Grafana, CloudWatch, and PagerDuty for monitoring and alerting.

The incident began when the payment success rate dropped significantly, from approximately 99.95% to 41%. Customers were experiencing payment failures and HTTP 5xx errors.

My first step would be to acknowledge the PagerDuty alert and investigate the customer impact through Grafana. I would check the error rate, latency, request volume, and Kubernetes availability.

Interestingly, all 12 Payment API Pods were Running and Ready, and the RDS database itself reported Available.

This indicated that the problem might be outside basic Kubernetes Pod health.

I would inspect the application logs, where I would find PostgreSQL connection timeouts. I would then work with the database and cloud platform teams to investigate database health and the network path from EKS to RDS.

I would check DNS resolution, TCP connectivity on port 5432, security group rules, and any relevant Kubernetes NetworkPolicies.

At the same time, I would review recent production changes. The release engineering team would identify a successful Terraform infrastructure deployment shortly before the errors started.

By comparing the Terraform change with the previous known-good configuration, we would identify that the RDS security group ingress rule had been changed to reference the wrong source security group.

The database itself was operational, but the application could no longer reliably establish new database connections.

I would coordinate with the incident commander and platform team to restore the previously approved security group rule using the authorized emergency change procedure.

After the correction, I would verify connectivity from the application network, confirm successful database operations, and monitor payment success rate, HTTP errors, latency, and transaction reconciliation.

I would not declare recovery based only on Kubernetes Pod status. I would verify that the customer-facing payment workflow was functioning correctly.

Following recovery, I would participate in the postmortem and recommend stronger Terraform policy checks, post-deployment connectivity testing, synthetic payment monitoring, and improved incident runbooks.

The main lesson from this incident is that healthy Kubernetes Pods and a healthy database do not necessarily mean the complete application is healthy. As a DevOps engineer, I need to investigate the full request path and work across infrastructure, application, database, and security teams to restore service safely.

Final evaluation rubric

Use this scorecard to assess each student during Class 3.

Mock interview score

Adjust scores as the student answers

0 / 100

Needs more practice

Incident impact and first response

0 / 15

Kubernetes investigation

0 / 15

Database and networking diagnosis

0 / 20

Terraform and CI/CD investigation

0 / 20

Mitigation and recovery verification

0 / 15

Communication and postmortem

0 / 15

Reset for next student

What interviewers are really evaluating

This incident is especially useful for DevOps Engineer, Site Reliability Engineer, Platform Engineer, Cloud Infrastructure Engineer, and DevSecOps interviews because it tests several competencies together:

Kubernetes: Understanding that Pod readiness is not the same as business functionality.
AWS: Knowing how VPC networking, security groups, EKS, and RDS interact.
Terraform: Understanding change management, infrastructure drift, and safe recovery.
CI/CD: Recognizing that a successful pipeline does not guarantee a successful production outcome.
Database: Knowing how to distinguish connectivity, authentication, performance, and availability failures.
Monitoring: Using technical and business metrics together.
Incident management: Communicating, escalating, mitigating, verifying, and documenting.

Students should be able to explain not just which command they would run, but what question that command answers.

Your final assignment: Explain why the Payment API could have 12 healthy Kubernetes Pods, an Available PostgreSQL RDS instance, and a successful Terraform deployment while customer payments were failing—and describe the evidence needed to identify and fix the real cause.

That is the central engineering challenge for these three classes.

JumpToTech DevOps Bootcamp Kubernetes Production Support Lab — From Deployment to HTTP Request Troubleshooting

Aisalkyn Aidarova — Thu, 29 Oct 2026 15:48:47 +0000

Level: Beginner to Intermediate\
Role: DevOps Engineer / Cloud Engineer / Site Reliability Engineer\
Project: Restaurant Company\
Environment: Kubernetes (Amazon EKS, kind, or minikube)\
Estimated time: 3–4 hours\
Assignment type: Hands-on Lab + Simulated Production Incident + Interview Preparation

PART 1 — YOUR ROLE AS A DEVOPS ENGINEER

Welcome to Restaurant Company

Imagine you have recently joined Restaurant Company as a DevOps Engineer.

Restaurant Company has a web application that customers use to view restaurant information and place orders.

The application runs in Kubernetes.

Your responsibility is to help ensure that the application is:

Available to customers.
Running on healthy Kubernetes Pods.
Accessible through the correct network routes.
Deployed reliably through CI/CD pipelines.
Monitored for failures.
Recoverable when incidents occur.

You are also responsible for communicating with developers, other DevOps engineers, SREs, engineering managers, and incident responders.

Your job is not simply to execute commands. Your job is to investigate problems, understand the impact, restore service safely, and communicate clearly.

1.1 Tools used by your company

For this simulated environment, assume your company uses:

Tool	Purpose
GitHub	Application source code
GitHub Actions	CI/CD automation
Docker	Container image packaging
Amazon ECR	Container image registry
Amazon EKS	Managed Kubernetes
kubectl	Kubernetes administration
Jira / ServiceNow	Incident tickets
Slack / Microsoft Teams	Team communication
Prometheus	Application and infrastructure metrics
Grafana	Monitoring dashboards
Amazon CloudWatch	AWS monitoring and logs
CoreDNS	Internal Kubernetes DNS
AWS ALB	External HTTP/HTTPS routing

Different companies may use different tools. The concepts remain similar.

PART 2 — UNDERSTAND THE APPLICATION ARCHITECTURE

Before troubleshooting production issues, you must understand how the application reaches Kubernetes.

2.1 How an application is deployed

A developer writes application code and pushes it to GitHub.

A typical CI/CD workflow then performs the following steps:

Developer pushes code to a GitHub repository.
GitHub Actions starts a CI/CD pipeline.
The pipeline checks out the source code.
The pipeline installs dependencies and runs tests.
Docker builds a container image.
The image is pushed to Amazon ECR.
The deployment workflow updates the Kubernetes Deployment to use the new image.
The Kubernetes API Server accepts the updated desired state.
The Deployment Controller manages the ReplicaSet.
The ReplicaSet Controller maintains the required number of Pods.
The Scheduler selects suitable Worker Nodes for new Pods.
kubelet on each assigned Worker Node works with the container runtime to start the containers.
The CNI implementation configures Pod networking.
Readiness probes determine whether the application Pods are ready to receive traffic.
Kubernetes Service endpoints are updated to reflect eligible backends.

Important: This is the typical production deployment architecture. In this lab, we will create Kubernetes resources manually using YAML and kubectl. We are not building a complete GitHub Actions/ECR pipeline today.

2.2 Kubernetes component responsibilities

API Server

The API Server is the entry point for Kubernetes API requests. When you execute kubectl apply, kubectl sends the requested configuration to the API Server.

etcd

etcd stores Kubernetes cluster state, including the desired state of Kubernetes resources.

Deployment Controller

The Deployment Controller manages ReplicaSets and application rollouts.

ReplicaSet Controller

The ReplicaSet Controller ensures that the desired number of Pods exists.

Scheduler

The Scheduler selects an appropriate Worker Node for a Pod that has not yet been assigned to a Node.

kubelet

kubelet is the agent running on a Kubernetes Node. It works with the container runtime to maintain assigned Pods.

Container Runtime

The container runtime, such as containerd, manages container image pulling and container execution.

Linux Kernel

The Linux Kernel provides the underlying operating system functionality, including processes, networking, namespaces, cgroups, and resource management.

CNI

The Container Network Interface implementation configures Pod networking. Depending on the CNI plugin, this includes interfaces, IP allocation, and routes.

CoreDNS

CoreDNS resolves Kubernetes Service DNS names into their corresponding Service IP addresses.

EndpointSlices

EndpointSlices publish backend endpoint information associated with Kubernetes Services.

kube-proxy

In clusters using kube-proxy, it watches Services and EndpointSlices and configures the Service networking dataplane. In iptables mode, Linux Netfilter processes packets according to the programmed rules.

Other Kubernetes clusters may use different networking implementations, such as eBPF-based Service routing.

PART 3 — YOUR FIRST PRODUCTION TICKET

Incident Ticket: INC-2026-1042

Title: Restaurant Backend Service Unavailable\
Priority: P2 — High (simulated)\
Environment: Kubernetes\
Assigned Team: DevOps / Platform Engineering\
Status: Open — Investigating

Incident description

Customers are reporting that the Restaurant application cannot retrieve backend information.

Monitoring indicates that application requests are failing.

The Kubernetes Deployment appears healthy, but requests to the backend Service are unsuccessful.

Business impact

Customers may be unable to view restaurant information or complete certain application workflows.

Your task is to investigate the issue, identify the root cause, restore connectivity, and document the resolution.

3.1 Where will you receive the ticket?

In a real company, incidents may be reported through:

Jira or ServiceNow.
PagerDuty or another on-call alerting system.
Grafana or Prometheus alerts.
CloudWatch alarms.
Slack or Microsoft Teams.
Customer support escalation.
Developer reports.

For example:

A monitoring system detects failed HTTP requests and creates an alert.

The alert reaches the on-call engineer.

The engineer acknowledges the alert, reviews the impact, and opens or updates the incident ticket.

The incident is then investigated according to the company's incident-management procedures.

3.2 What should you do first?

Before executing troubleshooting commands:

Acknowledge the ticket.
Identify the affected application and environment.
Determine the reported symptoms and business impact.
Check whether there is an active incident bridge or incident commander.
Review recent deployments and infrastructure changes.
Start with safe, read-only diagnostic commands.
Communicate your investigation status.

Do not immediately restart applications, delete Pods, or modify production configuration without understanding the situation and following company procedures.

3.3 Your first Slack message

Example:

Hi team,

I have acknowledged INC-2026-1042 regarding failed requests to the Restaurant backend service.

I am reviewing the Kubernetes Deployment, Pod readiness, Service configuration, and backend endpoints to determine the cause.

I will provide another update within 10 minutes.

Please let me know if there were any recent application deployments or infrastructure changes.

Thank you.

3.4 What information should you collect?

Before troubleshooting, identify:

Application name.
Namespace.
Cluster.
Environment.
Time the issue started.
User-facing symptoms.
Error codes, if available.
Recent deployments.
Monitoring alerts.
Relevant logs.
Current incident owner.

For our lab:

Application: Restaurant Company\
Namespace: jumptotech-lab\
Deployment: restaurant\
Service: restaurant-service\
Port: 80\
Replicas: 2

PART 4 — BUILD YOUR KUBERNETES ENVIRONMENT FROM ZERO

Step 1 — Verify Kubernetes access

Open your terminal.

If you are using Amazon EKS, make sure your AWS credentials, cluster access, and kubeconfig are configured.

If you are using kind or minikube, ensure your local cluster is running.

Execute:

kubectl cluster-info
kubectl get nodes -o wide
kubectl config current-context

Expected result:

Kubernetes API Server is reachable.
At least one Node is Ready.
Your current context points to the intended lab cluster.

Important: Never assume you are connected to the correct cluster. Always verify your Kubernetes context before making changes.

If you receive a connection error, stop and troubleshoot cluster access before continuing.

Step 2 — Create your project directory

mkdir -p ~/jumptotech-k8s-network-lab
cd ~/jumptotech-k8s-network-lab
pwd

Step 3 — Create a Namespace

kubectl create namespace jumptotech-lab

Verify:

kubectl get namespaces

Interview question: Why do we use Kubernetes Namespaces?

Answer:

Namespaces provide logical separation of namespaced Kubernetes resources within a cluster. Teams commonly use them to organize applications and environments, apply access controls, and manage resource quotas.

Namespaces alone are not a complete security boundary.

PART 5 — CREATE THE RESTAURANT DEPLOYMENT

Step 4 — Create deployment.yaml

Execute:

cat > deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: restaurant
  namespace: jumptotech-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: restaurant
  template:
    metadata:
      labels:
        app: restaurant
    spec:
      containers:
      - name: nginx
        image: nginx:1.27
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "50m"
            memory: "64Mi"
          limits:
            cpu: "250m"
            memory: "128Mi"
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 2
          periodSeconds: 5
EOF

Step 5 — Apply the Deployment

kubectl apply -f deployment.yaml

Check rollout status:

kubectl rollout status deployment/restaurant \
  -n jumptotech-lab --timeout=180s

Check resources:

kubectl get deployments -n jumptotech-lab

kubectl get replicasets -n jumptotech-lab

kubectl get pods -n jumptotech-lab -o wide

Expected result:

Deployment exists.
ReplicaSet exists.
Two Restaurant Pods are Running and Ready.

If the Pods are not Ready, do not continue blindly.

Investigate:

kubectl describe deployment restaurant \
  -n jumptotech-lab

kubectl describe pods \
  -n jumptotech-lab -l app=restaurant

kubectl get events -n jumptotech-lab \
  --sort-by=.lastTimestamp

Step 6 — Understand what happened

When you executed kubectl apply -f deployment.yaml, you submitted the desired application configuration to Kubernetes.

The API Server processed the request.

The Deployment Controller managed the ReplicaSet.

The ReplicaSet Controller created the required Pods.

The Scheduler selected Worker Nodes.

kubelet worked with the container runtime to start the containers.

The CNI implementation configured networking for the Pods.

Student questions:

What is a Deployment?
What is a ReplicaSet?
What is a Pod?
Who selects the Worker Node?
Who starts the container?
What is the role of the Linux Kernel?
Why does the Pod need a network interface and IP address?

Be prepared to explain every answer without reading your notes.

PART 6 — CREATE THE KUBERNETES SERVICE

Step 7 — Create service.yaml

cat > service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: restaurant-service
  namespace: jumptotech-lab
spec:
  type: ClusterIP
  selector:
    app: restaurant
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80
EOF

Apply:

kubectl apply -f service.yaml

Verify:

kubectl get svc -n jumptotech-lab

kubectl describe svc restaurant-service \
  -n jumptotech-lab

Check EndpointSlices:

kubectl get endpointslices \
  -n jumptotech-lab \
  -l kubernetes.io/service-name=restaurant-service \
  -o wide

Step 8 — Understand Service and Pod labels

Your Service has this selector:

selector:
  app: restaurant

Your Pods have this label:

labels:
  app: restaurant

The matching labels allow the EndpointSlice Controller to publish the correct backend endpoints.

The Service provides a stable virtual IP address, called a ClusterIP.

Pods can be replaced and receive new IP addresses, while the Service can retain its ClusterIP.

Interview question: Why do we need a Service if Pods already have IP addresses?

Answer:

Pod IP addresses may change when Pods are recreated. A Kubernetes Service provides a stable endpoint and Service discovery mechanism for reaching eligible backend Pods.

PART 7 — CREATE A DIAGNOSTIC CLIENT POD

Step 9 — Create client.yaml

cat > client.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: client
  namespace: jumptotech-lab
spec:
  containers:
  - name: netshoot
    image: nicolaka/netshoot:latest
    command: ["sleep", "36000"]
    resources:
      requests:
        cpu: "25m"
        memory: "32Mi"
      limits:
        cpu: "200m"
        memory: "128Mi"
EOF

Apply:

kubectl apply -f client.yaml

Wait:

kubectl wait \
  --for=condition=Ready \
  pod/client \
  -n jumptotech-lab \
  --timeout=180s

Enter the Pod:

kubectl exec -it client \
  -n jumptotech-lab -- bash

Your terminal prompt should change to something similar to:

client:~#

Important: You are now inside the Client Pod, not on your EC2 management host.

PART 8 — INVESTIGATE POD NETWORKING

Step 10 — Check the Pod hostname

Inside Client Pod:

hostname

Step 11 — Check network interfaces

ip addr

Find the eth0 interface and its IPv4 address.

Record the IP address.

Step 12 — Check routing

ip route

Look for the default route.

In AWS VPC CNI, you may see a gateway such as:

default via 169.254.1.1 dev eth0

Other Kubernetes network implementations may have different routing configurations.

Step 13 — Check DNS configuration

cat /etc/resolv.conf

Identify the configured DNS server and search domains.

Interview question: What is a Linux Network Namespace?

Answer:

A Network Namespace provides an isolated view of networking resources, including network interfaces, IP addresses, routing tables, and ports.

Containers in the same Kubernetes Pod normally share one Network Namespace.

Interview question: What does CNI do?

Answer:

The CNI implementation configures networking for Pods. Depending on the plugin, this includes network interfaces, IP addresses, and routing.

PART 9 — TRACE AN HTTP REQUEST

Step 14 — DNS resolution

Still inside Client Pod:

nslookup restaurant-service

Then:

nslookup restaurant-service.jumptotech-lab.svc.cluster.local

Expected:

The DNS lookup returns the ClusterIP assigned to restaurant-service.

Record the IP address.

What happened?

The Client Pod's resolver queried cluster DNS.

CoreDNS resolved the Kubernetes Service name.

The DNS response provided the Service IP address.

CoreDNS does not forward the HTTP request itself.

Step 15 — Send HTTP request

curl -v http://restaurant-service

Then:

curl -I http://restaurant-service

Expected HTTP status:

HTTP/1.1 200 OK

The default NGINX welcome page should be returned.

Step 16 — Understand the complete internal request flow

Client Pod initiates an HTTP request.
DNS resolves restaurant-service to its ClusterIP.
The client establishes a TCP connection to the Service IP on port 80.
The Kubernetes Service dataplane selects an eligible backend Pod.
In iptables-based kube-proxy mode, Linux Netfilter applies Service forwarding and address translation rules.
The packet reaches the selected Restaurant Pod.
NGINX receives the HTTP request on port 80.
NGINX returns the HTTP response.
The response travels back to the Client Pod.

Important distinction:

kube-proxy programs forwarding rules in its supported modes. In iptables mode, the Linux Kernel processes packets using those rules.

The exact implementation depends on the cluster networking configuration.

Step 17 — Compare direct Pod IP requests

Exit the Client Pod:

exit

Find the current Restaurant Pod IP addresses:

kubectl get pods -n jumptotech-lab \
  -l app=restaurant -o wide

Record both Pod IPs.

Enter Client Pod again:

kubectl exec -it client \
  -n jumptotech-lab -- bash

Run the following, replacing the placeholders with the real IP addresses:

curl -I http://POD_1_IP:80
curl -I http://POD_2_IP:80

For example, if a Pod IP is 172.31.46.36:

curl -I http://172.31.46.36:80

Interview question: What is the difference between a Service request and a direct Pod IP request?

Answer:

A Service request uses a stable Service endpoint and the cluster's Service routing implementation to reach an eligible backend.

A direct Pod IP request targets one specific Pod and bypasses Service load balancing.

PART 10 — REPRODUCE THE PRODUCTION INCIDENT

Now we will simulate a real Kubernetes networking incident.

We will intentionally create a Service with an incorrect selector.

We will not modify the healthy Restaurant Service.

Step 18 — Exit Client Pod

exit

Step 19 — Create a broken Service

cat > broken-service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: restaurant-broken
  namespace: jumptotech-lab
spec:
  type: ClusterIP
  selector:
    app: wrong-restaurant
  ports:
  - name: http
    port: 80
    targetPort: 80
EOF

Apply:

kubectl apply -f broken-service.yaml

Step 20 — Investigate the broken Service

Check:

kubectl get svc -n jumptotech-lab

Describe:

kubectl describe svc restaurant-broken \
  -n jumptotech-lab

Check Pod labels:

kubectl get pods -n jumptotech-lab \
  --show-labels

Check EndpointSlices:

kubectl get endpointslices \
  -n jumptotech-lab \
  -l kubernetes.io/service-name=restaurant-broken \
  -o wide

Observation:

The Service selector uses app: wrong-restaurant.

The Restaurant Pods use app: restaurant.

Because these values do not match, the Service has no eligible backend endpoints.

Step 21 — Reproduce the failed HTTP request

Test DNS from Client Pod:

kubectl exec client -n jumptotech-lab -- \
  nslookup restaurant-broken

The DNS name should resolve to the broken Service's ClusterIP.

Now test HTTP:

kubectl exec client -n jumptotech-lab -- \
  curl -v --max-time 5 http://restaurant-broken

The request should fail because the Service has no eligible backend endpoints.

The exact failure may be connection refused, timeout, or another network-level error depending on the cluster dataplane.

Do not automatically describe this as HTTP 503. A Service without endpoints does not necessarily return an HTTP response.

Step 22 — Identify the root cause

Compare:

Service selector:

app: wrong-restaurant

Pod label:

app: restaurant

Root cause:

The Kubernetes Service selector did not match the Restaurant Pod labels.

As a result, no eligible backend endpoints were available for that Service.

The application Pods were healthy, but traffic could not reach them through the broken Service.

PART 11 — COMMUNICATE YOUR FINDINGS

Before applying the fix, write an investigation update.

Example Slack message:

Hi team,

I identified a likely root cause for INC-2026-1042.

The Restaurant application Pods are Running and Ready, but the affected Kubernetes Service has no eligible backend endpoints.

The Service selector is configured with app: wrong-restaurant, while the application Pods use app: restaurant.

I reproduced the connectivity failure from a diagnostic Pod.

I am preparing a selector correction and will verify the backend endpoints and HTTP response after the change.

PART 12 — FIX THE INCIDENT

Step 23 — Correct the Service selector

For our isolated lab, execute:

kubectl patch svc restaurant-broken \
  -n jumptotech-lab \
  -p '{"spec":{"selector":{"app":"restaurant"}}}'

In real production, follow change authorization and GitOps procedures. If the Service is managed through Git, Helm, or Argo CD, update the source of truth to prevent the incorrect configuration from returning.

Step 24 — Verify the fix

kubectl describe svc restaurant-broken \
  -n jumptotech-lab

Check EndpointSlices:

kubectl get endpointslices \
  -n jumptotech-lab \
  -l kubernetes.io/service-name=restaurant-broken \
  -o wide

Confirm that backend Pod IPs appear.

Then:

kubectl exec client -n jumptotech-lab -- \
  curl -I --max-time 5 http://restaurant-broken

Expected:

HTTP/1.1 200 OK

Step 25 — Verify the original healthy Service

kubectl exec client -n jumptotech-lab -- \
  curl -I --max-time 5 http://restaurant-service

Expected:

HTTP/1.1 200 OK

For a real production incident, one successful curl is not enough to establish full recovery.

You should also verify monitoring metrics, error rates, readiness, backend health, and the actual user-facing path.

PART 13 — RESOLUTION COMMUNICATION

After verifying recovery, write a resolution update.

Example Slack message:

Hi team,

INC-2026-1042 has been resolved in our lab environment.

Root cause: The Kubernetes Service selector did not match the application Pod labels, leaving the Service without eligible backend endpoints.

Actions taken:

Confirmed the application Pods were Running and Ready.
Reviewed Service configuration and Pod labels.
Inspected EndpointSlices.
Reproduced the failed HTTP request from a diagnostic Pod.
Corrected the Service selector.
Verified backend endpoints were populated.
Confirmed HTTP 200 responses after the fix.

I recommend adding automated validation for Service selectors and backend endpoint availability to help prevent similar configuration errors.

I will document the investigation and resolution in the incident ticket.

Important: Only use this resolution message after completing and verifying the corresponding actions.

PART 14 — COMPLETE YOUR INCIDENT TICKET

Incident Report

Incident ID: INC-2026-1042

Application: Restaurant Company

Environment: Kubernetes Lab

Issue: Backend Service connectivity failure.

Impact: Simulated inability to reach the backend application through the affected Service.

Detection: Simulated monitoring alert and failed HTTP connectivity test.

Investigation:

Reviewed Kubernetes Deployment, Pod status, Service configuration, Pod labels, and EndpointSlices.

Confirmed the application Pods were Running and Ready.

Observed that the affected Service did not have eligible backend endpoints.

Root Cause:

Service selector mismatch.

The Service used app: wrong-restaurant, while the Pods used app: restaurant.

Resolution:

Updated the Service selector to match the Pod labels.

Verification:

Confirmed backend endpoints appeared and the Service returned HTTP 200 from the diagnostic Client Pod.

Preventive Actions:

Add deployment validation for Service selectors.
Monitor Service backend availability.
Review Kubernetes manifests before deployment.
Use GitOps and peer-reviewed changes where applicable.
Maintain clear incident runbooks.

Status: Resolved — after successful verification.

PART 15 — BONUS LAB: KUBERNETES SELF-HEALING

This lab demonstrates how Kubernetes maintains the desired number of Pods.

Step 26 — Check existing Pods

kubectl get pods -n jumptotech-lab \
  -l app=restaurant

Step 27 — Delete one Pod

Choose an actual Restaurant Pod name.

kubectl delete pod POD_NAME \
  -n jumptotech-lab

Replace POD_NAME with the real name.

Step 28 — Watch Kubernetes create a replacement

kubectl get pods -n jumptotech-lab -w

Press Ctrl+C when the new Pod becomes Ready.

Step 29 — Verify replicas

kubectl get deployment restaurant \
  -n jumptotech-lab

Expected: Two Ready replicas after recovery.

Interview question: What happens when a Pod is deleted?

Answer:

The ReplicaSet Controller detects that the number of existing Pods is below the desired replica count and creates a replacement Pod.

The Scheduler assigns the new Pod to a suitable Node, and kubelet starts its containers.

This is an example of Kubernetes reconciliation and self-healing.

PART 16 — BONUS LAB: ACCESS THE APPLICATION FROM A BROWSER

From the machine where kubectl is configured:

kubectl port-forward \
  -n jumptotech-lab \
  svc/restaurant-service \
  8080:80

Open:

http://localhost:8080

Expected: NGINX welcome page.

If kubectl is running on a remote EC2 instance, localhost:8080 refers to that EC2 instance, not your laptop. Use an approved SSH tunnel or another authorized access method if you need to view it locally.

Important: Port-forwarding is a diagnostic tool. It does not demonstrate the production AWS ALB and Ingress request path.

PART 17 — UNDERSTAND THE EXTERNAL HTTP REQUEST FLOW

A real customer usually does not access the application's internal Kubernetes Service name.

Consider a production architecture using AWS ALB.

A typical request flow is:

Customer opens the Restaurant Company website.
The browser resolves the public domain through DNS.
The browser connects to the public application endpoint, commonly over HTTPS.
AWS ALB receives the request.
ALB evaluates listener and routing rules.
The request is forwarded to a configured target group.
The target group sends the request to a healthy backend target.
The application processes the request.
The HTTP response returns to the browser.

With AWS Load Balancer Controller, ALB can target Pod IPs directly in IP target mode or NodePorts in instance target mode.

For a separate internal backend call, a frontend server Pod may resolve a Kubernetes Service name using CoreDNS and send traffic through the internal Service dataplane.

A browser-hosted React application may instead call a public backend API through ALB.

You must know which architecture your application actually uses.

Important HTTP status codes

Code	Meaning	What DevOps investigates
200	OK	Successful response
201	Created	Resource creation succeeded
301 / 302	Redirect	Redirect configuration
400	Bad Request	Invalid request or API validation
401	Unauthorized	Authentication
403	Forbidden	Authorization
404	Not Found	Missing route or resource
500	Internal Server Error	Application/backend failure
502	Bad Gateway	Gateway or upstream response failure
503	Service Unavailable	Backend availability or overload
504	Gateway Timeout	Upstream timeout

Important: A 404 does not necessarily mean the entire application is unavailable. A 400 does not automatically mean the application is down. HTTP status codes must be interpreted in context.

PART 18 — HOW TO TROUBLESHOOT DIFFERENT ERRORS

Scenario A — Pod is Pending

Check:

kubectl describe pod POD_NAME -n jumptotech-lab

Investigate:

Insufficient CPU or memory.
Scheduling constraints.
Node availability.
Taints and tolerations.
Persistent volume requirements.

Scenario B — ImagePullBackOff

Check:

kubectl describe pod POD_NAME -n jumptotech-lab

Investigate:

Incorrect image name.
Missing image tag.
Registry access.
Image pull permissions.
Registry connectivity.

Scenario C — CrashLoopBackOff

Check:

kubectl logs POD_NAME \
  -n jumptotech-lab --previous

Also inspect Pod events and container exit codes.

Investigate:

Application startup failure.
Incorrect configuration.
Missing dependencies.
Resource limits.
Probe failures.

Scenario D — Service DNS resolution fails

Check from a diagnostic Pod:

nslookup restaurant-service

Investigate:

Service existence.
Namespace and DNS name.
CoreDNS health.
Pod DNS configuration.
DNS connectivity and network policies.

Scenario E — DNS works but HTTP fails

Check:

kubectl get svc -n jumptotech-lab

kubectl get endpointslices \
  -n jumptotech-lab

kubectl get pods -n jumptotech-lab \
  --show-labels

Investigate:

Service selector.
Pod readiness.
Target port.
Application listener.
Network policies.
Service dataplane behavior.

Scenario F — External users receive 503

Investigate:

AWS ALB target health.
Ingress configuration.
Target groups.
Service and endpoint availability.
Pod readiness.
Application health checks.

The investigation depends on which component generated the 503.

PART 19 — HOW TO ANSWER IN A DEVOPS INTERVIEW

Interview Question 1: Tell me about yourself

Sample answer:

I am building hands-on experience in DevOps and cloud infrastructure, focusing on Linux, AWS, Docker, Kubernetes, Terraform, and CI/CD automation.

Through my Restaurant Company project, I have practiced deploying containerized applications to Kubernetes, configuring Deployments and Services, troubleshooting Pod networking, and investigating application connectivity issues.

I have also worked on simulated production incidents, where I followed a structured process to identify problems, verify fixes, and communicate technical findings.

I am especially interested in cloud infrastructure automation, Kubernetes operations, and production reliability.

Interview Question 2: Explain how an application gets deployed to Kubernetes

Sample answer:

In a typical CI/CD workflow, developers push their code to GitHub, which triggers an automated pipeline.

The pipeline builds and tests the application, creates a Docker image, and pushes it to a container registry such as Amazon ECR.

The deployment process updates the Kubernetes Deployment with the new image version.

The Kubernetes API Server accepts the desired configuration, and the Deployment Controller manages the rollout through ReplicaSets.

The Scheduler assigns new Pods to suitable Worker Nodes, and kubelet works with the container runtime to start the containers.

The CNI implementation configures Pod networking, and readiness probes help determine when the application is ready to receive traffic.

Interview Question 3: Explain how an HTTP request reaches a Kubernetes application

Sample answer:

For an external request, the client first resolves the public domain through DNS and connects to the application's public endpoint.

In an AWS environment, the request may reach an Application Load Balancer.

The ALB evaluates its listener rules and forwards traffic to a healthy backend target.

For internal communication between Kubernetes applications, a client Pod can use the backend Service DNS name.

CoreDNS resolves the Service name to its ClusterIP.

The Kubernetes Service dataplane then forwards the connection to an eligible backend Pod.

The application processes the HTTP request and sends the response back to the client.

Interview Question 4: Tell me about a Kubernetes incident you resolved

Sample answer:

During my DevOps training, I investigated a simulated production incident involving a Restaurant Company application running in Kubernetes.

The application Pods were healthy, but HTTP requests through a Kubernetes Service were failing.

I started by reviewing Pod readiness, the Service configuration, Pod labels, and EndpointSlices.

I reproduced the connectivity failure from a diagnostic Client Pod.

DNS resolution was successful, which helped me separate DNS issues from Service routing problems.

I found that the Service selector did not match the application Pod labels, leaving the Service without eligible backend endpoints.

I corrected the selector, verified that the expected endpoints appeared, and confirmed the application returned HTTP 200.

I also practiced communicating the incident status, documenting the root cause, and recommending preventive checks.

This exercise helped me understand how Kubernetes networking works and how to approach connectivity issues systematically.

Interview Question 5: How do you troubleshoot a Kubernetes application that is unavailable?

Sample answer:

I first confirm the scope and impact of the issue.

I check monitoring alerts and determine whether the problem affects all users or only specific requests.

Then I inspect the Kubernetes Deployment, Pod readiness, events, and application logs.

If the Pods are healthy, I investigate Service configuration, EndpointSlices, DNS resolution, and network connectivity.

For externally exposed applications, I also check Ingress configuration, load balancer target health, and relevant AWS networking components.

Once I identify the likely root cause, I coordinate the appropriate mitigation, verify recovery, monitor for recurrence, and document the incident.

Interview Question 6: What is the difference between CoreDNS, CNI, and kube-proxy?

Sample answer:

CoreDNS provides DNS resolution for Kubernetes Service names.

CNI is the interface through which the networking implementation configures Pod connectivity, including IP addressing and interfaces.

kube-proxy, when used, implements Kubernetes Service networking by programming forwarding rules based on Service and EndpointSlice information.

These components have different responsibilities, but they work together to support communication between Kubernetes applications.

Interview Question 7: How do you communicate during a production incident?

Sample answer:

I acknowledge the incident and confirm the business impact.

I provide regular updates to the incident channel, including what has been verified, what I am investigating, and when the next update is expected.

I avoid presenting assumptions as confirmed root causes.

If the incident requires escalation or a potentially disruptive change, I coordinate with the incident lead and relevant teams.

After service recovery, I document the root cause, resolution, verification results, and preventive actions.

PART 20 — YOUR FINAL HOMEWORK

Submit the following:

Screenshot showing Kubernetes Nodes in Ready status.
Screenshot showing your Restaurant Deployment and two Ready Pods.
Screenshot showing your Kubernetes Service and ClusterIP.
Screenshot showing the EndpointSlices.
Screenshot showing successful CoreDNS resolution.
Screenshot showing HTTP 200 from the healthy Service.
Screenshot showing the broken Service selector.
Screenshot showing missing backend endpoints.
Screenshot showing the failed HTTP request.
Screenshot showing the corrected selector.
Screenshot showing HTTP 200 after the fix.
Your first Slack incident acknowledgment.
Your investigation update.
Your resolution update.
Your completed incident report.
A written or recorded interview explanation of the incident.

Final practical assessment

Your instructor may ask you to share your screen and demonstrate:

How to find a Deployment.
How to check Pod health.
How to find the Pod IP.
How to inspect a Service.
How to inspect EndpointSlices.
How to check DNS resolution.
How to test HTTP connectivity.
How to identify a selector mismatch.
How to correct the configuration.
How to verify recovery.

You should explain what every command does and why you are using it.

PART 21 — CLEANUP

After your instructor confirms that the assignment is complete, remove the lab resources:

kubectl delete namespace jumptotech-lab

This removes the resources created in that Namespace.

Do not run this command until you have saved your screenshots and completed the assignment.

FINAL MESSAGE TO STUDENTS

Your goal is not to memorize Kubernetes commands. Your goal is to understand the full lifecycle of an application and how to troubleshoot it when something goes wrong.

A strong DevOps Engineer should be able to explain:

How application code moves from GitHub to Kubernetes.
How Kubernetes schedules and runs containers.
How Pods receive network connectivity.
How Services and EndpointSlices support application communication.
How DNS resolution differs from HTTP routing.
How an HTTP request reaches an application.
How to investigate a production incident safely.
How to communicate findings and progress.
How to verify that a problem is resolved.
How to explain the work clearly in a technical interview.

Remember the incident response structure:

Impact → Investigation → Evidence → Root Cause → Mitigation → Verification → Communication → Prevention

Practice explaining your project in your own words.

Be clear about which activities you performed in a lab and which tools or processes belong to the example production architecture. Never present a training exercise as actual employment experience.

JumpToTech DevOps Bootcamp — Learn It. Build It. Troubleshoot It. Explain It.

You can copy this entire handbook directly into your student group or development platform. It includes the complete lab, the simulated ticket, step-by-step commands, workplace communication, incident reporting, and interview preparation.

JumpToTech DevOps/SRE Bootcamp Production Incident Response Lab — Banking Application on AWS EKS

Aisalkyn Aidarova — Wed, 28 Oct 2026 21:17:39 +0000

Kubernetes Incident Response — Banking Application on AWS EKS

One of the most critical incidents I handled involved a customer-facing banking application running on AWS EKS.

Our production environment had approximately 60 frontend replicas distributed across three Availability Zones, with an AWS Application Load Balancer handling incoming traffic. We used Prometheus and Grafana for monitoring and alerting.

During a production release, we started receiving alerts that the banking application's HTTP 5xx error rate had increased significantly. The error rate reached approximately 18%, and customers were experiencing intermittent failures when accessing their accounts.

I was the primary SRE responsible for investigating the incident and coordinating the technical recovery.

My first step was to assess the impact. I checked Grafana dashboards for error rates, request latency, and available replicas. I noticed that available frontend replicas had dropped from 60 to 38, which explained why the remaining Pods were struggling to handle traffic.

I then checked the EKS cluster using kubectl get nodes, kubectl get pods, and kubectl get deployments. All worker nodes were healthy, so I ruled out a cluster-wide infrastructure failure.

When I inspected the affected Pods using kubectl describe pod, I found repeated readiness-probe failures. The new containers were Running, but they were not becoming Ready.

I checked the Kubernetes events and application logs and found that the readiness probes were returning HTTP 404.

After comparing the new deployment configuration with the previous release, I identified the root cause: the readiness probe had been changed from / to /healthz, but the application did not expose the new endpoint.

I also found that the deployment configuration allowed too many healthy Pods to become unavailable during the rollout, which contributed to the loss of serving capacity.

At that point, my priority was restoring service rather than troubleshooting the defective release further.

I rolled back the deployment using kubectl rollout undo deployment/banking-frontend and monitored the recovery using kubectl rollout status.

As the previous application version came back online, available replicas increased from 38 to 60, and the HTTP 5xx error rate returned to its normal baseline.

I verified recovery by checking Kubernetes Service endpoints, application health checks, load balancer metrics, and Grafana dashboards.

The application was restored within approximately 12 minutes of incident detection.

After the incident, I completed a root cause analysis and recommended three improvements: automated readiness-probe validation in CI/CD, safer RollingUpdate settings with maxUnavailable: 0, and canary deployments to detect faulty releases before they affect the entire application.

The outcome was that I restored full application availability, reduced the immediate customer impact, and helped establish deployment safeguards to prevent the same failure from recurring.

This incident reinforced the importance of understanding Kubernetes readiness probes, deployment strategies, monitoring, and rapid rollback procedures in a high-availability banking environment.

Numbers in this example
Metric Value
Frontend replicas 60
Availability Zones 3
Available replicas during incident 38
Unavailable replicas 22
Peak HTTP 5xx error rate 18%
Readiness-probe error HTTP 404
Recovery time 12 minutes
Replicas after recovery 60
Recovery method Kubernetes rollback

These numbers form a consistent hypothetical production incident, not a historical incident verified at JPMorgan Chase. In a real interview, students should use the metrics from their own incident evidence rather than present these figures as their personal employment history.

What makes this a senior-level answer: It demonstrates that the engineer assessed customer impact, distinguished application failure from infrastructure failure, investigated Kubernetes events and logs, identified the configuration defect, performed a controlled rollback, verified recovery, and implemented preventive measures.
Student Lab Guide · Day 2

Scenario: Banking Application Outage After a Failed Production Deployment\
Environment: AWS EKS, Kubernetes, Prometheus, Grafana, NGINX\
Role: DevOps / Site Reliability Engineer\
Difficulty: Intermediate\
Estimated duration: 3 hours\
Incident severity: Simulated SEV-2 / P2

Important note for students

This lab uses a fictional banking incident inspired by the types of production problems that can occur in large financial institutions such as JPMorgan Chase. It is not a documented incident at Chase, and you must not claim that you worked at Chase or resolved this incident in its actual production environment.

You will, however, perform a real hands-on investigation in your own Kubernetes cluster. You can discuss that experience confidently in interviews as an independently resolved production-style lab incident.

1. Your role and responsibilities

You are a DevOps/SRE Engineer supporting a banking application running on Amazon EKS.

The application allows customers to access banking services. Your team is responsible for keeping the infrastructure reliable, deploying application releases, monitoring service health, and restoring service when an incident occurs.

During this lab, you are responsible for:

Monitoring: Detect when application availability decreases.
Incident response: Assess the impact and prioritize recovery.
Troubleshooting: Investigate Deployments, ReplicaSets, Pods, Services, probes, events, and logs.
Mitigation: Restore the last known healthy application version.
Verification: Confirm application availability and healthy Kubernetes status.
Communication: Provide clear updates to the incident commander and application team.
Root cause analysis: Explain why the failure occurred and how to prevent recurrence.

Your objective is not simply to restart a Pod. Your objective is to restore customer-facing service safely, understand the failure, and reduce the chance of recurrence.

2. The incident you will investigate

INC-2026-001

Simulated SEV-2

Banking frontend unavailable after a Kubernetes deployment

Business impact: Customers may be unable to access the banking frontend during the failed rollout.

Technical symptoms: Deployment unavailable, Pods not Ready, application health checks failing.

Expected root cause: A newly released application version fails its readiness checks.

Recovery strategy: Roll back to the previous known-good Deployment revision.

What happened?

The development team released a new version of the banking frontend.

The new container starts, but a configuration defect causes its HTTP readiness endpoint to fail. Kubernetes therefore cannot mark the new Pods as Ready.

As the rollout progresses, the old working Pods are replaced. Eventually, the application has no Ready endpoints, and customers cannot access the service.

Prometheus detects the decrease in available replicas. Grafana displays the outage and can trigger an alert.

You are the engineer responsible for diagnosing the failure and restoring the previous working release.

What success looks like

At the end of the incident:

The Deployment is restored to its previous healthy revision.
Two frontend replicas are Ready.
The Kubernetes Service has healthy endpoints.
The application returns HTTP 200.
Monitoring shows recovery.
You have recorded the root cause, commands, evidence, and preventive actions.

Do not invent recovery times, financial losses, customer counts, or performance improvements. Measure your actual lab results and use those numbers in your interview explanation.

3. Understand the production architecture

Banking customers

Customer requests

Application access layer

Production would typically use HTTPS and an ALB or Ingress

Kubernetes Service

banking-service · ClusterIP · Port 80

Frontend Pod 1

NGINX · Port 80

Frontend Pod 2

NGINX · Port 80

Prometheus collects Kubernetes health metrics. Grafana visualizes them and evaluates alert rules.

In this lab, the frontend is represented by an NGINX container. It is not a complete banking system with authentication, payment processing, or a database. The incident concerns the frontend availability layer.

4. Lab prerequisites

From the previous lesson, you should already have:

Resource	Expected state
EKS cluster	Active
Worker nodes	Ready
Namespace	`banking-incident-lab`
Deployment	`banking-frontend`
Replicas	2/2 Ready
Service	`banking-service`
Prometheus	Running
Grafana	Running

Use the same cluster and namespace as yesterday. Do not delete or reinstall the monitoring stack.

Part A — Establish the healthy production baseline

Step 1. Verify your AWS and Kubernetes environment

Run these commands on the machine where your kubectl is configured.

kubectl config current-context

kubectl get nodes -o wide

kubectl get pods -n monitoring

kubectl get deployment -n banking-incident-lab

kubectl get pods -n banking-incident-lab -o wide

kubectl get svc -n banking-incident-lab

Expected Deployment:

NAME               READY   UP-TO-DATE   AVAILABLE
banking-frontend   2/2     2            2

Student task: Explain why READY 2/2 matters and how Kubernetes determines whether a Pod is ready.

Step 2. Inspect the readiness probe

kubectl describe deployment banking-frontend \
  -n banking-incident-lab

Look for:

Readiness: http-get http://:80/

The readiness probe is configured to request / on port 80.

The kubelet runs the probe. Successful HTTP responses (status codes 200–399) allow the container to become Ready.

A failed readiness probe does not necessarily restart the container. Instead, the Pod becomes Not Ready and is excluded from normal Service traffic.

Step 3. Verify the Service endpoints

kubectl get endpointslice \
  -n banking-incident-lab \
  -l kubernetes.io/service-name=banking-service \
  -o wide

Then:

kubectl get endpoints banking-service \
  -n banking-incident-lab

You should see two Pod addresses associated with the Service.

The EndpointSlice output is the preferred way to inspect endpoint readiness in modern Kubernetes. The legacy Endpoints command is included because it is still common in interviews and older runbooks.

Step 4. Verify the application responds

kubectl run baseline-client \
  -n banking-incident-lab \
  --image=curlimages/curl:8.11.1 \
  --restart=Never \
  --rm -i \
  --command -- \
  curl -sS -o /dev/null \
  -w "HTTP %{http_code}\n" \
  --max-time 10 \
  http://banking-service

Expected:

HTTP 200

A temporary warning about attaching to the diagnostic Pod is acceptable if the curl command still returns HTTP 200.

Evidence to save: Terminal output showing two Ready replicas and a successful HTTP response.

Step 5. Save the healthy Deployment configuration

This is an essential production habit: preserve the current state before making changes.

mkdir -p ~/banking-incident-lab/evidence

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o yaml \
  > ~/banking-incident-lab/evidence/deployment-before.yaml

kubectl rollout history deployment/banking-frontend \
  -n banking-incident-lab

Do not manually edit the saved baseline file during the incident.

Part B — Configure Prometheus and Grafana

Step 6. Open Grafana

Use the Grafana instance you configured in the previous lesson.

If Grafana is not accessible, start the port-forward on the management machine:

kubectl port-forward \
  -n monitoring \
  svc/monitoring-grafana \
  3000:80 \
  --address 127.0.0.1

If you access Grafana through an EC2 management machine, use your existing SSH tunnel to reach its localhost port securely.

Step 7. Create a production monitoring dashboard

In Grafana, go to Dashboards → New → New dashboard → Add visualization and select your Prometheus data source.

Create these four panels.

Panel	Visualization	Purpose
Available replicas	Stat	Number of available frontend replicas
Desired replicas	Stat	Number Kubernetes should maintain
Unavailable replicas	Stat	Detect loss of Deployment availability
Container restarts	Time series	Track application restarts

Panel 1 — Available replicas

kube_deployment_status_replicas_available{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}

Healthy value: 2

Panel 2 — Desired replicas

kube_deployment_spec_replicas{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}

Healthy value: 2

Panel 3 — Unavailable replicas

kube_deployment_spec_replicas{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}
-
kube_deployment_status_replicas_available{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}

Healthy value: 0

Panel 4 — Container restarts

sum(
  kube_pod_container_status_restarts_total{
    namespace="banking-incident-lab",
    pod=~"banking-frontend-.*"
  }
) or vector(0)

Healthy initial value: normally 0.

Save the dashboard as:

JumpToTech — Banking Production Incident Monitoring

If a query returns No Data, verify the Prometheus data source, kube-state-metrics, namespace, and Deployment labels before proceeding. Do not treat missing metrics as a healthy result.

Step 8. Create an incident alert

Navigate to Alerting → Alert rules → New alert rule.

Configure:

Field	Value
Alert name	`BankingFrontendUnavailable`
Data source	Prometheus
Evaluation interval	1 minute
Pending period	1 minute
Condition	Last value greater than 0
Severity label	`warning`
Folder	`Banking Production Lab`
Notification	Your configured email contact point

Use this PromQL expression:

kube_deployment_spec_replicas{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}
-
kube_deployment_status_replicas_available{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}

Set the alert condition to Last IS ABOVE 0. Depending on your Grafana version, this may require a Reduce expression and a Threshold expression.

Use these annotations:

Summary: Banking frontend has unavailable replicas.

Description: The banking-frontend Deployment has fewer available replicas than desired. Investigate Pod readiness, Deployment rollout status, Kubernetes events, and Service endpoints.

For the lab, use No Data = No Data and Error = Error so that missing metrics or evaluation failures are not silently treated as healthy. These states should be monitored separately in a real production alerting design.

Before proceeding, verify the alert is saved and that the email contact point is selected by the notification policy.

Part C — Simulate the production incident

Controlled incident simulation

The next commands deliberately make the banking frontend unavailable. Execute them only in your dedicated training namespace, never in a real production banking environment.

Verify that kubectl config current-context points to your training cluster before continuing.

Step 9. Understand the failure before creating it

Our healthy NGINX container responds to:

GET /
HTTP 200 OK

We will introduce a bad readiness probe that requests a nonexistent endpoint:

GET /healthz
HTTP 404 Not Found

NGINX will still be running, but the kubelet will mark the container Not Ready.

To make the full outage reproducible, we will temporarily use the Deployment's Recreate strategy. This intentionally removes the old Pods before starting replacements. This is not a recommended production rollout strategy for a high-availability banking frontend.

This combination demonstrates two failures: a bad health-check configuration and a rollout strategy that allows total service interruption.

Step 10. Record the incident start time

date -u +"Incident started: %Y-%m-%dT%H:%M:%SZ"

Record the output in your incident worksheet.

Step 11. Configure the controlled rollout strategy

kubectl patch deployment banking-frontend \
  -n banking-incident-lab \
  --type=merge \
  -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'

This changes the rollout strategy without changing the application template.

Confirm:

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o jsonpath='{.spec.strategy.type}{"\n"}'

Expected:

Recreate

Step 12. Introduce the faulty readiness probe

kubectl patch deployment banking-frontend \
  -n banking-incident-lab \
  --type=strategic \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"frontend","readinessProbe":{"httpGet":{"path":"/healthz","port":80},"initialDelaySeconds":5,"periodSeconds":5}}]}}}}'

This creates a new Deployment revision with a bad readiness endpoint.

Wait approximately 30–60 seconds.

Step 13. Observe the production failure

Run:

kubectl get pods -n banking-incident-lab

kubectl get deployment banking-frontend \
  -n banking-incident-lab

kubectl rollout status deployment/banking-frontend \
  -n banking-incident-lab \
  --timeout=30s

Expected symptoms may resemble:

NAME                                READY   STATUS
banking-frontend-xxxxxxxxxx-aaaaa   0/1     Running
banking-frontend-xxxxxxxxxx-bbbbb   0/1     Running

NAME               READY   UP-TO-DATE   AVAILABLE
banking-frontend   0/2     2            0

The rollout status command should time out because the new Pods never become available.

Important: A Pod can have STATUS Running while showing READY 0/1. These fields describe different aspects of Pod health.

Step 14. Confirm customer-facing impact

Inspect the endpoints:

kubectl get endpointslice \
  -n banking-incident-lab \
  -l kubernetes.io/service-name=banking-service \
  -o yaml

Look for:

conditions:
  ready: false

Now test the application:

kubectl run incident-client \
  -n banking-incident-lab \
  --image=curlimages/curl:8.11.1 \
  --restart=Never \
  --rm -i \
  --command -- \
  curl -v --max-time 10 \
  http://banking-service

Expected: the request fails or times out because the Service has no Ready backend endpoints.

The exact network error can vary with the cluster networking implementation.

Record the actual error. Do not assume every failed Service request returns HTTP 503.

Part D — Investigate like a production SRE

Step 15. Establish the scope

First determine whether the problem affects only the banking application or the whole Kubernetes cluster.

kubectl get nodes

kubectl get pods -n banking-incident-lab -o wide

kubectl get pods -n monitoring

kubectl get deployment banking-frontend \
  -n banking-incident-lab

Ask yourself:

Are Worker Nodes Ready?
Are Prometheus and Grafana healthy?
Are the banking Pods Running?
Are the banking Pods Ready?
Is the issue isolated to the frontend?

Do not immediately restart every Pod. That can destroy useful evidence without fixing the underlying configuration problem.

Step 16. Examine Kubernetes events

kubectl get events \
  -n banking-incident-lab \
  --sort-by=.lastTimestamp

Look for an event resembling:

Warning  Unhealthy  Readiness probe failed:
HTTP probe failed with statuscode: 404

This is a strong clue that the application is running but failing its health check.

Step 17. Describe the affected Pods

kubectl describe pods \
  -n banking-incident-lab \
  -l app=banking-frontend

Find the Readiness field and the Events section.

You should observe that the probe is requesting /healthz instead of /.

Step 18. Examine application logs

kubectl logs \
  -n banking-incident-lab \
  -l app=banking-frontend \
  -c frontend \
  --tail=100 \
  --prefix=true

Look for NGINX requests to /healthz returning HTTP 404.

This is the evidence that connects the Kubernetes probe failures to the application endpoint.

Step 19. Compare the deployment revisions

kubectl rollout history deployment/banking-frontend \
  -n banking-incident-lab

Inspect the live readiness configuration:

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o jsonpath='{.spec.template.spec.containers[0].readinessProbe.httpGet.path}{"\n"}'

Expected:

/healthz

Compare it with the baseline configuration saved before the incident, where the path was /.

Root cause analysis

Primary configuration defect

The new Deployment revision used /healthz as its readiness endpoint, but the NGINX application did not serve that path.

Failure mechanism

Kubelet received HTTP 404, marked the new Pods Not Ready, and the Service had no Ready endpoints.

Outage-enabling rollout configuration

The intentionally selected Recreate strategy removed the healthy old Pods before the replacement Pods passed readiness checks.

Business impact in the simulation

The banking frontend became unavailable to clients using its Kubernetes Service.

Part E — Recover production service

Step 20. Perform a Kubernetes rollback

We have identified the defective Deployment revision. Rather than manually editing the new release during the outage, restore the previous working version.

kubectl rollout undo deployment/banking-frontend \
  -n banking-incident-lab

Check the recovery:

kubectl rollout status deployment/banking-frontend \
  -n banking-incident-lab \
  --timeout=180s

Expected:

deployment "banking-frontend" successfully rolled out

A Deployment rollback restores the previous Pod template. It does not automatically restore the Deployment's rollout strategy, so we must correct that separately.

Step 21. Restore the safer RollingUpdate strategy

kubectl patch deployment banking-frontend \
  -n banking-incident-lab \
  --type=merge \
  -p '{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"}}}}'

With two replicas, this configuration allows a surge Pod while rounding the permitted unavailable count down to zero during normal rolling updates.

Verify:

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o jsonpath='{.spec.strategy.type}{"\n"}'

Expected:

RollingUpdate

Step 22. Verify recovery

kubectl get pods -n banking-incident-lab

kubectl get deployment banking-frontend \
  -n banking-incident-lab

kubectl get endpointslice \
  -n banking-incident-lab \
  -l kubernetes.io/service-name=banking-service \
  -o wide

The Deployment should return to 2/2 available.

Verify the readiness endpoint was restored:

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o jsonpath='{.spec.template.spec.containers[0].readinessProbe.httpGet.path}{"\n"}'

Expected:

Finally, test the Service again:

kubectl run recovery-client \
  -n banking-incident-lab \
  --image=curlimages/curl:8.11.1 \
  --restart=Never \
  --rm -i \
  --command -- \
  curl -sS -o /dev/null \
  -w "HTTP %{http_code}\n" \
  --max-time 10 \
  http://banking-service

Expected:

HTTP 200

Step 23. Record the recovery time

date -u +"Service restored: %Y-%m-%dT%H:%M:%SZ"

Calculate the elapsed time between the incident start and recovery.

This is your measured lab recovery duration. It is not automatically the same as production MTTR, which is typically calculated across multiple incidents.

Part F — Verify monitoring and notification

Return to the JumpToTech — Banking Production Incident Monitoring dashboard.

The expected pattern is:

Illustrative availability timeline

Example only. Replace these values with observations from your Grafana dashboard.

Desired replicasAvailable replicas

00.511.52HealthyFailureInvestigationRollbackRecovered

Check whether the alert transitioned through its expected states:

Normal → Pending → Firing → Resolved

Because alert evaluations run periodically, a short outage may not last long enough to enter Firing. If no alert fired, record that observation rather than claiming that an email was sent.

If an email was delivered, preserve the timestamp and alert name as evidence.

Part G — Your incident report

Every student must complete this worksheet using their own terminal outputs and observations.

Incident report — INC-2026-001

Complete after the hands-on exercise. Copy the report into your lab submission.

Engineer name

Incident start time (UTC)

How was the incident detected?

Observed application impact

Key commands and evidence

Root cause

Mitigation and recovery actions

Service restoration time (UTC)

Measured recovery duration

Preventive actions

Copy completed incident report

Prevention recommendations

A strong SRE does more than restore service. They identify engineering improvements.

For this incident, recommend:

Add automated readiness-endpoint checks to the CI/CD pipeline before deployment.
Use RollingUpdate with an appropriate availability budget for customer-facing applications.
Test application health checks in staging before promoting to production.
Monitor desired versus available replicas and alert on sustained unavailability.
Introduce canary deployments or progressive delivery for high-risk releases.
Maintain a documented rollback runbook and test it regularly.

Part H — Interview preparation

The question interviewers may ask

Tell me about a production incident you independently investigated and resolved. What was the impact, root cause, and outcome?

Sample answer in English

Production-Style Incident Response — AWS EKS Banking Application

During a hands-on production incident simulation, I was responsible for investigating and restoring a banking frontend application deployed on AWS EKS.

The application was running with two replicas behind a Kubernetes ClusterIP Service. We used Prometheus and Grafana to monitor Deployment availability, Pod readiness, and container restarts.

During a deployment, the frontend became unavailable. My initial investigation showed that the desired replica count was two, but zero replicas were available.

I checked the Deployment rollout status, inspected the Pods, reviewed Kubernetes events, and examined the application logs.

The containers were running, but their readiness probes were failing with HTTP 404 errors.

I traced the issue to an incorrect readiness probe path introduced in the new Deployment revision. The application was configured to check /healthz, but NGINX was only serving the root endpoint. The rollout also used a Recreate strategy, which removed the healthy Pods before the replacement Pods were ready.

After identifying the root cause, I rolled back the Deployment to the previous working revision and restored the RollingUpdate strategy.

I verified that both replicas returned to Ready, the Service had healthy endpoints, and the application responded with HTTP 200.

The key lesson was that a running container does not necessarily mean the application is ready to serve traffic.

To prevent recurrence, I recommended automated health-check validation in CI/CD, safer rollout strategies, and Prometheus/Grafana alerting for unavailable replicas.

This exercise gave me practical experience in Kubernetes incident investigation, root cause analysis, rollback, service recovery, and production reliability practices.

This is an interview practice script for a lab incident. Personalize it with your actual evidence and recovery time. Do not present the simulation as an incident you handled while employed by a bank.

Follow-up interview questions you must master

12 technical interview questions

Tap to reveal answers

1.Why were the Pods Running but not Ready?

2.What is the difference between readiness and liveness probes?

3.Why didn't Kubernetes automatically fix the incident?

4.What did the Service do when both Pods were Not Ready?

5.Why did the deployment cause a full outage?

6.Would the default RollingUpdate strategy behave the same way?

7.Why use kubectl rollout undo?

8.Did the rollback restore every Deployment setting?

9.How did Prometheus detect the problem?

10.What is the difference between mitigation and root cause analysis?

11.What would you communicate to stakeholders?

12.What is the most important prevention?

Part I — Student submission checklist

Evidence and deliverables

0/12 completed

Deployment healthy before incident: 2/2

Baseline application test: HTTP 200

Grafana dashboard with banking metrics

Screenshot showing failed readiness probes

Kubernetes events showing HTTP 404

Application logs showing failed /healthz requests

Deployment rollout history

Successful rollback output

Deployment recovered: 2/2

Application recovered: HTTP 200

Incident report with measured recovery time

Interview explanation practiced aloud

Final learning outcome

By completing this lab, you should be able to explain the entire incident lifecycle:

Detect

Investigate

Mitigate

Verify

Prevent

Your goal is not to memorize a fictional story. Your goal is to reproduce the failure, understand every command, independently restore the application, and explain your own results clearly in an interview.

This is the difference between knowing Kubernetes commands and demonstrating practical DevOps/SRE incident-response skills.

JumpToTech DevOps/SRE Bootcamp — Production Incident Response Lab, Day 2

Marta Dzekevich — Cloud & Platform Engineering Manager Interview Preparation

Aisalkyn Aidarova — Wed, 28 Oct 2026 03:48:56 +0000

1. Tell me about yourself

Sure, absolutely.

My name is Marta, and I'm a Senior DevOps Engineer with over eight years of experience in cloud infrastructure, automation, Kubernetes, CI/CD, and production support.

Currently, I'm working at Wells Fargo, where I focus on building and maintaining reliable AWS infrastructure, improving deployment automation, and supporting containerized applications.

A big part of my work involves Terraform, Amazon EKS, Docker, GitHub Actions, Jenkins, and AWS services such as EC2, VPC, IAM, ALB, and CloudWatch.

One of my key accomplishments was improving our CI/CD pipelines and reducing deployment time from approximately 45 minutes to 8 minutes through automation.

I also worked on improving our release and rollback processes, reducing rollback time from around 20 minutes to 2 minutes using blue-green deployment strategies.

Another area I've focused on is cloud cost optimization. By analyzing AWS resource utilization and right-sizing infrastructure, I helped reduce our monthly development AWS spending by approximately 35%.

Before Wells Fargo, I worked at Home Depot, where I supported cloud-hosted business applications, networking, DNS, SSL/TLS, application integrations, and production operations.

Over the years, I've developed a strong understanding of how cloud infrastructure, automation, networking, and application reliability work together.

What interests me about this opportunity is the chance to contribute at a broader platform level, improve engineering standards, support cloud strategy, and help teams build reliable, secure, and cost-efficient infrastructure.

I believe my hands-on engineering background and experience solving infrastructure problems would allow me to contribute effectively to this role.

Speaking time: approximately 90 seconds to 2 minutes.

2. What are your day-to-day responsibilities?

My day-to-day responsibilities involve cloud infrastructure management, automation, CI/CD pipeline improvements, Kubernetes operations, and production support.

I typically start by reviewing AWS CloudWatch dashboards and alerts to make sure our infrastructure and applications are healthy.

I check for any failed deployments, high CPU or memory utilization, application availability issues, and infrastructure alerts.

Then I review our priorities with the team and identify any blockers or production issues that need immediate attention.

A significant part of my work involves Terraform. I maintain reusable Terraform modules to provision and manage AWS infrastructure across multiple environments.

For example, if a development team needs a new application environment, I can use our Terraform modules to provision VPC networking, EC2 resources, security groups, IAM roles, and load balancers.

I also work with Jenkins and GitHub Actions to maintain automated build, testing, security scanning, and deployment workflows.

On the Kubernetes side, I support Amazon EKS workloads, investigate failed deployments, check pod health, and review resource requests, limits, and autoscaling configurations.

Another responsibility is cloud cost optimization. I review AWS Cost Explorer and CloudWatch utilization data to identify underutilized resources and opportunities for right-sizing.

I also collaborate with development teams, security engineers, and other infrastructure teams to troubleshoot issues and improve operational reliability.

My main goal is to make our infrastructure more reliable, secure, automated, and cost-efficient while reducing manual work for engineering teams.

3. Production incident #1 — Deployment failure and rollback

CI/CD · NGINX · Blue-green deployment · Recovery

Marta's resume confirms blue-green deployment automation and a reduction in rollback time from 20 minutes to 2 minutes. The specific failure described below is an illustrative incident to adapt.

Sure. One example I can share involves a deployment issue where a new application release was causing errors after being deployed.

We noticed the issue through application health checks and started investigating the deployment.

My first priority was to understand the impact and restore the previous stable version as quickly as possible.

We were using a blue-green deployment strategy, where the current stable version and the new application version were deployed separately.

I checked the application health, reviewed the deployment logs, and confirmed that the new version wasn't responding correctly.

Instead of spending too much time troubleshooting while users were affected, we switched traffic back to the previous stable version.

We used our automated NGINX and Bash-based traffic-switching process to perform the rollback.

Once the previous version was serving traffic again, we verified application availability and reviewed the failed release with the development team.

After the incident, we improved our deployment validation and health-check procedures to reduce the risk of similar failures.

One of the major improvements from our release automation work was reducing rollback time from approximately 20 minutes to 2 minutes.

This experience reinforced the importance of having a reliable rollback strategy and prioritizing service restoration during production incidents.

Technical follow-up questions

Q: What is blue-green deployment?

Blue-green deployment uses two separate application environments. One serves production traffic while the other hosts the new version. Once the new version passes validation, traffic switches to it. If problems occur, traffic can be redirected to the previous version.

Q: How did you switch traffic?

In this example, an automated Bash script updated the NGINX upstream configuration and reloaded NGINX after validating the configuration.

Q: What commands would you use?

nginx -t
sudo systemctl reload nginx
sudo systemctl status nginx
curl -I https://application.example.com

Q: What would you do if the new version had already changed the database schema?

I would verify database backward compatibility before switching traffic. An application rollback is not automatically a safe database rollback, so schema migrations need a separate recovery plan.

4. Production incident #2 — AWS infrastructure connectivity issue

AWS VPC · Security groups · ALB · Networking

This is a sample incident built around Marta's documented AWS networking and connectivity troubleshooting experience.

Another incident involved an application connectivity problem in our AWS environment.

The application was deployed, but users were experiencing connection failures, and the application wasn't responding as expected.

I started by checking the application health and reviewing the AWS load balancer configuration.

I noticed that the Application Load Balancer was reporting unhealthy targets.

My next step was to investigate the target group health checks, EC2 instance status, and security group configurations.

After reviewing the networking configuration, we identified a security group rule that was preventing the load balancer from reaching the application on the required port.

I worked with the team to update the security group configuration through Terraform, following our normal review and approval process.

After applying the change, we verified that the ALB health checks were passing and the application became accessible again.

We continued monitoring the application to confirm that connectivity was stable.

As a preventive measure, we reviewed our Terraform modules and added additional validation to reduce the possibility of similar networking configuration issues.

The key lesson was that even a small infrastructure configuration change can have a major impact on application availability, which is why automated validation and infrastructure code reviews are so important.

Technical follow-up questions

Q: How do you investigate an ALB reporting unhealthy targets?

I check target group health status and reason codes, listener rules, health-check path and port, application readiness, security groups, network ACLs, and route tables.

Q: What AWS commands might you use?

aws elbv2 describe-target-health \
  --target-group-arn <target-group-arn>

aws ec2 describe-security-groups \
  --group-ids <security-group-id>

aws ec2 describe-instances \
  --instance-ids <instance-id>

Q: Why manage security groups through Terraform?

Because it keeps infrastructure configuration version-controlled, reviewable, and consistent across environments.

5. Explain your company structure and engineering team

Marta's resume does not specify Wells Fargo's actual reporting structure or team size. This is a reasonable sample description, not a verified organizational chart.

At Wells Fargo, I work as part of the cloud infrastructure and DevOps engineering function.

Our responsibility is to support AWS infrastructure, application deployment automation, and production reliability.

We collaborate closely with application developers, QA engineers, security teams, networking engineers, and other infrastructure teams.

The DevOps engineers focus on infrastructure provisioning, CI/CD automation, Kubernetes workloads, monitoring, and production troubleshooting.

Development teams are responsible for application functionality and code changes, while we provide the infrastructure and automated deployment processes that support those applications.

Security engineers help establish access controls, vulnerability management, and security requirements.

Networking engineers support connectivity, routing, DNS, and network security.

When we work on major infrastructure changes or production issues, we coordinate with the relevant teams to make sure the changes are implemented safely.

Although my primary role has been hands-on engineering, I also contribute to engineering standardization through reusable Terraform modules, automated workflows, documentation, and technical collaboration.

Cloud Infrastructure / Engineering Leadership

DevOps / Platform Engineering Team

Marta — Senior DevOps Engineer

AWS · Terraform · Kubernetes · CI/CD · Reliability

Application / QA

Security / Networking

Operations / Support

Illustrative collaboration structure; actual reporting lines should be confirmed.

6. What meetings do you attend?

We typically work in an Agile-style environment, so I participate in regular engineering and operational meetings.

We have daily stand-ups to review our progress, discuss current priorities, and identify blockers.

During sprint planning, we review upcoming infrastructure work, automation improvements, and technical tasks.

We also have technical discussions and design reviews when we're introducing new AWS infrastructure, modifying Terraform modules, or improving deployment workflows.

For production releases, we coordinate deployment schedules, change approvals, risk assessments, and rollback plans.

If a significant production issue occurs, we conduct an incident review to understand what happened, identify the root cause, and discuss preventive actions.

I also participate in cross-team meetings with developers, security engineers, and infrastructure teams when changes affect multiple systems.

These meetings help us maintain visibility, manage dependencies, and ensure infrastructure changes support the broader engineering goals.

Meeting	Example frequency	Purpose
Daily stand-up	Daily	Priorities and blockers
Sprint planning	Every 2 weeks	Work allocation
Backlog refinement	Weekly	Technical requirements
Architecture review	As needed	Cloud design decisions
Change / release review	Before major releases	Deployment safety
Incident review	After significant incidents	RCA and prevention
Engineering 1:1	Weekly or biweekly	Development and priorities

7. Tell me about your cloud cost optimization experience

This is one of Marta's strongest documented accomplishments and is particularly relevant to the management position.

Cloud cost optimization has been an important part of my work at Wells Fargo.

We wanted to reduce unnecessary AWS spending, particularly in our development environments, without affecting application performance or reliability.

I started by reviewing AWS Cost Explorer reports to identify our major cost drivers.

Then I used CloudWatch metrics to analyze resource utilization, including CPU and memory trends where available.

We identified opportunities to right-size EC2 instances and optimize storage resources based on actual usage.

I also reviewed infrastructure configurations to make sure our development environments were provisioned consistently.

We implemented the approved changes and monitored the environments afterward to verify that application performance remained stable.

As a result of this optimization effort, we reduced monthly development AWS spending by approximately 35%.

For me, cloud cost optimization is not just about reducing expenses. It's about finding the right balance between performance, availability, security, and cost.

8. Tell me about your Terraform experience

Terraform is one of the main tools I use for infrastructure automation.

At Wells Fargo, I've worked with reusable Terraform modules to provision AWS resources across more than eight environments.

These modules cover infrastructure components such as VPCs, subnets, EC2 instances, IAM roles, security groups, load balancers, and Auto Scaling.

We use remote Terraform state in Amazon S3 and state locking to help prevent conflicting infrastructure changes.

Our infrastructure code is version-controlled, and changes go through a review process before being applied.

One of the biggest advantages of using reusable modules is consistency. Instead of manually configuring infrastructure for every environment, we can standardize the resources and customize them using variables.

This reduces configuration drift, makes troubleshooting easier, and helps us provision infrastructure more efficiently.

I've also worked with Terraform and Ansible together to automate AWS infrastructure provisioning and server configuration.

I believe Infrastructure as Code is essential for managing cloud environments reliably at scale.

9. How do you demonstrate leadership without being a people manager?

This is likely one of the most important questions for this particular job.

My background has primarily been in senior hands-on engineering rather than formal people management.

However, I've contributed to technical leadership by helping standardize infrastructure provisioning, improve deployment processes, and establish repeatable engineering practices.

For example, at Wells Fargo, I worked on reusable Terraform modules across multiple AWS environments, which helped make infrastructure provisioning more consistent.

I also contributed to improving CI/CD automation and release reliability.

I believe strong technical leadership involves understanding problems, evaluating trade-offs, communicating clearly, and helping teams adopt solutions that are reliable and maintainable.

I also understand that a formal engineering management role involves additional responsibilities, including people development, prioritization, resource planning, and stakeholder management.

Those are areas I'm interested in developing further while bringing a strong technical foundation to the team.

10. Why are you interested in Herc Rentals?

I'm interested in this opportunity because it combines cloud infrastructure engineering with broader platform ownership and leadership responsibilities.

My experience has been focused on building reliable AWS environments, automating infrastructure with Terraform, supporting Kubernetes workloads, and improving CI/CD and operational processes.

What interests me about this position is the opportunity to apply that experience at a broader organizational level, helping establish engineering standards, improve cloud governance, and support a long-term cloud platform strategy.

I also appreciate the focus on reliability, security, and cost optimization, because those are areas where I've delivered measurable improvements.

For example, I've helped reduce deployment time from 45 minutes to 8 minutes, improve rollback time from 20 minutes to 2 minutes, and reduce development AWS spending by 35%.

I see this as an opportunity to contribute my technical experience while continuing to grow in engineering leadership and platform strategy.

11. Advanced questions to prepare for

1. How would you design a multi-cloud platform strategy?

Start with business requirements, workload suitability, security, compliance, cost, and recovery objectives. Define consistent identity, networking, monitoring, tagging, IaC, and governance standards across AWS, Azure, and OCI. Avoid moving workloads between clouds without a clear business justification.

2. How would you manage a cloud engineering team?

3. How do you manage cloud governance?

4. How would you design disaster recovery?

5. How do you prevent Terraform configuration drift?

6. What is your experience with Azure and OCI?

7. How would you handle disagreement between engineers?

8. How would you measure platform engineering success?

12. Practice mock interview

DevOps Engineer (AWS) — Interview Preparation for Elena Abditalipova

Aisalkyn Aidarova — Wed, 28 Oct 2026 03:44:52 +0000

HICKS PROFESSIONAL GROUP · CATERPILLAR OPPORTUNITY · CHICAGO / IRVING

1. Tell me about yourself

Sure, absolutely.

My name is Elena, and I'm a DevOps Engineer with over five years of experience working with AWS cloud infrastructure, Kubernetes, Terraform, CI/CD automation, and production support.

Currently, I'm working at Northern Trust, where my main responsibility is managing and improving our AWS infrastructure and Kubernetes environments.

We use Amazon EKS to run containerized applications, Terraform to provision infrastructure, GitHub Actions and Jenkins for CI/CD automation, and Argo CD for GitOps-based deployments.

A big part of my work involves building reusable Terraform modules, improving deployment pipelines, monitoring production environments, and troubleshooting infrastructure and application issues.

For monitoring, I work with Prometheus, Grafana, Datadog, and AWS CloudWatch. We use these tools to monitor application health, resource utilization, and infrastructure performance.

Before Northern Trust, I worked at CDW, where I focused on infrastructure automation, containerization, and CI/CD pipelines. Earlier, at Apex Systems, I supported cloud migration projects and automated AWS infrastructure using Terraform and Bash.

I also have experience using AI tools like GitHub Copilot, OpenAI, and Claude to improve infrastructure automation, troubleshooting, and technical documentation.

What I enjoy most about DevOps is solving complex production problems, reducing manual work through automation, and building reliable infrastructure that development teams can use efficiently.

I'm particularly interested in this opportunity because it closely aligns with my experience in AWS, EKS, Terraform, GitHub Actions, and Kubernetes automation.

Target speaking time: 90 seconds to 2 minutes.

2. What are your day-to-day responsibilities?

My daily responsibilities are a combination of infrastructure automation, Kubernetes operations, CI/CD improvements, monitoring, and production support.

I usually start my day by reviewing our monitoring dashboards in Grafana, Datadog, and CloudWatch to make sure our production applications and Kubernetes clusters are healthy.

I check for any critical alerts, failed deployments, pod restarts, high CPU or memory utilization, and infrastructure issues.

After that, we have our daily stand-up meeting, where we discuss what we completed yesterday, what we're working on today, and whether we have any blockers.

Throughout the day, I work on Jira tickets, which might involve updating Terraform modules, improving GitHub Actions workflows, deploying applications through Argo CD, or troubleshooting Kubernetes issues.

For example, if a development team needs a new AWS resource, I update the appropriate Terraform module, open a pull request, and go through our code review and approval process before applying the changes.

I also maintain our CI/CD pipelines. When developers push code, GitHub Actions automatically runs tests, builds Docker images, performs security scans, and pushes approved images to Amazon ECR. Argo CD then helps synchronize our Kubernetes deployments with the desired configuration in Git.

Another important part of my job is production support. If an application becomes unavailable or a deployment fails, I investigate the issue using kubectl, application logs, Kubernetes events, and monitoring dashboards.

I also collaborate closely with developers, QA engineers, security teams, and other infrastructure engineers to make sure our releases are secure and reliable.

My overall goal is to keep our infrastructure stable, automate repetitive tasks, and help development teams deliver applications faster and more safely.

3. Tell me about a production incident you resolved — Kubernetes application outage

Scenario 1 · EKS · CrashLoopBackOff · Application availability

This is a realistic incident example consistent with Elena's listed EKS and troubleshooting skills, but the specific incident is not verified by the resume.

Sure. One incident I can describe involves an application running in our production Amazon EKS environment.

After a new deployment, we started receiving alerts from Datadog and Prometheus indicating that some application pods were unhealthy and the application error rate was increasing.

My first step was to assess the impact and check the health of the Kubernetes workloads.

I used kubectl get pods to review the pod status and noticed that several pods were entering CrashLoopBackOff.

Then I used kubectl describe pod and kubectl logs --previous to investigate why the containers were restarting.

The logs showed that the application was failing during startup because it couldn't load a required environment variable.

I compared the new deployment configuration with the previous working version and found that a required configuration value had been omitted from the new release.

Since this was affecting production, our first priority was restoring service.

We rolled back to the previous stable deployment using Argo CD and verified that the application pods became healthy again.

After service was restored, I worked with the development team to correct the missing configuration and validate it in our lower environment.

We also improved our deployment validation process by adding configuration checks to the GitHub Actions pipeline.

The result was that we restored application availability and reduced the risk of the same issue happening again.

The biggest lesson from this incident was the importance of automated configuration validation and having a reliable rollback strategy.

Technical follow-up questions

Q: How did you identify CrashLoopBackOff?

kubectl get pods -n production
kubectl describe pod <pod-name> -n production
kubectl logs <pod-name> -n production --previous
kubectl get events -n production --sort-by=.lastTimestamp

Q: Why did the pod restart?

The application process exited during startup because a required configuration value was missing. Kubernetes restarted the container according to the pod's restart policy, and repeated failures caused the backoff delay.

Q: How did you roll back?

For an Argo CD-managed application, we restored the last known-good configuration in Git and synchronized the application. A direct kubectl rollout undo may also be used as an emergency mitigation, but the GitOps desired state must be corrected to prevent Argo CD from restoring the faulty version.

Q: How did you prevent recurrence?

We added required configuration checks in CI, tested the release in a lower environment, and improved deployment monitoring.

4. Tell me about another production incident — High CPU and EKS performance degradation

Scenario 2 · AWS EKS · HPA · Resource limits · Performance

Another incident involved performance degradation in our Kubernetes environment during a period of increased application traffic.

We received alerts from Prometheus and Grafana showing high CPU utilization and increased application response times.

I started by checking the cluster and workload health using kubectl top nodes and kubectl top pods.

I noticed that several application pods were consuming significantly more CPU than usual, and some workloads were approaching their configured CPU limits.

I checked the Horizontal Pod Autoscaler configuration and noticed that the application wasn't scaling as expected.

After investigating the HPA configuration and resource metrics, we identified an issue with the CPU resource requests used by the workload, which affected the autoscaling behavior.

I worked with the team to update the resource requests and limits, verify that the metrics pipeline was functioning correctly, and adjust the HPA configuration.

We also reviewed node capacity to make sure the cluster had enough resources to schedule additional replicas.

After applying the changes, the HPA started scaling the application appropriately as traffic increased.

We monitored the deployment through Grafana and CloudWatch and confirmed that CPU utilization stabilized and application response times improved.

As a preventive measure, we added more specific alerts for HPA scaling conditions, resource saturation, and application latency.

This incident reinforced the importance of properly configuring Kubernetes resource requests, autoscaling policies, and cluster capacity together.

Technical follow-up questions

Q: What commands did you use?

kubectl top nodes
kubectl top pods -n production

kubectl get hpa -n production
kubectl describe hpa <hpa-name> -n production

kubectl describe deployment <deployment-name> -n production
kubectl get events -n production --sort-by=.lastTimestamp

Q: How does HPA calculate CPU utilization?

For CPU utilization-based scaling, HPA compares actual CPU usage against the CPU requests configured for containers. If resource requests are missing or unsuitable, scaling can behave incorrectly.

Q: What if HPA increases replicas but pods remain Pending?

I would check node capacity, scheduler events, node taints, resource requests, and whether Cluster Autoscaler or Karpenter is configured to provision additional capacity.

Q: What's the difference between HPA and Karpenter?

HPA scales the number of application pod replicas. Karpenter provisions and manages worker-node capacity so the cluster can accommodate unschedulable workloads.

5. Explain your company structure and DevOps team

The resume confirms Northern Trust as Elena's current employer but does not specify her reporting structure or team size. The following is a sample enterprise team description to adapt to her actual situation.

At Northern Trust, I work within the cloud infrastructure and DevOps engineering organization.

Our team is responsible for supporting AWS infrastructure, Kubernetes platforms, CI/CD automation, and application deployment processes.

We work closely with several application development teams, as well as QA, security, networking, and infrastructure teams.

In a typical team structure, we have a DevOps engineering manager, senior DevOps engineers, DevOps engineers, and platform or cloud engineers.

Our DevOps team is responsible for infrastructure provisioning, Kubernetes operations, deployment automation, monitoring, and production reliability.

Developers focus on application code and business functionality, while our team provides the infrastructure and automation needed to build, test, and deploy their applications.

We also collaborate with security engineers on IAM permissions, secrets management, vulnerability scanning, and infrastructure security.

When production incidents occur, we coordinate with the application owners and relevant infrastructure teams to identify the root cause, restore service, and implement preventive improvements.

We use Jira to track tasks and GitHub for source control, pull requests, and infrastructure code reviews.

Example organizational structure

Engineering / Cloud Infrastructure Leadership

DevOps / Platform Engineering Manager

DevOps / SRE Team

Senior Engineers · DevOps Engineers · Platform Engineers

Elena — DevOps Engineer

Development / QA

Security / Networking

Cloud / Operations

Illustrative team structure, not a verified Northern Trust organization chart.

6. What meetings do you attend?

We follow an Agile-style workflow, so I participate in several regular meetings.

Every morning, we have a daily stand-up, usually around 15 minutes, where we discuss what we worked on yesterday, our priorities for today, and any blockers.

We also have sprint planning meetings, where we review upcoming Jira tickets, estimate the effort required, and decide which infrastructure or automation tasks we can complete during the sprint.

During backlog refinement, we review technical requirements and clarify dependencies before starting new work.

I also participate in technical design discussions, especially when we're planning Terraform infrastructure changes, Kubernetes upgrades, or CI/CD improvements.

Before major production deployments, we may have release readiness or change review meetings to discuss risks, rollback plans, and deployment procedures.

If we experience a significant production incident, we hold a post-incident review to discuss the root cause, business impact, and preventive actions.

Finally, I have regular one-on-one meetings with my manager to discuss priorities, performance, and professional development.

Example meeting schedule

Meeting	Frequency	Purpose
Daily stand-up	Daily, 15 min	Progress and blockers
Sprint planning	Every 2 weeks	Assign sprint work
Backlog refinement	Weekly	Review Jira tickets
Technical design	As needed	Architecture and changes
Release / change review	Before major releases	Risk and approval
Incident review	After significant incidents	Root cause and prevention
Manager 1:1	Weekly or biweekly	Priorities and feedback

7. Describe your GitHub Actions pipeline

This question is especially important because the job description requires extensive GitHub Actions experience.

In my current role, I work with GitHub Actions to automate our application build, testing, security scanning, and deployment processes.

Our typical pipeline starts when a developer opens a pull request or pushes changes to the repository.

First, we run code quality checks, unit tests, and dependency scanning.

Once the code is approved and merged, the pipeline builds a Docker image using a multi-stage Dockerfile.

We then run container security scanning using tools such as Trivy and publish the approved image to Amazon ECR.

For AWS authentication, I prefer using GitHub Actions OIDC with IAM roles instead of storing long-lived AWS access keys.

After the image is published, the deployment process updates the appropriate Kubernetes configuration in Git, and Argo CD synchronizes the changes to our EKS environment.

We also use environment-specific configurations and approval controls for production deployments.

One of the main benefits of this approach is that it gives us consistent, repeatable deployments with better security and traceability.

I'm also comfortable working with reusable workflows to avoid duplicating the same CI/CD logic across multiple repositories.

Typical deployment flow

GitHub Pull Request / Merge

Unit Tests

Code Quality

Security Scan

Docker Build → Amazon ECR

Update GitOps Repository

Argo CD → Amazon EKS

8. Why are you interested in this position?

I'm interested in this position because the technical requirements align very closely with the work I've been doing.

My strongest areas are AWS infrastructure, Kubernetes, Terraform, GitHub Actions, and automation.

What particularly interests me is the opportunity to work on enterprise-scale EKS environments, improve CI/CD workflows, and contribute to infrastructure standardization.

I also like that the position focuses on reliability, security, and cost optimization rather than just deploying applications.

I'm looking for an opportunity where I can take more ownership of infrastructure automation, solve challenging production problems, and continue growing as a DevOps and platform engineer.

9. Additional questions Elena should prepare for

The recruiter specifically emphasizes several advanced topics that are not clearly demonstrated in the resume, especially Karpenter, Actions Runner Controller, OIDC, IRSA, and zero-downtime EKS upgrades.

1. How do you implement GitHub Actions OIDC authentication with AWS?

Configure the GitHub OIDC identity provider in IAM, create an IAM role with a restricted trust policy for the repository and environment, and use aws-actions/configure-aws-credentials to assume that role. This avoids storing long-lived AWS access keys.

2. How would you upgrade a production EKS cluster with minimal downtime?

3. What is IRSA?

4. How do you troubleshoot a Kubernetes pod stuck in Pending?

5. How do you handle Terraform state?

6. What is Actions Runner Controller?

7. What is the difference between Argo CD and GitHub Actions?

JumpToTech DevOps + AI Bootcamp Hands-On Lab: Kubernetes Monitoring, Grafana Persistence, Gmail SMTP, and Production Alerts

Aisalkyn Aidarova — Mon, 26 Oct 2026 23:24:09 +0000

Environment: AWS EKS

Namespace: dev

Tools: Kubernetes, Helm, Prometheus, Grafana, AWS EBS CSI Driver, Gmail SMTP

Estimated duration: 2–3 hours

Difficulty: Intermediate

Learning objectives

By the end of this lab, students will be able to:

Deploy Prometheus and Grafana into Kubernetes.
Configure persistent storage for Grafana using AWS EBS.
Understand the difference between emptyDir and PersistentVolumeClaims.
Configure Gmail SMTP securely using Kubernetes Secrets.
Create Grafana email contact points and notification policies.
Create an ImagePullBackOff alert using PromQL.
Simulate a Kubernetes incident and receive an email notification.
Verify Grafana data survives a Pod restart.

Important architecture decision

Students must use their own development EKS cluster or an instructor-approved isolated lab environment.

Do not modify the existing production monitoring release.

The namespace dev isolates Kubernetes objects, but it does not provide complete security or resource isolation. Students must use their own credentials and must not share passwords.

Part 1 — Verify the Kubernetes environment

Step 1: Verify AWS identity

aws sts get-caller-identity

This confirms which AWS account and IAM identity the CLI is using.

Step 2: Connect to EKS

Replace the cluster name and region with your assigned development cluster.

export AWS_REGION="us-east-1"
export EKS_CLUSTER="YOUR-DEV-CLUSTER"

aws eks update-kubeconfig \
  --region "$AWS_REGION" \
  --name "$EKS_CLUSTER"

Verify:

kubectl config current-context
kubectl get nodes -o wide
kubectl cluster-info

Expected result: At least one worker node is Ready.

Step 3: Create the dev namespace

kubectl create namespace dev \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl get namespaces

All application and monitoring resources in this lab will be deployed into dev.

Part 2 — Verify AWS EBS storage

AWS EBS provides persistent block storage for Kubernetes workloads.

Step 1: Check the EBS CSI driver

aws eks describe-addon \
  --cluster-name "$EKS_CLUSTER" \
  --addon-name aws-ebs-csi-driver \
  --region "$AWS_REGION" \
  --query 'addon.{status:status,version:addonVersion,issues:health.issues}'

Expected:

ACTIVE

Also check:

kubectl get pods -n kube-system | grep ebs-csi

If the add-on is missing, stop and ask the instructor to install it with the correct IAM permissions. Do not create an EBS CSI IAM role or install the add-on independently in a shared cluster.

Step 2: Check StorageClasses

kubectl get storageclass

Look for a class provisioned by:

ebs.csi.aws.com

For this lab, we will create a dedicated gp3 StorageClass if it does not already exist.

cat > dev-gp3-storageclass.yaml <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: dev-gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF

Apply:

kubectl apply -f dev-gp3-storageclass.yaml
kubectl get storageclass dev-gp3

Explanation:

gp3 is the AWS EBS volume type.
encrypted: true enables EBS encryption.
Delete allows Kubernetes to delete the EBS volume when the PVC is deleted, subject to the provisioner and cloud cleanup succeeding.
WaitForFirstConsumer helps schedule the Pod and EBS volume in the same availability zone.

StorageClasses are cluster-scoped, so this step requires instructor approval in shared environments.

Part 3 — Install Prometheus and Grafana in dev

Step 1: Install Helm

Check first:

helm version

If Helm is not installed, use the approved installation method for your operating system.

Step 2: Add the Helm repository

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts

helm repo update

Step 3: Create a working directory

mkdir -p ~/jumptotech-monitoring-lab
cd ~/jumptotech-monitoring-lab

Step 4: Create monitoring values

cat > monitoring-values.yaml <<'EOF'
alertmanager:
  enabled: false

grafana:
  persistence:
    enabled: true
    type: pvc
    storageClassName: dev-gp3
    accessModes:
      - ReadWriteOnce
    size: 5Gi

  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 300m
      memory: 512Mi

prometheus:
  prometheusSpec:
    retention: 6h
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
      limits:
        cpu: 500m
        memory: 1Gi
EOF

We are enabling persistent storage from the beginning, so students do not need to migrate an existing Grafana database.

Step 5: Install the monitoring stack

helm upgrade --install dev-monitoring \
  prometheus-community/kube-prometheus-stack \
  --version 92.7.0 \
  --namespace dev \
  -f monitoring-values.yaml \
  --wait \
  --timeout 15m

Verify:

helm list -n dev

kubectl get pods -n dev

kubectl get deployments -n dev

kubectl get pvc -n dev

Expected:

Grafana Pod running
Prometheus Pod running
Grafana PVC bound to EBS
Helm release deployed

If the installation fails, inspect Pod events and resource capacity before retrying.

Part 4 — Verify Grafana persistent storage

Step 1: Find the Grafana Deployment

kubectl get deployment -n dev \
  -l app.kubernetes.io/name=grafana

Expected Deployment name:

dev-monitoring-grafana

Step 2: Inspect the storage volume

kubectl get deployment dev-monitoring-grafana \
  -n dev \
  -o jsonpath='{.spec.template.spec.volumes}'
echo

Look for a persistentVolumeClaim storage volume.

Step 3: Verify the PVC

kubectl get pvc -n dev
kubectl describe pvc -n dev

The Grafana claim should be Bound.

Discussion question: What would happen to Grafana dashboards if the application used emptyDir instead of a PVC?

Part 5 — Access Grafana

Step 1: Retrieve the Grafana admin password

Do not paste the password into chat or a shared document.

kubectl get secret dev-monitoring-grafana \
  -n dev \
  -o jsonpath='{.data.admin-password}' \
  | base64 --decode
echo

Step 2: Start port-forwarding

kubectl port-forward \
  -n dev \
  svc/dev-monitoring-grafana \
  3000:80

Open:

http://localhost:3000

If Kubernetes is managed from a remote EC2 machine, use an SSH tunnel to forward port 3000 to your own computer. Do not expose Grafana publicly just to complete this lab.

Username: admin
Password: password retrieved from Kubernetes Secret

Step 3: Verify Grafana health

From a second terminal, run:

curl -sS http://localhost:3000/api/health

Expected:

{
  "database": "ok"
}

Additional fields may be returned.

Part 6 — Configure Gmail SMTP

Step 1: Create a Google app password

Students must use their own Gmail account.

Open:

https://myaccount.google.com/apppasswords

Requirements:

Google 2-Step Verification enabled
A new app password created for the Grafana lab

Never use a normal Gmail login password.

Step 2: Create a Kubernetes Secret

Run:

read -r -p "Enter Gmail address: " GMAIL_ADDRESS

read -r -s -p "Enter Gmail app password: " GMAIL_APP_PASSWORD
echo

kubectl create secret generic grafana-smtp \
  -n dev \
  --from-literal=username="$GMAIL_ADDRESS" \
  --from-literal=password="$GMAIL_APP_PASSWORD"

unset GMAIL_APP_PASSWORD

Verify without revealing credentials:

kubectl get secret grafana-smtp -n dev

Expected:

grafana-smtp   Opaque   2

Step 3: Create SMTP Helm values

cat > grafana-smtp-values.yaml <<'EOF'
grafana:
  env:
    GF_SMTP_ENABLED: "true"
    GF_SMTP_HOST: "smtp.gmail.com:587"
    GF_SMTP_STARTTLS_POLICY: "MandatoryStartTLS"
    GF_SMTP_FROM_NAME: "JumpToTech Dev Alerts"
  envValueFrom:
    GF_SMTP_USER:
      secretKeyRef:
        name: grafana-smtp
        key: username
    GF_SMTP_PASSWORD:
      secretKeyRef:
        name: grafana-smtp
        key: password
    GF_SMTP_FROM_ADDRESS:
      secretKeyRef:
        name: grafana-smtp
        key: username
EOF

This uses the same Gmail address for authentication and the email sender.

Step 4: Preview the Helm deployment

helm template dev-monitoring \
  prometheus-community/kube-prometheus-stack \
  --version 92.7.0 \
  --namespace dev \
  -f monitoring-values.yaml \
  -f grafana-smtp-values.yaml \
  > monitoring-preview.yaml

Inspect the Grafana SMTP environment variables and Secret references in the rendered Deployment.

Step 5: Upgrade Helm

helm upgrade dev-monitoring \
  prometheus-community/kube-prometheus-stack \
  --version 92.7.0 \
  --namespace dev \
  -f monitoring-values.yaml \
  -f grafana-smtp-values.yaml \
  --wait \
  --timeout 15m

Verify:

kubectl get pods -n dev \
  -l app.kubernetes.io/name=grafana

kubectl get deployment dev-monitoring-grafana \
  -n dev \
  -o jsonpath='{range .spec.template.spec.containers[?(@.name=="grafana")].env[*]}{.name}{"\n"}{end}' \
  | grep GF_SMTP

Students should see seven SMTP environment variable names.

Part 7 — Configure email notifications in Grafana

Open Grafana in the browser.

Step 1: Create a contact point

Navigate to:

Alerting → Notification configuration → Contact points

Click New contact point.

Enter:

Name: Student-Dev-Email
Integration: Email
Address: your own Gmail address

Save.

Step 2: Send a test notification

Click Test → Send test notification.

Expected:

Test notification sent successfully

Check Gmail inbox and Spam.

The test may arrive with a subject containing FIRING. This is a simulated alert and does not necessarily indicate a real incident.

Step 3: Configure the notification policy

Navigate to:

Alerting → Notification configuration → Notification policies

For an isolated student Grafana installation:

Open the Default policy menu.
Select Edit.
Change the default contact point to Student-Dev-Email.
Keep grouping and timing settings unchanged.
Click Update policy.

Verify:

Delivered to Student-Dev-Email

Do not change a shared organization's default notification policy.

Part 8 — Create a real ImagePullBackOff incident

We will deliberately deploy an application with a nonexistent container image.

Use only the dev namespace.

Step 1: Create the broken Deployment

cat > broken-app.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: broken-restaurant
  namespace: dev
  labels:
    app: broken-restaurant
spec:
  replicas: 1
  selector:
    matchLabels:
      app: broken-restaurant
  template:
    metadata:
      labels:
        app: broken-restaurant
    spec:
      containers:
        - name: restaurant
          image: nginx:jumptotech-image-does-not-exist-2026
          imagePullPolicy: Always
          ports:
            - containerPort: 80
EOF

Apply:

kubectl apply -f broken-app.yaml

Step 2: Observe the failure

kubectl get pods -n dev

kubectl describe deployment broken-restaurant -n dev

kubectl get events -n dev \
  --sort-by=.lastTimestamp \
  | tail -20

The Pod should eventually show:

ImagePullBackOff

Step 3: Verify the Prometheus metric

In Grafana, open Explore, select the Prometheus data source, and enter:

kube_pod_container_status_waiting_reason{
  namespace="dev",
  reason="ImagePullBackOff"
}

Run the query in Instant mode.

Expected: a matching series with value 1 while the image-pull failure is present.

If no series appears, verify that kube-state-metrics is running and that Prometheus is scraping it before continuing.

Part 9 — Create the ImagePullBackOff alert

Navigate to:

Alerting → Alert rules → New alert rule

Step 1: Alert name

DEV - Kubernetes ImagePullBackOff

Step 2: Prometheus query A

kube_pod_container_status_waiting_reason{
  namespace="dev",
  reason="ImagePullBackOff"
}

Use an Instant query.

Step 3: Alert condition

Configure Grafana expressions:

Query A: Prometheus metric
Expression B: Reduce, Last of A
Expression C: Threshold, B is above 0.5
Set C as the alert condition

This evaluates each matching container series and fires when its value is 1.

Step 4: Evaluation

Evaluation interval: 1 minute
Pending period: 2 minutes
Keep firing for: 0 minutes

Use a dedicated folder and evaluation group, such as:

Folder: JumpToTech-Dev
Evaluation group: dev-kubernetes-alerts

Step 5: Notification routing

Use the Grafana notification policy configured earlier:

Default policy → Student-Dev-Email

Step 6: Alert message

Summary:

ImagePullBackOff detected in dev namespace

Description:

Kubernetes cannot pull a container image.
Check the Pod image, image registry, credentials, and Kubernetes events.

Step 7: Save the alert

Wait for the alert to evaluate.

Expected progression:

Normal → Pending → Firing

The email notification may arrive later because Grafana's notification policy also has grouping and delivery delays.

Part 10 — Fix the incident

Once the alert is firing, students must troubleshoot and resolve the problem.

Step 1: Inspect the Pod

kubectl get pods -n dev

kubectl describe deployment broken-restaurant -n dev

kubectl get events -n dev \
  --sort-by=.lastTimestamp \
  | tail -20

Step 2: Fix the container image

kubectl set image deployment/broken-restaurant \
  restaurant=nginx:1.27 \
  -n dev

Step 3: Verify recovery

kubectl rollout status deployment/broken-restaurant \
  -n dev \
  --timeout=180s

kubectl get pods -n dev

Expected:

Running
1/1

Step 4: Verify alert recovery

Return to Grafana.

The alert should transition back toward Normal after Prometheus and Grafana observe that the image-pull problem has cleared.

Students should also check whether a resolved email notification arrives.

Part 11 — Test Grafana persistence

Students must prove that Grafana can retain data after Pod replacement.

Step 1: Create a test dashboard

In Grafana:

Create a new dashboard.
Name it JumpToTech Dev Persistence Test.
Add a simple Prometheus panel.
Save the dashboard.

Step 2: Restart Grafana

kubectl rollout restart deployment/dev-monitoring-grafana \
  -n dev

kubectl rollout status deployment/dev-monitoring-grafana \
  -n dev \
  --timeout=180s

Step 3: Verify the PVC

kubectl get pvc -n dev
kubectl get pods -n dev \
  -l app.kubernetes.io/name=grafana

Step 4: Check Grafana

Log back into Grafana and verify:

The test dashboard still exists.
The email contact point still exists.
The alert rule still exists.
The notification policy is unchanged.

Expected result: The Grafana database and its saved objects survive the Pod restart.

Part 12 — Troubleshooting

Problem	What to investigate
Grafana PVC Pending	EBS CSI driver, IAM permissions, scheduling, storage class
Grafana Pod Pending	Node capacity, volume zone, Pod events
Grafana HTTP 401	Authentication or API permissions
Gmail 535 error	Incorrect or revoked Gmail app password
SMTP timeout	Outbound connectivity, port 587, security controls
No email received	Contact point, policy routing, SMTP logs, Gmail Spam
No Prometheus metric	kube-state-metrics, scrape targets, namespace labels
Alert stays Normal	Query results, threshold, evaluation group
Dashboard disappears	PVC mount, storage configuration, database errors

Useful troubleshooting commands:

kubectl get pods -n dev

kubectl get pvc -n dev

kubectl get events -n dev \
  --sort-by=.lastTimestamp \
  | tail -30

kubectl logs -n dev \
  deployment/dev-monitoring-grafana \
  -c grafana \
  --tail=100

helm status dev-monitoring -n dev

Never paste credentials or decoded Kubernetes Secret values into reports.

Part 13 — Cleanup and cost control

After the instructor approves cleanup, delete the deliberately broken application:

kubectl delete deployment broken-restaurant \
  -n dev \
  --ignore-not-found

If this is a disposable, dedicated student cluster, the instructor may authorize removal of the monitoring release:

helm uninstall dev-monitoring -n dev

Helm uninstall does not guarantee deletion of all PVCs or cloud volumes.

Before deleting any PVC, identify its exact PV and EBS volume, verify that it belongs to this lab, and confirm the data is no longer needed.

Do not delete shared namespaces, StorageClasses, EBS CSI drivers, or production volumes.

AWS EBS volumes and EKS resources may incur charges while they exist.

Homework submission requirements

Students must submit a report containing:

AWS EKS cluster and node verification.
dev namespace creation.
Helm monitoring installation output.
Grafana PVC showing Bound.
Grafana HTTP health result.
SMTP environment variable names (no passwords).
Grafana email contact point screenshot.
Successful test notification screenshot.
Broken Kubernetes Pod showing ImagePullBackOff.
Prometheus query result.
Grafana alert showing Firing.
Gmail alert notification screenshot.
Deployment recovery showing Running.
Grafana dashboard persistence verification.

Student reflection questions

Question 1: Why do we use a PVC for Grafana instead of emptyDir?

Question 2: Why is a Kubernetes Secret better than putting an SMTP password directly into Helm values?

Question 3: What is the difference between ImagePullBackOff and CrashLoopBackOff?

Question 4: What does Prometheus do, and what does Grafana do?

Question 5: Why do we configure both an alert rule and a notification policy?

Question 6: Why can an alert be Firing before an email arrives?

Question 7: Why should monitoring alerts include namespace, Pod, and container labels?

Question 8: What are the limitations of storing Grafana's SQLite database on a single EBS volume?

Final expected architecture

Kubernetes application → kube-state-metrics → Prometheus → Grafana alert evaluation → Grafana notification policy → Gmail SMTP → Student email.

Completion criteria: The student must demonstrate a real Kubernetes image-pull failure, receive an alert, fix the Deployment, and verify recovery while Grafana's configuration persists across a Pod restart.

JumpToTech DevOps School ## HOMEWORK — Production Incident Response: Kubernetes HTTP 503

Aisalkyn Aidarova — Mon, 26 Oct 2026 16:46:07 +0000

Course: DevOps + AI Engineering

Topic: Kubernetes Troubleshooting, Service Networking, EndpointSlices, Incident Response

Difficulty: Beginner to Intermediate

Estimated time: 60–90 minutes

Role: DevOps Engineer / Site Reliability Engineer

1. Production Incident Ticket

Ticket ID: INC-503-001

Priority: P2 — High

Status: OPEN

Environment: Production Simulation

Application: Restaurant Company

Namespace: production-lab

Service: restaurant-service

Reported by: Customer Support

Assigned to: DevOps / SRE Team

Customer Complaint

Our customers are unable to access the Restaurant Company application. The website is not responding correctly. Please investigate and restore service as soon as possible.

Business Impact:

Customers may be unable to access the restaurant menu, browse products, or place orders.

Your Responsibility:

You are the on-call DevOps Engineer. Your job is to investigate the incident, identify the root cause, restore application connectivity, verify the recovery, and document your findings.

Important: This lab simulates an application outage. An actual HTTP 503 response is not guaranteed by the Kubernetes Service failure alone. Different network components may return a connection error, timeout, or HTTP 503.

2. Learning Objectives

By the end of this homework, you should be able to:

Understand how Kubernetes Pods, Deployments, Services, labels, selectors, and EndpointSlices work together.
Explain the difference between a healthy Pod and an accessible application.
Investigate a production incident using kubectl.
Identify a Service selector mismatch.
Understand why a Service can have zero endpoints while Pods are running.
Restore connectivity without unnecessarily restarting healthy Pods.
Verify application recovery.
Prepare a professional incident report.

3. Architecture Overview

Our application uses the following architecture:

              Customer / Client
                      |
                      v
              HTTP Request
                      |
                      v
           restaurant-service
              ClusterIP :80
                      |
              Service Routing
                      |
             +--------+--------+
             |                 |
             v                 v
        Restaurant Pod 1  Restaurant Pod 2
           NGINX :80        NGINX :80

Kubernetes components involved:

Deployment: Maintains the desired number of application replicas.

Pods: Run the NGINX application containers.

Service: Provides a stable virtual IP and port for accessing selected backend Pods.

Labels: Identify Kubernetes resources.

Service Selector: Determines which Pods belong to a Service.

EndpointSlice Controller: Maintains EndpointSlice resources that describe matching backend endpoints and their readiness.

kube-proxy: In clusters using kube-proxy, watches Services and EndpointSlices and configures the node's network forwarding rules.

CoreDNS: Resolves Kubernetes Service names into IP addresses. It does not select the backend Pod or forward HTTP requests.

4. Phase 1 — Prepare Your Environment

Before beginning, verify that kubectl is connected to your authorized Kubernetes cluster.

kubectl cluster-info

kubectl get nodes

kubectl get namespaces

Create an isolated namespace for the homework:

kubectl create namespace production-lab

If it already exists, reuse it only if it is your own dedicated training namespace. Do not modify another student's environment.

Verify:

kubectl get namespace production-lab

Question 1: What is a Kubernetes Namespace? Why do companies separate development, staging, and production resources?

Question 2: When a real production incident occurs, do DevOps engineers normally create a new production namespace, or investigate the existing one?

5. Phase 2 — Deploy a Healthy Application

Create a file named:

restaurant-lab.yaml

Add the following YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: restaurant
  namespace: production-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: restaurant
  template:
    metadata:
      labels:
        app: restaurant
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 2
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: restaurant-service
  namespace: production-lab
spec:
  selector:
    app: restaurant
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP

Deploy the application:

kubectl apply -f restaurant-lab.yaml

Wait for the Deployment:

kubectl rollout status deployment/restaurant \
  -n production-lab --timeout=180s

Check the Pods:

kubectl get pods -n production-lab

Check the Service:

kubectl get svc -n production-lab

Check the EndpointSlices:

kubectl get endpointslices -n production-lab \
  -l kubernetes.io/service-name=restaurant-service \
  -o wide

Expected Healthy State

Two NGINX Pods are running and ready.
The Service has a ClusterIP.
The Service has matching backend endpoints.
Requests to the application succeed.

Verify HTTP Access

Use an internal Kubernetes client:

kubectl run restaurant-http-test \
  -n production-lab \
  --rm -i \
  --restart=Never \
  --image=curlimages/curl \
  -- curl -sS -o /dev/null \
  -w 'HTTP Status: %{http_code}\n' \
  http://restaurant-service

Expected:

HTTP Status: 200

Optional Browser Access

If your instructor has configured an EC2 browser demonstration, you may use the instructor-provided EC2 address and port.

Remember that kubectl port-forward is a debugging tunnel. It is not the same as normal Service ClusterIP routing and should not be used as proof that Service endpoints are healthy.

Screenshot Required: Healthy Pods, Service, EndpointSlices, and HTTP 200 verification.

6. Phase 3 — Production Incident Begins

Your instructor or lab partner will introduce a controlled failure.

Students must not read the instructor-only fault-injection command until they have completed their investigation.

Incident Notification

Slack — #production-incidents

ALERT: Restaurant Company application connectivity failure.

Customer Support has reported that customers cannot access the restaurant application.

The Kubernetes cluster is reachable, but application connectivity is failing.

Investigate immediately and provide a root-cause analysis.

Your Mission

You have been assigned this ticket.

Do not immediately delete Pods, restart the Deployment, or recreate the Namespace.

Begin by collecting evidence.

7. Phase 4 — Troubleshooting Investigation

Task A — Check the Pods

kubectl get pods -n production-lab

kubectl get pods -n production-lab -o wide


bash
kub

WEEKEND HOMEWORK — DEVOPS FOUNDATION + KUBERNETES

Aisalkyn Aidarova — Sat, 10 Oct 2026 00:21:07 +0000

This homework is not about memorizing definitions. Your task is to prove that you understand where everything lives, what each component does, and how everything connects together.

Your assignment

Create a 10–15 minute screen recording with your voice.

You may draw the architecture on paper, whiteboard, PowerPoint, Excalidraw, or on your computer.

Do not read definitions from Google, ChatGPT, or your notes during the recording. Explain everything in your own words as if you are answering an interview question.

TASK 1 — Build a Computer

Draw a computer/server:

COMPUTER / SERVER
┌──────────────────────────────┐
│ CPU                          │
│ RAM                          │
│ Storage / SSD                │
│ Network                      │
│ Operating System             │
└──────────────────────────────┘

Explain:

CPU

What is CPU?
What does CPU do?
What is a CPU core?
What does 100m, 500m, 1000m mean in Kubernetes?
Does CPU store files?

RAM

What is RAM?
Why does an application need RAM?
RAM vs Storage?
What can happen when an application runs out of memory?
What does OOMKilled mean?

Storage

What is SSD/storage?
What normally remains after a reboot?
RAM vs persistent storage?

TASK 2 — Explain Operating System

Explain:

What is an Operating System?
Why does a computer need an OS?
What is Linux?
What does the OS manage?
What is the Kernel?
Application vs Process vs Service.
Why do servers commonly run Linux?

You must be able to explain this relationship:

Hardware / Virtual Hardware
        ↓
CPU + RAM + Storage + Network
        ↓
Operating System
        ↓
Applications / Processes

TASK 3 — Computer vs Server vs EC2

Explain:

What is a computer?
What is a server?
Is a server also a computer?
What makes a computer act as a server?
What is AWS EC2?
Is EC2 an Operating System?
Where does Linux run when we create an Ubuntu EC2 instance?
Where are CPU and RAM in this architecture?

Draw:

AWS
 ↓
EC2 Instance
 ↓
CPU + RAM + Storage + Network
 ↓
Linux OS
 ↓
Applications

TASK 4 — Networking

Explain:

What is an IP address?
What is a Private IP?
What is a Public IP?
Private IP vs Public IP.
What is a Port?
What are ports 22, 80, and 443 commonly used for?
What is an AWS Security Group?
Why can an application be running but still not open in your browser?

Explain this:

Your Laptop
     ↓
Internet
     ↓
Public endpoint / Public IP
     ↓
Security Group
     ↓
Port
     ↓
Server
     ↓
Application

TASK 5 — Image vs Container

This section is very important.

Explain:

What is a Docker Image?
What is a Container?
Image vs Container.
Is an Image a running application?
Can we create multiple Containers from the same Image?
What is a Container Runtime?
What is Docker/containerd doing?
Where does an Image come from?

Draw:

Container Registry
ECR / Docker Hub
        ↓
      Image
        ↓
Container Runtime
        ↓
     Container
        ↓
   Application

Use Nginx as your example:

nginx:1.27 Image
       ↓
Container Runtime
       ↓
Nginx Container
       ↓
Nginx Process

Do not say that the Image itself is running.

TASK 6 — Connect This to Kubernetes

Now extend your architecture.

Explain:

What is Kubernetes?
What is EKS?
What is a Cluster?
What is a Control Plane?
What is a Worker Node?
What is a Pod?
Container vs Pod.
Worker Node vs Pod.
Where does the Container run?
Where does the Pod run?
Where are CPU and RAM?

Draw:

EKS CLUSTER

Control Plane
      ↓
Scheduler
      ↓

WORKER NODE / EC2
┌─────────────────────────────┐
│ CPU                         │
│ RAM                         │
│ Storage                     │
│ Network                     │
│ Linux OS                    │
│                             │
│   Pod                       │
│   ┌─────────────────────┐   │
│   │ Container           │   │
│   │                     │   │
│   │ Nginx Application   │   │
│   └─────────────────────┘   │
└─────────────────────────────┘

You must clearly explain:

Worker Node is the server. Pod runs on the Worker Node. Container runs inside the Pod. Application runs inside the Container.

TASK 7 — Explain How Kubernetes Uses an Image

Explain what happens when Kubernetes sees:

containers:
  - name: nginx
    image: nginx:1.27

Explain the complete process:

Deployment / Pod specification
          ↓
image: nginx:1.27
          ↓
Scheduler chooses Worker Node
          ↓
Container Runtime on Node
          ↓
Obtains Image if necessary
          ↓
Creates Container
          ↓
Nginx starts running

Explain the difference between:

Image → Container → Pod → Worker Node

TASK 8 — Deployment and ReplicaSet

Explain:

What is a Deployment?
What is a ReplicaSet?
What are replicas?
What does replicas: 3 mean?
What is desired state?
What happens if you delete one Deployment-managed Pod?
Who recreates it?

Draw:

Deployment
    ↓
ReplicaSet
    ↓
┌─────┬─────┬─────┐
Pod   Pod   Pod

Then explain:

Desired = 3
Actual = 2
     ↓
Controller reconciliation
     ↓
New Pod
     ↓
Actual = 3

TASK 9 — Service

Explain:

Why shouldn't users connect directly to changing Pod IP addresses?
What is a Kubernetes Service?
What is ClusterIP?
What is LoadBalancer?
What are Labels?
What is a Selector?
What are Endpoints?
How does a Service know which Pods belong to it?

Draw:

Users
  ↓
AWS Load Balancer
  ↓
Kubernetes Service
  ↓
selector: app=nginx
  ↓
┌─────┬─────┬─────┐
Pod   Pod   Pod

Explain what happens if:

Service selector = app: nginx

Pod label = app: restaurant

TASK 10 — ConfigMap and Secret

Explain:

What is ConfigMap?
What should we store in ConfigMap?
What is Secret?
ConfigMap vs Secret.
Why shouldn't passwords be hard-coded into application code or normal configuration?
How can a Pod receive a value from ConfigMap or Secret?

Draw:

ConfigMap ──────┐
                ↓
               Pod
                ↑
Secret ─────────┘

TASK 11 — Resources

Explain:

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"

  limits:
    cpu: "500m"
    memory: "256Mi"

Explain:

What is requests?
What is limits?
Why does Scheduler care about requests?
What does 100m CPU mean?
What does 500m CPU mean?
What can happen when CPU reaches its limit?
What can happen when Memory exceeds its limit?

Calculate:

6 Pods
Each requests 100m CPU

Total CPU request = ?

Then:

6 Pods
Each requests 128Mi Memory

Total Memory request = ?

TASK 12 — Readiness and Liveness

Explain the difference:

Readiness
"Should Kubernetes send traffic to this Pod?"

Liveness
"Is this workload healthy enough to keep running?"

Answer:

Can a Pod be Running but 0/1 Ready?
What happens when Readiness fails?
What can happen after repeated Liveness failures?
Why do we need both?

TASK 13 — HPA

Explain:

What is HPA?
What does Horizontal scaling mean?
What is Metrics Server?
Where does HPA get CPU metrics?
What is minReplicas?
What is maxReplicas?
What does averageUtilization: 50 mean?
Why is requests.cpu important for CPU-utilization HPA?
Does HPA directly create Pods?
Does HPA create Worker Nodes?

Draw:

Users / Traffic ↑
       ↓
Service
       ↓
Pods
       ↓
CPU utilization ↑
       ↓
kubelet
       ↓
Metrics Server
       ↓
HPA
       ↓
Desired replicas ↑
       ↓
Deployment / ReplicaSet
       ↓
More Pods

TASK 14 — FULL ARCHITECTURE

This is the most important part of the homework.

Without notes, draw one complete architecture:

USER
 ↓
Browser
 ↓
DNS
 ↓
Public Endpoint / Load Balancer
 ↓
AWS Networking / Security
 ↓
Kubernetes Service
 ↓
Pod
 ↓
Container
 ↓
Nginx Application
 ↓
CPU + RAM


                 EKS CLUSTER

              Control Plane
                    ↓
                 Scheduler
                    ↓

              WORKER NODE
        ┌──────────────────────┐
        │ EC2 Server           │
        │                      │
        │ CPU                  │
        │ RAM                  │
        │ Storage              │
        │ Network              │
        │ Linux OS             │
        │                      │
        │ Pod                  │
        │  └─ Container        │
        │      └─ Nginx        │
        └──────────────────────┘

Then explain the entire journey of one user request.

Example scenario:

A customer opens the Restaurant Company website.

Start with the customer and continue until Nginx returns the response.

Do not skip CPU, RAM, OS, Server, Network, Service, Pod or Container.

TASK 15 — TROUBLESHOOTING INTERVIEW

Imagine the interviewer tells you:

“Our website is down. How would you troubleshoot it?”

Do NOT answer:

“I will restart the Pod.”

You don't know the root cause yet.

Explain your investigation:

Is Worker Node healthy?
        ↓
Are Pods Running?
        ↓
Are Pods Ready?
        ↓
What do Pod Events say?
        ↓
What do application logs say?
        ↓
Is Service correct?
        ↓
Does Service have Endpoints?
        ↓
Do Labels match Selectors?
        ↓
Are Probes passing?
        ↓
Are ConfigMap/Secret correct?
        ↓
Are Resources sufficient?
        ↓
Is Load Balancer working?
        ↓
Networking / Security Group / Ports?
        ↓
DNS?

Mention commands you would use:

kubectl get nodes
kubectl get pods
kubectl describe pod <POD>
kubectl logs <POD>
kubectl get svc
kubectl get endpoints
kubectl get pods --show-labels
kubectl top nodes
kubectl top pods
kubectl get hpa
kubectl describe hpa <HPA>

FINAL SUBMISSION

Submit one 10–15 minute video recording.

Your video must include:

Draw and explain CPU, RAM, Storage, OS and Server.
Explain Private IP vs Public IP.
Explain Image vs Container.
Explain Container vs Pod.
Explain Worker Node vs Pod.
Explain Deployment → ReplicaSet → Pods.
Explain Service → Labels → Selectors → Endpoints.
Explain ConfigMap and Secret.
Explain Requests and Limits.
Explain Readiness and Liveness.
Explain Metrics Server and HPA.
Draw the complete architecture from User → Application.
Explain one troubleshooting scenario.

Important rules

Do not read from notes. Do not just show commands. Do not memorize sentences without understanding them.

For every component, answer:

What is it?
Where does it live?
What does it do?
Why do we need it?
What happens if it fails?

The goal is that on Monday, if an interviewer asks:

“Explain how your application runs in Kubernetes.”

you can draw the architecture and explain the complete system confidently from CPU and Operating System all the way to Kubernetes and HPA.

FINAL KUBERNETES PRODUCTION CAPSTONE Dev Staging Production | Helm | Argo CD | Security | Storage | 10 Production Incidents

Aisalkyn Aidarova — Sat, 10 Oct 2026 00:15:08 +0000

“I built and operated a production-style Kubernetes platform with separate dev, staging, and production environments. I deployed applications with Helm, implemented GitOps with Argo CD, configured resources, probes, autoscaling, availability, RBAC, networking, persistent storage, and practiced production incident response.”

Your role

You have joined a financial company as a DevOps Engineer.

The company is building an online banking platform:

Customers
    ↓
Banking Application
    │
    ├── frontend
    ├── account-service
    ├── payment-service
    └── transaction-service
             ↓
          Database

The company wants Kubernetes environments built using production practices.

You are responsible for designing, deploying, securing, operating, troubleshooting, and explaining the platform.

This is not a copy/paste homework.

Every student must record a video showing the environment and explaining every major decision.

PART 1 — Architecture Design

Before creating anything, draw your architecture.

Your recording must explain:

                         INTERNET
                            │
                            ▼
                    Load Balancer / ALB
                            │
                            ▼
                         Ingress
                            │
                 ┌──────────┼──────────┐
                 ▼          ▼          ▼
             frontend    account    payment
                            │          │
                            └────┬─────┘
                                 ▼
                             Database

Then explain the Kubernetes architecture underneath:

                    KUBERNETES / EKS
                           │
              ┌────────────┴────────────┐
              │                         │
        CONTROL PLANE              WORKER NODES
              │                         │
        API Server                     kubelet
        Scheduler                      runtime
        Controllers                    CNI
        etcd                           kube-proxy
                                        │
                                       Pods

In your recording explain:

What happens from kubectl apply until the container is actually running?

You should be able to explain approximately:

kubectl
   ↓
API Server
   ↓
desired state stored
   ↓
Controllers
   ↓
Scheduler
   ↓
Worker Node selected
   ↓
kubelet
   ↓
CRI
   ↓
container runtime
   ↓
Container starts
   ↓
CNI networking
   ↓
Readiness
   ↓
Service can route traffic

Do not just read component definitions.

Explain the process.

PART 2 — Build Three Environments

The company requires:

DEV
STAGING
PRODUCTION

For this homework, you can implement them as namespaces:

kubectl create namespace dev
kubectl create namespace staging
kubectl create namespace production

Verify:

kubectl get namespaces

But explain an important production point

In your recording answer:

“Would a bank necessarily run dev, staging and production merely as three namespaces in one cluster?”

Expected explanation:

Not necessarily.

For stronger isolation, especially in regulated/security-sensitive organizations, production may use separate clusters and often separate AWS accounts/VPC boundaries.

For example:

AWS Organization
│
├── Development Account
│      └── Dev EKS
│
├── Staging / Pre-Prod Account
│      └── Staging EKS
│
└── Production Account
       └── Production EKS

Namespaces are acceptable for our training environment because creating several full EKS environments costs additional money and operational effort.

But students must understand the difference between:

Lab architecture and real production architecture.

PART 3 — Build the Application

Create a Deployment for the application.

Requirements:

Application: banking-app
Production replicas: 3
Container: nginx or your own application
Container port: 80

Your Deployment must include:

resources:
  requests:
    cpu:
    memory:
  limits:
    cpu:
    memory:

and:

readinessProbe:

and:

livenessProbe:

and a production rolling strategy:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

Students must decide appropriate values and explain them.

PART 4 — Explain Requests and Limits

In the recording answer:

Why do we use requests?

Why do we use limits?

What happens when memory exceeds its limit?

What can happen when CPU reaches/exceeds its limit?

Why should we not blindly give every Pod huge requests?

Explain:

Requests
   ↓
Scheduling / reserved expectation

Limits
   ↓
Maximum resource boundary

Also explain why bad resource configuration can create:

Pending Pods
CPU throttling
OOMKilled
Wasted cluster capacity
Higher AWS cost

PART 5 — Probes

Implement:

Startup Probe (optional if appropriate)
Readiness Probe
Liveness Probe

Explain:

Readiness:
Should this Pod receive traffic?

Liveness:
Should Kubernetes restart the container?

Startup:
Has this slow-starting application successfully started?

Demonstrate what happens when readiness fails.

PART 6 — Service

Create:

ClusterIP Service

Explain:

Why shouldn't another application communicate directly with a Pod IP?

Explain:

Service
   ↓
Selector
   ↓
Pod labels
   ↓
EndpointSlices
   ↓
Ready Pods

Show:

kubectl get svc -n production
kubectl get endpointslice -n production
kubectl get pods -n production --show-labels

PART 7 — Ingress

Create an Ingress for the application.

Conceptually:

Customer
   ↓
DNS
   ↓
Load Balancer
   ↓
Ingress Controller
   ↓
Ingress Rules
   ↓
Service
   ↓
Pods

In your recording explain:

Ingress vs Ingress Controller

Ingress vs Service

Where HTTPS/TLS could terminate

How traffic reaches the application

For AWS/EKS, explain how an AWS load-balancing solution can integrate with Kubernetes rather than claiming the Ingress resource itself magically creates everything.

PART 8 — ConfigMap and Secret

Create a ConfigMap containing something such as:

APP_ENV=production
LOG_LEVEL=info

Create a Secret containing a dummy training credential.

Application must consume configuration through environment variables or mounted files.

Explain:

ConfigMap vs Secret

And answer:

“Is base64 encryption?”

Correct answer:

No.

For the banking scenario, discuss why real secrets should be managed carefully and why cloud secret-management/encryption solutions may be integrated rather than committing credentials to Git.

Never commit real secrets to the repository.

PART 9 — RBAC / Least Privilege

Create:

ServiceAccount
Role
RoleBinding

Create a developer ServiceAccount.

It should be allowed to:

get Pods
list Pods
watch Pods

It should not be allowed to:

delete Pods
delete Deployments
delete Secrets

Test:

kubectl auth can-i list pods \
--as=system:serviceaccount:production:developer \
-n production

Then:

kubectl auth can-i delete pods \
--as=system:serviceaccount:production:developer \
-n production

Explain:

Authentication
=
Who are you?

Authorization
=
What are you allowed to do?

Then explain why least privilege is especially important for a banking application.

PART 10 — NetworkPolicy

Create a production NetworkPolicy.

Architecture:

frontend
   │
   ▼
payment-service
   │
   ▼
database

Allowed:

frontend → payment ✓
payment → database ✓

Not allowed:

frontend → database ✗
random Pod → database ✗

Students must explain why RBAC cannot solve this problem.

Expected:

RBAC
→ Kubernetes API authorization

NetworkPolicy
→ workload network communication

Also explain that NetworkPolicy requires a network implementation that enforces it.

PART 11 — HPA

Configure HPA.

Example requirements:

minimum replicas: 3
maximum replicas: 10
CPU target: 60%

Students must show:

kubectl get hpa -n production
kubectl describe hpa -n production

Explain:

What happens when traffic increases?

And:

What happens if HPA reaches maxReplicas but traffic continues increasing?

Also answer:

What does CPU-based HPA need?

Metrics and appropriate resource requests.

PART 12 — PDB

Create a PodDisruptionBudget.

For example:

3 replicas

minAvailable: 2

Explain why this matters during voluntary disruption such as node maintenance.

Also explain:

Does PDB protect against every failure?

No.

PART 13 — Scheduling

Demonstrate or explain:

nodeSelector
node affinity
pod anti-affinity
taints
tolerations

For production, explain why we might avoid placing every replica on the same node.

Bad:

Node 1
├── payment-1
├── payment-2
└── payment-3

Node 1 fails
→ payment completely unavailable

Better:

Node 1 → payment-1
Node 2 → payment-2
Node 3 → payment-3

Discuss topology/AZ resilience where appropriate.

PART 14 — StatefulSet + Persistent Storage

Create a StatefulSet.

It must demonstrate:

Stable Pod identity
PVC
StorageClass
PV
Persistent data

Show:

kubectl get statefulset
kubectl get pvc
kubectl get pv
kubectl get storageclass

Write data.

Delete the Pod.

Allow StatefulSet to recreate it.

Prove the data still exists.

Then explain:

Stable identity and persistent storage are related but different concepts.

PART 15 — AWS EBS CSI

Explain:

StatefulSet
    ↓
PVC
    ↓
StorageClass
    ↓
EBS CSI Driver
    ↓
AWS EBS
    ↓
PV
    ↓
PVC Bound
    ↓
Pod Running

Students must explain the storage incident we previously encountered:

PVC Pending
↓
StorageClass existed
↓
ebs.csi.aws.com expected
↓
EBS CSI add-on/components missing
↓
No dynamic EBS volume
↓
Pod Pending

And how to investigate it.

PART 16 — Helm

Convert the application into a Helm chart.

Suggested structure:

banking-app/
│
├── Chart.yaml
├── values.yaml
├── values-dev.yaml
├── values-staging.yaml
├── values-prod.yaml
│
└── templates/
    ├── deployment.yaml
    ├── service.yaml
    ├── ingress.yaml
    ├── configmap.yaml
    ├── serviceaccount.yaml
    ├── hpa.yaml
    ├── pdb.yaml
    └── networkpolicy.yaml

Different environment values should demonstrate meaningful differences.

For example:

DEV
replicas: 1
smaller resources

STAGING
replicas: 2
production-like configuration

PRODUCTION
replicas: 3+
HPA
PDB
stronger availability settings

Demonstrate:

helm lint
helm template
helm upgrade --install
helm list
helm history

Explain why Helm is useful in a company with many applications and environments.

PART 17 — Argo CD / GitOps

Put Kubernetes/Helm configuration in Git.

Architecture:

Developer
   ↓
Pull Request
   ↓
Review
   ↓
Merge
   ↓
Git
   ↓
Argo CD
   ↓
EKS

Students must create/configure an Argo CD Application and demonstrate:

Synced
Healthy

Then manually modify something in Kubernetes.

Observe:

OutOfSync

Explain drift.

Then restore/reconcile through GitOps.

Critical production question:

“If Argo CD manages production, should engineers routinely modify production using kubectl edit?”

Expected:

Generally no. Desired changes should normally go through the controlled Git/change process, with emergency procedures handled according to company policy.

PART 18 — Banking Production Requirements

This section is mandatory in the recording.

Imagine this is a banking system handling financial transactions and sensitive data.

Before saying “production ready,” discuss:

High availability
Least privilege
Encryption
Secret management
Network segmentation
Auditability
Backups
Disaster recovery
Monitoring
Alerting
Logging
Change control
Image scanning
Vulnerability management
Patch management
Capacity planning
Rollback strategy
Business continuity
Compliance requirements

Do not simply say:

“We have three replicas, therefore the bank is production-ready.”

Production readiness is much larger than Kubernetes manifests.

PART 19 — What Costs Money in AWS?

Students must be able to answer this because engineers should understand cost.

Discuss potential charges for things such as:

EKS cluster/control-plane service
EC2 worker nodes
EBS volumes/snapshots
Load balancers
NAT Gateway
Data transfer
CloudWatch logs/metrics
ECR storage/data transfer
Public IPv4 where applicable
Route 53
Backup/storage services
Other managed AWS services

Do not memorize exact prices because pricing changes.

Instead explain:

“Which architectural decisions create cost?”

For example:

More nodes → more compute cost

More/larger EBS → more storage cost

Multiple ALBs → load-balancer cost

Large log ingestion → observability cost

NAT traffic → networking cost

PART 20 — Production Monitoring

Your application must have an observability plan.

Explain what you would monitor:

CPU
Memory
Pod restarts
Ready replicas
Node health
Request rate
Error rate
Latency
HTTP 5xx
HPA behavior
PVC/storage
Database health
Dependency health

And explain:

Metrics
Logs
Traces

A strong student should mention service-level signals such as:

Traffic
Errors
Latency
Saturation

not only Kubernetes CPU.

PART 21 — TEN REQUIRED PRODUCTION INCIDENTS

This is the most important part of the homework.

Each student must break the environment intentionally, investigate it without jumping directly to the answer, fix/mitigate it, and verify recovery on video.

Incident 1 — Bad Deployment / ImagePullBackOff

Break:

Deploy an invalid/nonexistent image tag.

Student must discover:

New rollout
↓
New Pods unhealthy
↓
ImagePullBackOff
↓
describe Pod
↓
Events
↓
Image cannot be pulled

Mitigate:

kubectl rollout history deployment/<name> -n production
kubectl rollout undo deployment/<name> -n production
kubectl rollout status deployment/<name> -n production

Interview explanation:

“A new release introduced an invalid image reference. I correlated the incident with the rollout, confirmed the image pull failure from Pod events, rolled back to the known-good revision, and verified service recovery.”

Incident 2 — Production Slow / HPA at Maximum

Generate load.

Students investigate:

Application slow
↓
Pods Running
↓
CPU/resource pressure
↓
HPA target exceeded
↓
maxReplicas reached
↓
capacity insufficient for current load

They must not say “traffic” without evidence.

Incident 3 — Service Selector Mismatch

Break Service selector:

Service:
app=wrong-app

Pods:
app=banking-app

Symptoms:

Pods 1/1 Running
Service exists
Application unreachable
No expected Service backend

Student must compare:

kubectl describe svc
kubectl get pods --show-labels
kubectl get endpointslice

Incident 4 — Readiness Probe Failure

Break:

readinessProbe:
  httpGet:
    path: /does-not-exist

Expected:

Running 0/1

Student must explain:

Running ≠ Ready.

Use describe to find probe failure.

Fix/rollback and verify readiness and Service routing.

Incident 5 — OOMKilled

Configure a workload with insufficient memory or a controlled memory-hungry training application.

Student must find:

RESTARTS ↑
↓
CrashLoopBackOff
↓
describe
↓
Last State
↓
OOMKilled

Then:

kubectl logs <pod> --previous

Explain why simply increasing memory may only be mitigation if the application has unbounded memory growth.

Incident 6 — Dependency Failure

Application logs:

Cannot connect to database/Kafka

Student is not allowed to say:

“Kafka is down.”

until proven.

Investigate:

DNS
Service
EndpointSlice
Port
NetworkPolicy
Authentication
Dependency health
Application configuration

Interview statement:

“The log proves the application cannot connect to the dependency; it does not by itself prove the dependency is down.”

Incident 7 — PVC Pending

Break storage provisioning.

Expected:

Pod Pending
↓
PVC Pending
↓
Events
↓
StorageClass
↓
CSI
↓
Cloud permissions/storage

Students must walk the entire storage dependency chain.

Incident 8 — RBAC Forbidden

Make a ServiceAccount attempt:

kubectl delete pod ...

Expected:

Forbidden

Student investigates:

kubectl auth can-i ...

Then Role/RoleBinding.

They must explain:

Authentication succeeded
but
Authorization denied

A Forbidden response is different from a network failure.

Incident 9 — NetworkPolicy Blocks Traffic

Create:

frontend → payment ✗

because NetworkPolicy blocks the connection.

Students must prove:

Pods Running
Service exists
Endpoints exist
DNS may work
BUT network connection blocked

Then inspect:

kubectl get networkpolicy
kubectl describe networkpolicy

They must explain why restarting the application doesn't fix the policy.

Incident 10 — Argo CD OutOfSync / Configuration Drift

Git says:

replicas: 3

Manually change live cluster:

kubectl scale deployment banking-app \
--replicas=1 \
-n production

Observe Argo CD.

Students explain:

Git desired state
      ≠
Live state

→ OutOfSync

Restore desired state through the GitOps workflow/reconciliation.

Explain why uncontrolled manual production changes are dangerous.

PART 22 — For Every Incident, Use the Same Professional Method

Students may not begin with random commands.

Every recording must follow:

1. ALERT / TICKET

2. CUSTOMER IMPACT
   What is broken?

3. SCOPE
   Which application/service/environment?

4. WHAT CHANGED?
   Deployment?
   Config?
   Traffic?
   Infrastructure?

5. OBSERVE
   Metrics
   Logs
   Events
   Traces

6. HYPOTHESIS
   What might explain the evidence?

7. PROVE / DISPROVE
   Collect evidence.

8. ROOT CAUSE
   What actually caused the problem?

9. MITIGATION / FIX
   Restore service safely.

10. VERIFY
    Prove service is healthy.

11. RCA
    Why did this happen?
    How do we prevent recurrence?

The sentence I want every student to learn is:

“I don't want to guess the root cause. I form a hypothesis and validate it with evidence.”

PART 23 — Production Verification

After every fix, students must prove recovery.

They cannot say:

“The command succeeded, so production is fixed.”

They must check the relevant layers:

Deployment healthy
       ↓
Pods Ready
       ↓
Restart count stable
       ↓
Service backend healthy
       ↓
Application responds
       ↓
Errors normal
       ↓
Latency normal
       ↓
Critical customer flow works

PART 24 — Git Repository Requirements

Repository should look professional.

For example:

banking-platform/
│
├── README.md
│
├── docs/
│   ├── architecture.md
│   ├── troubleshooting.md
│   └── incidents.md
│
├── helm/
│   └── banking-app/
│       ├── Chart.yaml
│       ├── values.yaml
│       ├── values-dev.yaml
│       ├── values-staging.yaml
│       ├── values-prod.yaml
│       └── templates/
│
├── argocd/
│   ├── dev.yaml
│   ├── staging.yaml
│   └── production.yaml
│
└── kubernetes/
    ├── rbac/
    ├── network-policy/
    ├── storage/
    └── troubleshooting/

README should explain architecture, environments, deployment process, security controls, availability, scaling, storage, GitOps and troubleshooting.

PART 25 — Required Video Recording

I would require approximately 45–60 minutes, but quality matters more than exact duration.

Tell students:

Do not read definitions from Google or from your notes. Share your screen and explain your own environment as though I am the interviewer.

The recording must show and explain the full lifecycle:

Architecture
    ↓
AWS / EKS
    ↓
Dev / Staging / Production
    ↓
Docker Image / ECR
    ↓
Deployment / ReplicaSet / Pods
    ↓
Resources / Probes
    ↓
Service / EndpointSlice
    ↓
Ingress
    ↓
ConfigMap / Secret
    ↓
RBAC
    ↓
NetworkPolicy
    ↓
HPA / PDB
    ↓
Scheduling
    ↓
StatefulSet / PV / PVC / StorageClass / CSI
    ↓
Helm
    ↓
Argo CD / GitOps
    ↓
Observability
    ↓
10 Production Incidents
    ↓
RCA / Verification

At random points in the recording they should be able to answer:

Why did you configure it this way?

That is more important than merely showing YAML.

JumpToTech — Weekend Kubernetes Production Lab

Aisalkyn Aidarova — Sat, 10 Oct 2026 00:02:17 +0000

Restaurant Company: Deploy, Configure, Scale & Troubleshoot

Goal

You are the DevOps Engineer responsible for deploying the Restaurant Company application into Kubernetes.

By the end of this lab, you should be able to explain and demonstrate:

EKS Cluster
    ↓
Worker Node
    ↓
Deployment
    ↓
ReplicaSet
    ↓
Pods
    ↓
Service
    ↓
Application

ConfigMap + Secret
        ↓
       Pods

Resources
CPU + Memory
        ↓
       Pods

Metrics Server
        ↓
       HPA
        ↓
Deployment replicas

Do not simply copy commands. After every section, understand what Kubernetes created and why.

Part 1 — Verify Your Kubernetes Environment

Before creating anything:

kubectl get nodes

Your Worker Node should show:

STATUS
Ready

Run:

kubectl get nodes -o wide

Then:

kubectl describe node <NODE-NAME>

Find:

Capacity:
  cpu:
  memory:
  pods:

Allocatable:
  cpu:
  memory:
  pods:

Questions

Students must answer:

What is a Worker Node?
What is Capacity?
What is Allocatable?
Why can Allocatable be smaller than Capacity?
How many Pods can your Node run?
Is Pod capacity the same thing as CPU capacity?
What does 1930m CPU mean if you see it?

Remember:

1000m = 1 CPU core
500m  = 0.5 CPU
100m  = 0.1 CPU

Part 2 — Create Your Working Directory

mkdir -p ~/weekend-k8s-lab
cd ~/weekend-k8s-lab

Verify:

pwd

You will create:

weekend-k8s-lab/
├── configmap.yaml
├── secret.yaml
├── deployment.yaml
├── service.yaml
└── hpa.yaml

Part 3 — ConfigMap

Create:

nano configmap.yaml

Add:

apiVersion: v1
kind: ConfigMap
metadata:
  name: restaurant-config
data:
  APP_ENV: "production"
  APP_NAME: "restaurant-company"

Apply:

kubectl apply -f configmap.yaml

Verify:

kubectl get configmap

kubectl describe configmap restaurant-config

Understand

A ConfigMap stores non-sensitive configuration separately from the application image.

Examples:

APP_ENV
APP_NAME
LOG_LEVEL
API_URL

Interview Question

Why shouldn't configuration always be hard-coded into the Docker image?

Part 4 — Secret

Create:

nano secret.yaml

For this classroom lab only:

apiVersion: v1
kind: Secret
metadata:
  name: restaurant-secret
type: Opaque
stringData:
  DB_PASSWORD: "JumpToTech123"

Apply:

kubectl apply -f secret.yaml

Check:

kubectl get secret

kubectl describe secret restaurant-secret

You may see:

Data
====
DB_PASSWORD:  13 bytes

Important

Do not commit real production passwords to GitHub.

Also remember:

Kubernetes Secret is not automatically "safe because it is a Secret." Base64 encoding is not encryption.

Part 5 — Deployment

Create:

nano deployment.yaml

Use:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment

spec:
  replicas: 2

  selector:
    matchLabels:
      app: nginx

  template:
    metadata:
      labels:
        app: nginx

    spec:
      containers:
        - name: nginx
          image: nginx:1.27

          ports:
            - containerPort: 80

          env:
            - name: APP_ENV
              valueFrom:
                configMapKeyRef:
                  name: restaurant-config
                  key: APP_ENV

            - name: APP_NAME
              valueFrom:
                configMapKeyRef:
                  name: restaurant-config
                  key: APP_NAME

            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: restaurant-secret
                  key: DB_PASSWORD

          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"

            limits:
              cpu: "500m"
              memory: "256Mi"

          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 10

          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 20

Apply:

kubectl apply -f deployment.yaml

Check:

kubectl get deployment

kubectl get pods

kubectl get rs

Students should understand:

Deployment
     ↓
ReplicaSet
     ↓
Pods

Part 6 — Prove Self-Healing

Get Pods:

kubectl get pods

Delete ONE Deployment-managed Pod:

kubectl delete pod <POD-NAME>

Immediately:

kubectl get pods -w

Observe

You deleted one Pod.

But the desired state says:

replicas: 2

Therefore:

Desired = 2
Actual  = 1

ReplicaSet
    ↓
creates replacement
    ↓
Actual = 2

Interview Question

Why did Kubernetes recreate the Pod?

Do not answer:

"Because Kubernetes always recreates Pods."

Correct idea:

The Pod is managed by a ReplicaSet created by the Deployment, and the controller reconciles actual state with desired state.

Part 7 — Verify ConfigMap and Secret Injection

Find a Pod:

kubectl get pods

Check ConfigMap values:

kubectl exec -it <POD-NAME> -- printenv APP_ENV

Expected:

production

Then:

kubectl exec -it <POD-NAME> -- printenv APP_NAME

Expected:

restaurant-company

For this classroom lab only, verify that the Secret variable exists:

kubectl exec -it <POD-NAME> -- printenv DB_PASSWORD

Do not expose production secrets this way.

Part 8 — Understand Resources

Your Deployment contains:

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"

  limits:
    cpu: "500m"
    memory: "256Mi"

Requests

Requests are especially important to scheduling.

New Pod
   ↓
CPU request = 100m
Memory request = 128Mi
   ↓
Scheduler
   ↓
Find suitable Node

Limits

Limits constrain resource usage.

CPU limit exceeded
→ CPU may be throttled

Memory limit exceeded
→ Container may be terminated / OOMKilled

Verify:

kubectl describe pod <POD-NAME>

Find:

Requests:
  cpu:     100m
  memory:  128Mi

Limits:
  cpu:     500m
  memory:  256Mi

Calculation

If there are 6 Pods:

6 × 100m CPU = 600m requested CPU

6 × 128Mi = 768Mi requested Memory

Part 9 — Readiness Probe

Your Deployment contains:

readinessProbe:
  httpGet:
    path: /
    port: 80

Readiness asks:

Should Kubernetes send normal Service traffic to this Pod?

Now intentionally break it.

Change:

path: /

to:

path: /wrong-page

Apply:

kubectl apply -f deployment.yaml

Watch:

kubectl get pods -w

You may see:

READY
0/1

But:

STATUS
Running

Important Interview Question

Can a Pod be Running but not Ready?

YES.

Investigate:

kubectl describe pod <POD-NAME>

Look at:

Events:

Now fix:

path: /

Apply again:

kubectl apply -f deployment.yaml

Watch:

kubectl get pods -w

Expected eventually:

1/1 Running

Part 10 — Liveness Probe

Your Deployment contains:

livenessProbe:
  httpGet:
    path: /
    port: 80

Remember:

Readiness
→ Should this workload receive traffic?

Liveness
→ Is this workload healthy enough to keep running?

Repeated liveness failures can cause kubelet to restart the Container.

Interview Question

Explain the difference between:

Readiness Probe
vs
Liveness Probe

in your own words.

Part 11 — Create a Service

Create:

nano service.yaml

Add:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service

spec:
  selector:
    app: nginx

  ports:
    - port: 80
      targetPort: 80

  type: ClusterIP

Apply:

kubectl apply -f service.yaml

Check:

kubectl get svc

Then:

kubectl get endpoints nginx-service

You should see Pod IP addresses.

Architecture:

Client
   ↓
nginx-service
   ↓
selector:
app: nginx
   ↓
matching Pods

Important

The Service does not randomly discover your application.

It uses:

selector:
  app: nginx

and finds Pods with:

labels:
  app: nginx

Part 12 — Break the Service

Now create a real incident.

Change:

selector:
  app: nginx

to:

selector:
  app: wrong-label

Apply:

kubectl apply -f service.yaml

Check:

kubectl get endpoints nginx-service

You may see no endpoints.

Now troubleshoot:

kubectl get pods --show-labels

Compare:

Service selector
vs
Pod labels

Fix:

selector:
  app: nginx

Apply:

kubectl apply -f service.yaml

Check again:

kubectl get endpoints nginx-service

Interview Scenario

Pods are Running, but the Service has no endpoints. What do you check?

Answer:

Pod labels and Service selector.

Part 13 — Metrics Server

Now check resource usage:

kubectl top nodes

Then:

kubectl top pods

Understand the difference:

REQUEST
100m CPU

≠

ACTUAL USAGE
for example 5m CPU

≠

LIMIT
500m CPU

Three different concepts.

Architecture:

Pod / Container
      ↓
    kubelet
      ↓
Metrics Server
      ↓
Kubernetes Metrics API
      ↓
HPA

Part 14 — Create HPA

Create:

nano hpa.yaml

Add:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa

spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-deployment

  minReplicas: 2
  maxReplicas: 8

  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

Apply:

kubectl apply -f hpa.yaml

Check:

kubectl get hpa

You may see:

TARGETS
cpu: 0%/50%

Interpret:

0% = current CPU utilization
50% = target

Also:

kubectl describe hpa nginx-hpa

Part 15 — Understand HPA Before Testing It

HPA does not simply mean:

CPU reaches exactly 50%
→ create exactly 1 Pod

Instead, HPA evaluates the metric and calculates a desired replica count.

Also:

HPA
 ↓
changes desired replicas
 ↓
Deployment / ReplicaSet
 ↓
Pods are created/removed

For this CPU-utilization HPA, your CPU request matters:

requests:
  cpu: "100m"

Very simplified example:

CPU request = 100m

actual usage 10m
≈ 10%

actual usage 50m
≈ 50%

actual usage 80m
≈ 80%

Part 16 — Generate Traffic

Make sure:

kubectl get svc nginx-service

Then create a load generator:

kubectl run load-generator \
  --image=busybox:1.36 \
  --restart=Never \
  -- /bin/sh -c \
  'for i in 1 2 3 4 5 6 7 8 9 10; do while true; do wget -q -O- http://nginx-service > /dev/null; done & done; wait'

Now use multiple terminals.

Terminal 1

kubectl get hpa -w

Terminal 2

kubectl get pods -w

Terminal 3

kubectl top pods

Observe:

Load Generator
      ↓
HTTP requests
      ↓
Service
      ↓
Nginx Pods
      ↓
CPU utilization ↑
      ↓
Metrics Server
      ↓
HPA
      ↓
Desired replicas ↑
      ↓
Deployment / ReplicaSet
      ↓
More Pods

Do not expect exact CPU percentages or exact scaling timing to be identical on every student's cluster.

Part 17 — Stop Traffic

Delete:

kubectl delete pod load-generator

Watch:

kubectl get hpa -w

CPU usage should decrease.

Eventually HPA can reduce replicas toward:

minReplicas = 2

Scale-down may not happen immediately.

Part 18 — Production Incident: Pending Pods

This is based on a real incident from class.

Suppose:

Node Pod Capacity = 29
Current Pods      = 29

HPA wants additional replicas.

What happens?

HPA
 ↓
Desired replicas ↑
 ↓
ReplicaSet creates new Pod objects
 ↓
Scheduler tries to place them
 ↓
Node = 29/29 Pods
 ↓
No available Pod slot
 ↓
Pending

Investigate:

kubectl get pods

Then:

kubectl describe pod <PENDING-POD>

Look at:

Events:

You might find:

FailedScheduling
Too many pods

Important

This is different from:

Insufficient cpu

or:

Insufficient memory

Do not guess the root cause.

Read the Events.

Part 19 — Kubernetes Troubleshooting Challenge

For each scenario, write:

1. Symptom
2. Command used
3. Root cause
4. Fix
5. Verification

Incident A

Pod = Pending

Start:

kubectl describe pod <POD>

Check Events.

Incident B

Pod = Running
READY = 0/1

Think:

Readiness Probe

Incident C

Pods Running
Service exists
Endpoints = none

Think:

Service selector
vs
Pod labels

Incident D

Container keeps restarting

Start:

kubectl describe pod <POD>
kubectl logs <POD>

Consider liveness failures, application errors, OOMKilled, etc.

Incident E

HPA shows unknown CPU metric

Check:

kubectl top pods
kubectl describe hpa nginx-hpa

Also verify the workload has appropriate CPU requests for utilization-based CPU HPA.

Part 20 — Commands You Must Know

By Monday, students should be comfortable with:

kubectl get nodes
kubectl describe node <NODE>

kubectl get pods
kubectl get pods -o wide
kubectl get pods -w
kubectl describe pod <POD>
kubectl logs <POD>

kubectl get deployment
kubectl describe deployment nginx-deployment
kubectl get rs

kubectl get svc
kubectl describe svc nginx-service
kubectl get endpoints nginx-service

kubectl get configmap
kubectl get secret

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

kubectl top nodes
kubectl top pods

kubectl get hpa
kubectl get hpa -w
kubectl describe hpa nginx-hpa

Final Architecture — Draw This Yourself

Every student must be able to draw and explain this without notes:

                    USERS
                      ↓
              Load Balancer
                      ↓
                   Service
                      ↓
          ┌───────────┼───────────┐
          ↓           ↓           ↓
        Pod 1       Pod 2       Pod 3
          │           │           │
          └──── Deployment ───────┘
                    ↓
               ReplicaSet


ConfigMap ─────────→ Pods
Secret ────────────→ Pods


Worker Node
├── CPU
├── Memory
├── Storage
├── Networking
└── Pod Capacity


Pods
 ↓
CPU / Memory usage
 ↓
kubelet
 ↓
Metrics Server
 ↓
HPA
 ↓
Deployment desired replicas
 ↓
ReplicaSet
 ↓
More / Fewer Pods

Weekend Interview Questions

Students should prepare answers to these without reading from notes:

What is Kubernetes?
What is Amazon EKS?
What is the Control Plane?
What is a Worker Node?
What is a Pod?
What is a Container?
What does kubectl do?
What does the Scheduler do?
What is a Deployment?
What is a ReplicaSet?
What does replicas: 3 mean?
What is desired state?
What is reconciliation?
What happens when you delete a Deployment-managed Pod?
Why can Pod IP addresses change?
What problem does a Service solve?
What is ClusterIP?
What is LoadBalancer?
How does a Service find Pods?
What are labels?
What is a selector?
What are endpoints?
What happens when Service selector does not match Pod labels?
What is a ConfigMap?
What is a Secret?
ConfigMap vs Secret?
What are resource requests?
What are resource limits?
Who uses requests during scheduling?
What is 100m CPU?
What is 1000m CPU?
What can happen when CPU reaches its limit?
What can happen when Memory exceeds its limit?
What does OOMKilled mean?
What is a Readiness Probe?
What is a Liveness Probe?
Can a Pod be Running but 0/1 Ready?
What is Metrics Server?
What does kubectl top pods show?
What is HPA?
What does Horizontal scaling mean?
What is minReplicas?
What is maxReplicas?
What does averageUtilization: 50 mean?
Why are CPU requests important for CPU-utilization HPA?
Does HPA directly create Pods?
Does HPA create Worker Nodes?
What happens if HPA wants more Pods but the Node is full?
What does Pending mean?
How do you find why a Pod is Pending?
What does FailedScheduling mean?
What does Too many pods mean?
What's the difference between CPU capacity and Pod capacity?
What do you check when Pods are Running but Service has no endpoints?
What commands would you use if an application is down?

Required Submission

Each student should submit screenshots showing:

1. kubectl get nodes
2. kubectl get pods
3. kubectl get deployment
4. kubectl get rs
5. kubectl get svc
6. kubectl get endpoints nginx-service
7. kubectl get configmap
8. kubectl get secret
9. kubectl describe pod showing Requests/Limits
10. kubectl top nodes
11. kubectl top pods
12. kubectl get hpa
13. HPA before load
14. Load generator Running
15. HPA during CPU load
16. Additional Nginx Pods created by scaling
17. HPA after load is removed
18. One troubleshooting example and its Events

Production Kubernetes Troubleshooting Lab

Aisalkyn Aidarova — Thu, 08 Oct 2026 01:37:58 +0000

Observability, Incident Response & Root-Cause Analysis

Scenario: You are the DevOps/SRE on-call team for an e-commerce company.

Architecture:

                    CUSTOMER
                        │
                        ▼
                   order-service
                        │
                        ▼
                    Service
                        │
              ┌─────────┼─────────┐
              ▼         ▼         ▼
            Pod-1     Pod-2     Pod-3
                        │
                        ▼
                  Dependencies

Today we will deliberately create these incidents:

INCIDENT 1 → New deployment → CrashLoopBackOff → Rollback

INCIDENT 2 → Production slow → High traffic / CPU → HPA investigation

INCIDENT 3 → Running Pods but application unavailable → Service selector

INCIDENT 4 → Running 0/1 → Readiness failure

INCIDENT 5 → OOMKilled → Memory investigation

INCIDENT 6 → Application dependency problem → Logs

LAB 0 — Check the Cluster

Everybody starts here.

kubectl cluster-info

Then:

kubectl get nodes

Expected:

NAME                         STATUS   ROLES    AGE
ip-192-168-10-10.ec2...      Ready    <none>   ...
ip-192-168-20-20.ec2...      Ready    <none>   ...

Check:

kubectl get nodes -o wide

Now create our production namespace:

kubectl create namespace production

Check:

kubectl get namespace production

LAB 1 — Build Healthy Production First

Before troubleshooting production, we need healthy production.

Create:

nano production.yaml

Paste:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: production
spec:
  replicas: 3

  selector:
    matchLabels:
      app: order-service

  template:
    metadata:
      labels:
        app: order-service

    spec:
      containers:
      - name: order-service
        image: nginx:1.27

        ports:
        - containerPort: 80

        resources:
          requests:
            cpu: "100m"
            memory: "64Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5

        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 10
          periodSeconds: 10

---
apiVersion: v1
kind: Service
metadata:
  name: order-service
  namespace: production
spec:
  selector:
    app: order-service

  ports:
  - port: 80
    targetPort: 80

  type: ClusterIP

Apply:

kubectl apply -f production.yaml

Watch:

kubectl get pods -n production -w

Eventually:

NAME                             READY   STATUS
order-service-xxxxxxxxxx-abcde   1/1     Running
order-service-xxxxxxxxxx-fghij   1/1     Running
order-service-xxxxxxxxxx-klmno   1/1     Running

Press:

Ctrl+C

Check the Entire Production Environment

kubectl get all -n production

Check labels:

kubectl get pods -n production --show-labels

You should see:

app=order-service

Check Service:

kubectl get svc -n production

Now:

kubectl get endpoints order-service -n production

Depending on Kubernetes version, you may also use:

kubectl get endpointslice -n production

You should have backend addresses.

Concept:

Service
   │
   │ selector:
   │ app=order-service
   │
   ▼
Pod        Pod        Pod
app=       app=       app=
order      order      order
-service   -service   -service

Everything matches.

Production is healthy.

Verify the Application

Create a temporary troubleshooting Pod:

kubectl run test-client \
  --image=curlimages/curl \
  --restart=Never \
  -n production \
  -- sleep 3600

Check:

kubectl get pod test-client -n production

Then:

kubectl exec -n production test-client -- \
curl -s http://order-service

You should receive the Nginx HTML page.

Now tell students:

This is our known-good production baseline.

That's important.

INCIDENT 1 — New Deployment Breaks Production

STUDENTS ONLY RECEIVE THIS MESSAGE

🚨 INCIDENT: Customers cannot place orders. The Order Service started failing immediately after today's deployment.

Ask students:

What changed?

Answer:

New deployment.

First check:

kubectl get pods -n production

But currently everything works.

INSTRUCTOR — Break Production

Don't show students the command.

Run:

kubectl set image deployment/order-service \
order-service=nginx:this-version-does-not-exist \
-n production

Now students run:

kubectl get pods -n production

They should eventually see something similar to:

order-service-old        1/1   Running
order-service-new        0/1   ImagePullBackOff
order-service-new        0/1   ImagePullBackOff

Important teaching point:

Because this is a rolling update, old healthy Pods may remain available while new Pods fail. So production may be degraded rather than completely unavailable.

Student Investigation

First:

kubectl rollout status deployment/order-service -n production

It will not complete normally.

Then:

kubectl get pods -n production

Pick one failing Pod:

kubectl describe pod <NEW-POD-NAME> -n production

Look at Events.

Students should find something similar to:

Failed to pull image

and:

ErrImagePull

then:

ImagePullBackOff

Now ask:

Is ImagePullBackOff the root cause?

More precisely, it's the Kubernetes symptom/state. The useful cause in the Events is that the requested image/tag cannot be pulled.

Check Deployment:

kubectl get deployment order-service \
-n production \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

It should show:

nginx:this-version-does-not-exist

Check Rollout History

kubectl rollout history deployment/order-service -n production

You'll have revisions.

Now ask:

Production worked before the release. The new rollout cannot start. Should we sit in production and experiment with the image?

No.

Mitigate first.

kubectl rollout undo deployment/order-service -n production

Then:

kubectl rollout status deployment/order-service -n production

Check:

kubectl get pods -n production

Eventually:

READY   STATUS
1/1     Running
1/1     Running
1/1     Running

Check image:

kubectl get deployment order-service \
-n production \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Should be back to:

nginx:1.27

Verify from customer path:

kubectl exec -n production test-client -- \
curl -s http://order-service

Incident #1 conclusion

New Deployment
      ↓
New Pods fail
      ↓
ImagePullBackOff
      ↓
Events
      ↓
Invalid/unavailable image tag
      ↓
ROLLBACK
      ↓
Previous known-good release
      ↓
VERIFY

INCIDENT 2 — Production Is Slow

Now we make the lab more professional.

Tell students only:

🚨 INCIDENT: Production is slow. Customers report that Order Service responses are taking much longer than normal. There has been no new deployment.

Ask:

What's the cause?

Students should NOT answer:

“Traffic.”

They should say:

We need evidence.

First Establish Baseline

Check:

kubectl get pods -n production

All should be:

1/1 Running

Now:

kubectl top pods -n production

If this command works, save/observe the CPU usage.

If it returns something like:

error: Metrics API not available

that itself is an observability problem: Metrics Server isn't available/configured. Don't confuse that with the application incident.

Check HPA:

kubectl get hpa -n production

There probably isn't one yet.

Create HPA

kubectl autoscale deployment order-service \
--cpu-percent=50 \
--min=3 \
--max=6 \
-n production

Check:

kubectl get hpa -n production

Then:

kubectl describe hpa order-service -n production

If Metrics Server is functioning, after metrics become available you'll see current/target CPU utilization.

Generate Traffic

Run a load-generator Pod:

kubectl run load-generator \
--image=busybox:1.36 \
--restart=Never \
-n production \
-- /bin/sh -c \
'while true; do wget -q -O- http://order-service >/dev/null; done'

One loop may not produce enough CPU to demonstrate scaling. Create several:

for i in 1 2 3 4 5; do
  kubectl run load-$i \
  --image=busybox:1.36 \
  --restart=Never \
  -n production \
  -- /bin/sh -c \
  'while true; do wget -q -O- http://order-service >/dev/null; done'
done

Now watch:

kubectl top pods -n production

Then:

kubectl get hpa -n production -w

And in another terminal:

kubectl get pods -n production -w

What should students investigate?

Request/load increased
        ↓
CPU may increase
        ↓
HPA observes CPU relative to requests
        ↓
Desired replicas may increase
        ↓
Deployment creates additional Pods

Important: don't promise a particular CPU percentage or replica count. It depends on your cluster, Nginx workload, Metrics Server, timing, and how much load the generators produce.

That's actually good for the class: students must observe evidence rather than copy an expected number.

Stop watchers with:

Ctrl+C

Make Scaling Intentionally Too Restrictive

Now create a more interesting scenario.

kubectl patch hpa order-service \
-n production \
-p '{"spec":{"maxReplicas":3}}'

Check:

kubectl get hpa order-service -n production

Now you have:

MINPODS = 3
MAXPODS = 3

The HPA cannot scale beyond three.

Keep load running.

Check:

kubectl top pods -n production

Then:

kubectl describe hpa order-service -n production

Now ask:

What evidence would you need before saying traffic/capacity is the reason for slowness?

Students should correlate:

Load/request rate ↑
CPU/resource pressure ↑
Latency ↑
HPA target exceeded
Replicas already at configured maximum

Not:

Production slow
→ probably traffic.

Stop Traffic

kubectl delete pod load-generator -n production --ignore-not-found

And:

kubectl delete pod \
load-1 load-2 load-3 load-4 load-5 \
-n production \
--ignore-not-found

Restore HPA:

kubectl patch hpa order-service \
-n production \
-p '{"spec":{"maxReplicas":6}}'

INCIDENT 3 — All Pods Running, Application Doesn't Work

This is one of the best incidents.

Tell students:

🚨 INCIDENT: All Order Service Pods show Running 1/1, but requests to the Order Service fail.

Ask:

If Pods are Running, does that prove production is working?

No.

INSTRUCTOR — Break Service

Run:

kubectl patch service order-service \
-n production \
-p '{"spec":{"selector":{"app":"wrong-order-service"}}}'

Students don't see this command.

Student Investigation

They start:

kubectl get pods -n production

Result:

Running
Running
Running

Then try:

kubectl exec -n production test-client -- \
curl --max-time 5 http://order-service

It fails/times out depending on the exact networking behavior.

Now:

kubectl get svc -n production

Service exists.

kubectl get endpoints order-service -n production

Likely:

ENDPOINTS   <none>

On newer clusters also inspect:

kubectl get endpointslice -n production

Now compare:

kubectl describe service order-service -n production

with:

kubectl get pods -n production --show-labels

Service selector:

app=wrong-order-service

Pod label:

app=order-service

There is your evidence.

Pods Running ✓
      ↓
Service exists ✓
      ↓
Endpoints ✗
      ↓
Selector does not match labels
      ↓
ROOT CAUSE

Fix It

kubectl patch service order-service \
-n production \
-p '{"spec":{"selector":{"app":"order-service"}}}'

Check:

kubectl get endpoints order-service -n production

Addresses should return.

Verify:

kubectl exec -n production test-client -- \
curl -s http://order-service

Production works again.

INCIDENT 4 — Running 0/1

Tell students:

🚨 INCIDENT: Pods are running, but they are not becoming Ready.

Instructor breaks readiness:

kubectl patch deployment order-service \
-n production \
--type='json' \
-p='[
  {
    "op":"replace",
    "path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path",
    "value":"/health-does-not-exist"
  }
]'

Watch:

kubectl rollout status deployment/order-service -n production

Students check:

kubectl get pods -n production

New Pods may show:

0/1 Running

Ask:

What's the difference between STATUS and READY?

Now:

kubectl describe pod <POD-NAME> -n production

They should find readiness probe failures such as an HTTP failure status.

Now check:

kubectl get endpoints order-service -n production

Ready backends can disappear as the bad rollout progresses.

Concept:

Container
   ↓
Running
   ↓
Readiness check
   ↓
FAIL
   ↓
Pod not Ready
   ↓
Not a healthy Service backend

What Do We Do in Production?

Again ask:

This happened immediately after a new configuration/deployment. Previous revision worked. What is our fastest safe mitigation?

Rollback.

kubectl rollout history deployment/order-service -n production

Then:

kubectl rollout undo deployment/order-service -n production

Verify:

kubectl rollout status deployment/order-service -n production

Then:

kubectl get pods -n production

And:

kubectl get endpoints order-service -n production

Finally:

kubectl exec -n production test-client -- \
curl -s http://order-service

INCIDENT 5 — OOMKilled

For this one use a separate workload so we don't destroy the main application.

Create:

nano memory-app.yaml

Paste:

apiVersion: v1
kind: Pod
metadata:
  name: memory-app
  namespace: production
spec:
  containers:
  - name: memory-app
    image: python:3.12-alpine

    resources:
      requests:
        memory: "20Mi"
      limits:
        memory: "40Mi"

    command:
      - python
      - -c
      - |
        import time
        data = []
        while True:
            data.append(bytearray(10 * 1024 * 1024))
            print("Allocated more memory", flush=True)
            time.sleep(1)

Apply:

kubectl apply -f memory-app.yaml

Watch:

kubectl get pod memory-app -n production -w

The container should exceed its memory limit and be terminated/restarted. Depending on timing, you may see OOMKilled most clearly in describe rather than as the STATUS column.

Run:

kubectl describe pod memory-app -n production

Look for:

Last State:
  Terminated
Reason:
  OOMKilled

Now:

kubectl logs memory-app -n production

And critically:

kubectl logs memory-app -n production --previous

This is a very good place to teach --previous.

Ask:

Was there a deployment?

No.

Should we automatically rollback?

No.

Evidence points toward memory behavior/resource limit.

Remove:

kubectl delete pod memory-app -n production

INCIDENT 6 — Dependency Failure

Now we'll simulate the idea of:

Order Service
     ↓
Kafka / Database / API

without requiring Kafka installation.

Create:

nano dependency-app.yaml

Paste:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: production
spec:
  replicas: 1

  selector:
    matchLabels:
      app: payment-service

  template:
    metadata:
      labels:
        app: payment-service

    spec:
      containers:
      - name: payment-service
        image: busybox:1.36

        command:
        - /bin/sh
        - -c
        - |
          echo "Starting payment-service"
          echo "Connecting to Kafka..."
          sleep 2
          echo "ERROR: Cannot connect to kafka.production.svc:9092"
          echo "Application exiting"
          exit 1

Apply:

kubectl apply -f dependency-app.yaml

Watch:

kubectl get pods -n production -w

Eventually:

CrashLoopBackOff

Now tell students only:

🚨 INCIDENT: Payment Service is unavailable.

They should investigate:

kubectl get pods -n production

Then:

kubectl logs deployment/payment-service -n production

Eventually use the actual Pod name:

kubectl get pods -n production \
-l app=payment-service

Then:

kubectl logs <PAYMENT-POD> \
-n production \
--previous

They'll see:

Starting payment-service
Connecting to Kafka...
ERROR: Cannot connect to kafka.production.svc:9092
Application exiting

Now ask:

Have we proven Kafka itself is down?

No.

This is very important.

We have proven:

Payment application reports that it cannot connect to the Kafka endpoint.

We haven't yet proven why.

Potential investigation branches:

                 Kafka connection failure
                           │
         ┌─────────────────┼─────────────────┐
         ↓                 ↓                 ↓
      DNS/name           Network           Kafka
      wrong?             blocked?          unhealthy?
         │                 │                 │
         ↓                 ↓                 ↓
      Service?        NetworkPolicy?      Brokers?
         │
         ↓
       Port?

That's production thinking.

Delete:

kubectl delete deployment payment-service -n production

FINAL INCIDENT — Students Get No Hints

Now combine everything.

Tell them:

🚨 SEV-1 INCIDENT

Customers report that the Order Service is unavailable.

You are the on-call DevOps team. Investigate and restore service.

Don't tell them anything else.

Instructor secretly breaks:

kubectl patch service order-service \
-n production \
-p '{"spec":{"selector":{"app":"orders"}}}'

Then sit back.

Students should develop this process themselves:

What's affected?
       ↓
Order Service
       ↓
Any deployment?
       ↓
No
       ↓
Pods?
       ↓
Running / Ready
       ↓
Resources?
       ↓
Healthy
       ↓
Service?
       ↓
Exists
       ↓
Endpoints?
       ↓
NONE
       ↓
Why?
       ↓
Selector vs labels
       ↓
Mismatch
       ↓
ROOT CAUSE

They fix:

kubectl patch service order-service \
-n production \
-p '{"spec":{"selector":{"app":"order-service"}}}'

But don't accept:

“Fixed.”

Ask:

PROVE IT.

They need to verify:

kubectl get endpoints order-service -n production

Then:

kubectl exec -n production test-client -- \
curl -s http://order-service

Then:

kubectl get pods -n production

Only now:

Production restored.

The Troubleshooting Command Sheet

This is the part I would have every student save.

# =====================================================
# 1. WORKLOAD
# =====================================================

kubectl get pods -n production

kubectl get pods -n production -o wide

kubectl describe pod <pod> -n production


# =====================================================
# 2. LOGS
# =====================================================

kubectl logs <pod> -n production

kubectl logs <pod> -n production --previous


# =====================================================
# 3. EVENTS
# =====================================================

kubectl get events \
-n production \
--sort-by=.metadata.creationTimestamp


# =====================================================
# 4. METRICS
# =====================================================

kubectl top pods -n production

kubectl top nodes


# =====================================================
# 5. DEPLOYMENT
# =====================================================

kubectl get deployment -n production

kubectl describe deployment <deployment> -n production

kubectl rollout status deployment/<deployment> -n production

kubectl rollout history deployment/<deployment> -n production

kubectl rollout undo deployment/<deployment> -n production


# =====================================================
# 6. NETWORK
# =====================================================

kubectl get svc -n production

kubectl describe svc <service> -n production

kubectl get endpoints -n production

kubectl get endpointslice -n production

kubectl get ingress -n production

kubectl get networkpolicy -n production


# =====================================================
# 7. STORAGE
# =====================================================

kubectl get pvc -n production

kubectl get pv

kubectl get storageclass

kubectl describe pvc <pvc> -n production


# =====================================================
# 8. SCALING
# =====================================================

kubectl get hpa -n production

kubectl describe hpa <hpa> -n production


# =====================================================
# 9. SECURITY
# =====================================================

kubectl auth can-i get pods -n production

kubectl auth can-i delete pods -n production


# =====================================================
# 10. CONFIGURATION
# =====================================================

kubectl get configmap -n production

kubectl get secret -n production

The Production Rule Students Need to Remember

Don't teach them this:

Production Down
     ↓
kubectl get pods
     ↓
restart everything

Teach them this:

                    INCIDENT
                       │
                       ▼
              WHAT IS THE IMPACT?
                       │
                       ▼
               WHAT CHANGED?
                       │
                       ▼
          IDENTIFY AFFECTED COMPONENT
                       │
                       ▼
         ┌─────────────┼─────────────┐
         ▼             ▼             ▼
       LOGS          METRICS       EVENTS
         │             │             │
         └─────────────┼─────────────┘
                       ▼
                  HYPOTHESIS
                       │
                       ▼
                    EVIDENCE
                       │
                       ▼
                  ROOT CAUSE
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
         MITIGATE               FIX
       e.g. rollback
             │
             ▼
           VERIFY
             │
             ▼
      PRODUCTION RESTORED
             │
             ▼
       ROOT CAUSE ANALYSIS

And throughout the lab, whenever a student says:

“I think it's traffic.”

Your response should be:

“Show me the evidence.”

If they say:

“It's Kubernetes.”

Ask:

“Which Kubernetes component? Show me the evidence.”

If they say:

“Kafka is down.”

Ask:

“Have you proven Kafka is down, or have you only proven that your application cannot connect to Kafka?”

If they say:

“I fixed it.”

DEV Community: Aisalkyn Aidarova

2-2 The Midnight Payment Processing Incident — Kubernetes, PostgreSQL, Terraform, CI/CD, Grafana and PagerDuty

Your mission

1. Incident ticket

2. Production architecture and team responsibilities

3. What you will build

4. Requirements

First, verify your environment

Part A — Create all application and Kubernetes files

File 1 — app/requirements.txt

File 2 — app/app.py

File 3 — app/Dockerfile

File 4 — k8s/00-namespace.yaml

File 5 — k8s/01-postgres.yaml

File 6 — k8s/02-payment-api.yaml

File 7 — k8s/03-networkpolicy-outage.yaml

File 8 — k8s/04-servicemonitor-optional.yaml

Part B — Build, deploy and verify

Step 1. Build and push the Docker image

Step 2. Deploy PostgreSQL

Step 3. Deploy Payment API

Step 4. Verify the database table

Step 5. Test the application

Part C — Configure Grafana and Prometheus

Panel 1 — Payment API request rate by HTTP status

Panel 2 — HTTP 5xx error rate

Panel 3 — Database errors over five minutes

Panel 4 — Available replicas

Panel 5 — Desired replicas

Panel 6 — Pod readiness

Panel 7 — Payment request p95 latency

Key learning point

2 / 2

503

Part D — Create the incident

Class 2: Midnight outage simulation

Step 1. Confirm the healthy baseline

Step 2. Introduce the controlled failure

Step 3. Investigate the incident

Student investigation script

Root-cause analysis

Part E — Restore service

Part F — Terraform and AWS RDS investigation

Why Terraform matters

File 9 — terraform/EXAMPLE_ONLY.tf

Terraform investigation commands

CI/CD investigation

Part G — Three-class teaching schedule

Class 1 — Build the platform

Class 2 — Investigate the outage

Class 3 — Recover and explain

Part H — Interview preparation

1. Your Pods are Running and Ready, but customers receive HTTP 503. What do you investigate?

2. How can all Pods be Ready while payment processing is failing?

3. What would you check first to determine whether the database is the problem?

4. How would you prove that a Terraform change caused the outage?

5. Would you immediately run kubectl rollout undo?

6. What is the difference between HTTP 502, 503 and 504?

7. Who would you involve in a SEV-1 incident?

8. How would you investigate a delayed failure after a successful release?

9. What database metrics would you inspect?

10. What is a database connection pool?

11. How would you restore service without creating additional risk?

12. What would you do after recovery?

Part I — Student incident report

File 10 — scripts/01-check.sh

File 11 — scripts/06-cleanup.sh

Final student challenge

JumpToTech DevOps Production Incident Lab #2 The Midnight Payment Processing Incident

41%

37%

12/12

Available

Your three-class mission

1. The company and production environment

Production architecture

Example workload sizes

2. Your production engineering team

Incident response team structure

Who is responsible for what?

File 1 — `app/requirements.txt`

File 2 — `app/app.py`

File 3 — `app/Dockerfile`

File 4 — `k8s/00-namespace.yaml`

File 5 — `k8s/01-postgres.yaml`

File 6 — `k8s/02-payment-api.yaml`

File 7 — `k8s/03-networkpolicy-outage.yaml`

File 8 — `k8s/04-servicemonitor-optional.yaml`

File 9 — `terraform/EXAMPLE_ONLY.tf`

5. Would you immediately run `kubectl rollout undo`?

File 10 — `scripts/01-check.sh`

File 11 — `scripts/06-cleanup.sh`