DEV Community: Michael Zion

Kubernetes HPA: How It Works and How to Stop Flapping

Michael Zion — Mon, 02 Nov 2026 07:30:09 +0000

This guide was written by Arthur Azrieli, CTO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

The Kubernetes Horizontal Pod Autoscaler (HPA) adds or removes pod replicas for a Deployment, StatefulSet or other scalable workload based on a metric such as CPU utilization. Every 15 seconds it compares the current metric to your target, computes ceil(currentReplicas × current / target), and scales within your minReplicas and maxReplicas.

That is the whole idea. Most of the trouble comes later, when replica counts start bouncing up and down, targets show <unknown>, or a GitOps tool keeps resetting the replica count. This guide covers how the algorithm really behaves (including the parts most tutorials skip), copy-paste manifests for the common cases, the behavior settings that stop scaling flaps, how HPA fits with VPA, KEDA, Karpenter and Argo Rollouts, and a troubleshooting table built from the actual controller error messages.

Versions referenced: Kubernetes 1.37 (the current release in October 2026), metrics-server v0.9.0 and KEDA v2.21. The autoscaling/v2 API used throughout has been stable since Kubernetes 1.23, so everything here works on any supported cluster unless a section says otherwise.

Kubernetes autoscalers compared: HPA, VPA, KEDA and node autoscalers

"Kubernetes autoscaling" means four different things, and mixing them up causes most bad designs. The HPA only changes the number of pods. It never changes pod size and never adds nodes.

Autoscaler	What it changes	Typical signal	Use it when
Horizontal Pod Autoscaler (HPA)	Replica count of a workload	CPU or memory utilization, custom or external metrics	Stateless services where more pods means more capacity
Vertical Pod Autoscaler (VPA)	CPU and memory requests of each pod	Historical usage	Right-sizing requests, or workloads that cannot scale out
KEDA	Replica count, including to and from zero (it creates and manages an HPA for you)	Event sources: queue length, Kafka lag, cron, Prometheus queries, cloud services	Queue consumers and event-driven workers
Cluster Autoscaler or Karpenter	Number and type of nodes	Pending pods that cannot be scheduled	Always, if the HPA can create more pods than your nodes can hold

The usual production setup is HPA (or KEDA) for pods plus Cluster Autoscaler or Karpenter for nodes. If the node layer is missing, the HPA happily creates pods that sit in Pending, and you get the cost of a scale-up decision without the capacity. Our guide on setting up Kubernetes autoscaling without cost surprises covers the node side.

How the Horizontal Pod Autoscaler algorithm works

The HPA is a control loop inside kube-controller-manager. It does not react to events. It wakes up on an interval (--horizontal-pod-autoscaler-sync-period, 15 seconds by default), reads metrics, and computes a desired replica count:

desiredReplicas = ceil( currentReplicas × currentMetricValue / desiredMetricValue )

Example: 4 replicas averaging 90% CPU utilization against a 60% target gives ceil(4 × 90 / 60) = ceil(6.0) = 6 replicas. At 45% utilization it gives ceil(4 × 45 / 60) = 3.

For resource metrics, "utilization" is usage divided by the container's request, not its limit and not node capacity. A pod requesting 250m CPU and using 200m is at 80%. This single fact explains a large share of HPA surprises: change the request and you change the scaling math, even though nothing about the traffic changed.

The HPA recomputes desired replicas on a fixed interval and skips any change that falls inside the 10% tolerance.

The details that shape real behavior

The formula is the easy part. These rules, all from the Kubernetes documentation, decide what actually happens:

Tolerance. If the ratio current / target is within 10% of 1.0, the HPA does nothing. With a 60% CPU target, anything between 54% and 66% is a no-op. This dead band is your first protection against jitter.
Unready pods are set aside. When scaling on CPU, pods that are not yet ready, or whose latest metric sample predates readiness, are excluded from the average. If the result would be a scale up, those pods are then assumed to use 0% of the target, which dampens the scale up.
Missing metrics are treated conservatively. Pods with no metric are assumed to use 100% of the target when the HPA is considering a scale down, and 0% when considering a scale up. If that reverses the direction or lands inside the tolerance, the HPA skips the action.
Startup windows. --horizontal-pod-autoscaler-initial-readiness-delay (default 30 seconds) and --horizontal-pod-autoscaler-cpu-initialization-period (default 5 minutes) control how the controller treats CPU samples from freshly started pods. These are control plane flags, so on EKS, GKE or AKS you cannot change them. Design the workload around the defaults instead.
Multiple metrics take the maximum. With CPU and a requests-per-second metric, the HPA computes a desired count for each and uses the highest. If one metric cannot be fetched and the others suggest a scale down, the scale down is skipped. Scale up still works.
Status shows the raw value. kubectl get hpa reports the original average, not the adjusted one used for the decision. That is why the TARGETS column can look like it should trigger a scale and nothing happens.
Scale down is smoothed by default. The HPA keeps the highest recommendation from the last 5 minutes before scaling down. That default stabilization window is why replicas linger after a spike.

Prerequisites: metrics-server and resource requests

Resource metrics (CPU and memory) come from the metrics.k8s.io API, which metrics-server provides. Many managed clusters ship it, many do not. Check first:

kubectl get apiservice v1beta1.metrics.k8s.io
kubectl top pods -n production

If kubectl top fails, install metrics-server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Then make sure every container in the target pods has a CPU request (and a memory request if you scale on memory). The HPA computes utilization per pod across all its containers, so a single sidecar without a request breaks CPU utilization scaling for the whole workload. If your requests are guesses, fix them before tuning the HPA. Our guide to Kubernetes requests and limits shows how to size them from real usage.

Create your first HPA

The fastest way is kubectl autoscale. Recent kubectl versions use --cpu with a percentage (the older --cpu-percent flag is deprecated):

kubectl autoscale deployment api -n production --min=3 --max=20 --cpu=60%

For anything you run in production, keep the HPA in version control as YAML instead:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

Watch it work:

kubectl get hpa api -n production --watch
kubectl describe hpa api -n production

describe is where the useful information lives: current metrics, the three status conditions (AbleToScale, ScalingActive, ScalingLimited) and recent scaling events with reasons.

Remove spec.replicas when the HPA owns the count

The Kubernetes docs recommend removing spec.replicas from the Deployment once an HPA manages it. Otherwise every kubectl apply, Helm upgrade or Argo CD sync resets the replica count to the value in Git, the HPA scales it back, and you get a sawtooth that looks exactly like flapping. In Helm charts, the common pattern is to render replicas only when autoscaling is disabled. In Argo CD, you can also add /spec/replicas to ignoreDifferences for the Deployment.

HPA metric types: resource, container, pods, object and external

The autoscaling/v2 API supports five metric source types. Pick the one that measures the real bottleneck, not the one that is easiest to wire up.

Type	What it measures	Needs	Good for
`Resource`	CPU or memory across the whole pod, as utilization or average value	metrics-server	CPU-bound web services
`ContainerResource`	CPU or memory of one named container	metrics-server, Kubernetes 1.30 or newer (stable)	Pods with sidecars (service mesh proxies, log shippers) that distort the pod average
`Pods`	A custom per-pod metric averaged across pods	A custom metrics adapter such as Prometheus Adapter	Requests per second or in-flight requests per pod
`Object`	A metric describing another Kubernetes object	A custom metrics adapter	Requests per second on an Ingress
`External`	A metric from outside the cluster	An external metrics adapter (KEDA provides one)	Queue depth, Pub/Sub backlog, Kafka lag

Scaling only the application container when a sidecar is present:

metrics:
  - type: ContainerResource
    containerResource:
      name: cpu
      container: app
      target:
        type: Utilization
        averageUtilization: 60

Scaling on a per-pod custom metric (for example, exposed by Prometheus Adapter as http_requests_per_second):

metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "50"

A few rules of thumb from running these in production:

CPU is a good signal when traffic turns directly into CPU work: serialization, compression, TLS, rendering.
Memory is a poor scaling signal for most runtimes. The JVM, Go, Node.js and Python rarely give memory back quickly after load drops, so a memory-based HPA scales up easily and then refuses to scale down. Use it only when memory per pod clearly falls as you add pods.
Requests per second or in-flight requests beat CPU for I/O-bound APIs that spend their time waiting on databases or other services.
Queue depth or lag is the right signal for workers. A worker blocked on a slow dependency shows low CPU while the queue grows, so CPU scaling never fires. Use KEDA here.

Why HPAs flap, and how to stop it

Flapping means the replica count oscillates: up on a spike, down a minute later, up again. Users feel it as latency spikes during scale ups, connection churn during scale downs, and a cluster that never settles. The causes are almost always one of these:

Requests set too low. A pod that normally uses 300m with a 100m request sits at 300% utilization at idle. Small traffic changes swing the ratio wildly. Fix the request first.
A noisy metric. Per-second request rates, garbage collection pauses and batchy traffic produce sharp short peaks. The HPA sees them every 15 seconds and reacts.
Startup CPU spikes. JIT compilation, cache warm-up and connection pool creation make new pods look overloaded, which triggers more scale up, which starts more pods.
Scale down that is too fast. Removing many pods at once pushes the survivors over the target, and the HPA adds them back.
A target with no headroom. A 90% CPU target leaves nothing for the minute or two it takes a new pod to become ready.
Something else is changing replicas. A GitOps sync resetting spec.replicas, two HPAs pointing at the same Deployment, or KEDA and a hand-written HPA on the same target.

Causes 1, 5 and 6 are configuration errors you fix in the workload. Causes 2, 3 and 4 are what the behavior field exists for.

Half of flapping is a configuration error in the workload; the other half is what scaling behavior settings are for.

HPA behavior: stabilization windows and scaling policies

The behavior field lets you control scale up and scale down separately. Each direction has a stabilization window, one or more rate-limiting policies, and a selectPolicy that decides which policy wins. Fields you do not set are merged with these defaults:

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
      - type: Percent
        value: 100
        periodSeconds: 15
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
      - type: Percent
        value: 100
        periodSeconds: 15
      - type: Pods
        value: 4
        periodSeconds: 15
    selectPolicy: Max

In words: by default the HPA can double the replica count (or add 4 pods, whichever is more) every 15 seconds with no delay, and can remove every pod above minReplicas in one step once the 5-minute scale-down window has passed. That is aggressive in both directions.

What each field does

stabilizationWindowSeconds: the HPA looks at all recommendations made during this window and uses the most conservative one. For scale down that is the highest recommendation, so a brief dip does not remove pods. For scale up it is the lowest, so a brief spike does not add pods.
policies: rate limits. type: Pods, value: 4, periodSeconds: 60 means at most 4 pods per minute. type: Percent, value: 10, periodSeconds: 60 means at most 10% of current replicas per minute (fractions are rounded up, so 10% of 72 replicas allows 8).
selectPolicy: Max picks the policy allowing the biggest change, Min the smallest, and Disabled turns off scaling in that direction entirely.
tolerance (per direction): overrides the cluster-wide 10% dead band for this HPA. It is beta in Kubernetes 1.35 and 1.36 and generally available from 1.37, so check your cluster version before relying on it.

A stabilization window ignores short dips and spikes by acting on the most conservative recent recommendation.

Anti-flap profiles you can copy

Starting points, not final answers. Tune them with the load test in the next section.

Workload	Scale up	Scale down	Why
Latency-sensitive API, fast startup (under 30 seconds)	No window, max of 100% or 4 pods per 60 seconds	300 second window, 20% per 60 seconds	React fast to real load, shed capacity slowly
Slow-starting service (JVM, large caches)	60 second window, 50% per 60 seconds	600 second window, 10% per 60 seconds	Stop startup spikes from compounding; keep warm pods
Spiky, bursty traffic	No window, 100% per 30 seconds	900 second window, 1 pod per 120 seconds	Bursts return; keeping capacity is cheaper than churn
Batch or queue worker on CPU	No window, 4 pods per 60 seconds	300 second window, 25% per 60 seconds	Usually better on KEDA with queue depth

The latency-sensitive API profile as YAML:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      selectPolicy: Max
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
        - type: Pods
          value: 4
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      selectPolicy: Min
      policies:
        - type: Percent
          value: 20
          periodSeconds: 60
        - type: Pods
          value: 2
          periodSeconds: 60

With selectPolicy: Min on scale down, the HPA removes whichever is smaller, 20% or 2 pods, per minute. At 30 replicas that is 2 pods a minute, so once the 5-minute window has passed it takes about 13 more minutes to return to 3. That is deliberate: slow scale down is the cheapest insurance against flapping.

Holding capacity for five minutes and then shedding 2 pods a minute is cheap insurance against flapping.

How to pick the target utilization

The target is really a headroom setting. Ask how much traffic can grow in the time it takes a new pod to become ready and serve. If traffic can rise 50% in the two minutes it takes to start a pod, existing pods need room to absorb that, so a target near 100 / 1.5, about 65% of the point where latency degrades, is the ceiling. If your service degrades at 80% CPU, that gives roughly 50% to 55%.

Most CPU-bound services land between 50% and 70%. Below 50% you pay for a lot of idle capacity. Above 75% you are betting that scale ups always finish before the next spike.

Test HPA behavior before production traffic does

Do not trust an HPA you have not watched under load. In a staging environment with realistic requests:

# terminal 1: decisions and events
kubectl get hpa api -n staging --watch
kubectl get events -n staging --field-selector involvedObject.name=api --watch

# terminal 2: pods coming and going
kubectl get pods -n staging -l app=api --watch

# terminal 3: load from outside the cluster for 10 minutes, then stop
hey -z 10m -c 100 https://staging.example.com/api/health-heavy

Record five things: time to first scale up, replicas added per minute, whether new pods were ready before the next decision, whether latency recovered, and how replicas stepped down after the load stopped. If replicas climb past what the load needs, your scale up is outrunning readiness. If they drop and come straight back, slow the scale down.

HPA with VPA, KEDA, Karpenter and Argo Rollouts

HPA and VPA

Do not let both act on the same resource metric. If the HPA scales on CPU utilization and VPA raises CPU requests, utilization drops, the HPA removes pods, utilization rises, and the two loops chase each other. Safe combinations are VPA in recommendation-only mode (updateMode: "Off") to inform your requests, or HPA on a custom metric such as requests per second while VPA manages CPU and memory.

Give the HPA and VPA different signals, or keep VPA in recommendation mode, so they never fight over one metric.

HPA and KEDA

KEDA is not a competitor to the HPA. A KEDA ScaledObject creates and manages an HPA for you and feeds it external metrics from its 70+ built-in scalers, plus it handles activation from zero replicas. Never point your own HPA and a KEDA ScaledObject at the same Deployment. Choose KEDA when the signal lives outside the pod (queues, streams, schedules) or when you need scale to zero.

HPA and node autoscaling

When the HPA adds pods that do not fit, they stay Pending until Cluster Autoscaler or Karpenter adds a node, which commonly takes one to several minutes. Your scale-up speed is therefore the slower of pod startup and node provisioning. If that is too slow for your traffic pattern, raise minReplicas, lower the target, or keep a little spare node capacity with low-priority placeholder pods that real workloads can preempt (see our guide to Kubernetes PriorityClasses).

HPA during rollouts

During a rolling update the HPA scales the Deployment, and the Deployment controller spreads replicas across old and new ReplicaSets. Startup CPU from new pods can trigger scale ups mid-rollout, so the slow-starting profile above helps here too. If you use progressive delivery, point scaleTargetRef at the Rollout instead of a Deployment (apiVersion: argoproj.io/v1alpha1, kind: Rollout). Our Argo Rollouts guide covers how canary and blue-green steps interact with the HPA.

Can the HPA scale to zero?

Only for object and external metrics, and only with the HPAScaleToZero feature gate. That gate was alpha (off by default) from Kubernetes 1.16 to 1.36 and became beta, on by default, in 1.37. CPU and memory cannot scale to zero because they can only be measured on running pods. On older clusters, or for anything event driven, use KEDA, which has supported scale to zero for years.

HPA troubleshooting: error messages and fixes

These are the messages the HPA controller actually emits, as seen in kubectl describe hpa.

Symptom or message	Cause	Fix
TARGETS shows `<unknown>/60%`	No metrics yet, metrics-server missing, or a container without a request	Wait one or two minutes after creation; run `kubectl top pods`; add requests to every container
`failed to get cpu utilization: missing request for cpu in container X of Pod Y`	A container (often a sidecar or init-injected proxy) has no CPU request	Add a CPU request to every container, or switch to `ContainerResource`
`unable to get metrics for resource cpu: ...` with reason `FailedGetResourceMetric`	The `metrics.k8s.io` API is unavailable	Check `kubectl get apiservice v1beta1.metrics.k8s.io` and the metrics-server pod logs
`did not receive metrics for targeted pods (pods might be unready)`	Pods are crash looping or never become ready	Fix readiness first; see our guide to liveness probes without restart loops
`ScalingLimited True TooManyReplicas`	The desired count hit `maxReplicas`	Raise `maxReplicas` only if downstream systems (database, cache, APIs) can take it
`ScaleDownStabilized` while load is low	The scale-down window is holding the highest recent recommendation	Expected. Shorten the window only if idle cost matters more than churn
Replicas jump back to a fixed number after every deploy	`spec.replicas` in the manifest overrides the HPA on each apply or sync	Remove `spec.replicas`, or ignore it in Argo CD
Memory HPA never scales down	The runtime keeps memory after load drops	Scale on CPU or a work metric instead
More pods, same latency	The bottleneck is a database, lock or external API	Cap `maxReplicas` at downstream capacity, add pooling and backpressure

Production HPA checklist

Every container in the target has a CPU request based on measured usage.
spec.replicas is removed from the Deployment manifest, or ignored by your GitOps tool.
minReplicas is at least 2 (3 for multi-zone services) and covers normal traffic.
maxReplicas is set from downstream capacity and node budget, not from a round number.
The metric matches the bottleneck: CPU for compute, requests per pod for I/O-bound APIs, queue depth (via KEDA) for workers.
Scale up is faster than scale down, and scale down has a window of at least 300 seconds.
Readiness probes reflect real readiness, and startup time is known.
A node autoscaler is in place and its provisioning time is part of your scale-up budget.
VPA is not changing the same resource the HPA scales on.
You have watched the HPA through a load test and a scale down.

Kubernetes HPA FAQ

Is HPA useful for Kubernetes?

Yes, for stateless workloads where more pods means more capacity. It lets you run fewer replicas at quiet times and absorb peaks without manual intervention. It is the wrong tool for stateful systems that cannot simply add replicas, and for workloads whose bottleneck is outside the pod.

What is a good HPA CPU threshold?

Most CPU-bound services work well between 50% and 70% of the CPU request. Use the headroom calculation above: the slower your pods and nodes start, and the faster your traffic grows, the lower the target should be.

What is the difference between HPA and VPA?

The HPA changes how many pods run. VPA changes how much CPU and memory each pod requests. They can work together if they act on different signals, for example HPA on requests per second and VPA on resources, or VPA in recommendation mode only.

What is the difference between KEDA and HPA?

KEDA builds on the HPA. It creates an HPA for you, feeds it metrics from event sources like queues and streams, and adds scale to zero. Use plain HPA for CPU or custom metrics on services; use KEDA for event-driven workers.

How often does the HPA check metrics?

Every 15 seconds by default. Metrics-server itself scrapes kubelets on its own interval, so the data the HPA sees can be somewhat older than that.

Why does my HPA show unknown targets?

Either metrics are not available yet (normal for a minute or two after creation), metrics-server is missing or broken, or at least one container in the pods has no request for the resource you scale on. kubectl describe hpa shows which one.

Why is my HPA not scaling down?

The 5-minute default scale-down stabilization window, a metric still above the target minus the 10% tolerance, a memory metric that never drops, or a second metric that keeps the maximum recommendation high. Check the conditions and events in kubectl describe hpa.

Can I use HPA and a fixed replica count in Helm?

Not on the same object. Render replicas in the Deployment only when autoscaling is disabled, so a Helm upgrade does not reset the HPA's decision.

If your autoscaling still behaves unpredictably after this, the cause is usually in the workload or the node layer rather than the HPA itself. MeteorOps engineers tune this for startups as part of Kubernetes consulting.

SRE vs DevOps: Key Differences and Which Your Startup Needs

Michael Zion — Sun, 01 Nov 2026 07:30:04 +0000

This guide was written by Michael Zion, CEO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

SRE vs DevOps in one paragraph: DevOps is a culture and set of practices for shipping software faster by removing the wall between development and operations. Site reliability engineering (SRE) is a specific, opinionated way to run production that came out of Google: it sets reliability targets (SLOs), spends an error budget against them, and treats operations as a software problem. Google sums it up as "class SRE implements interface DevOps". DevOps optimizes the path to production. SRE optimizes what happens once you are there.

Below is how the two differ in goals, daily work, metrics, skills and pay, how they fit together, and which one a startup actually needs at each stage.

SRE vs DevOps at a glance

	DevOps	SRE
What it is	A culture and set of practices across the whole delivery lifecycle	A job role and an engineering discipline for running production
Origin	A movement among practitioners at many companies, around 2009	Google, from 2003, written up in the Site Reliability Engineering book (2016)
Main question	How do we ship changes faster and safer?	How reliable must this service be, and are we meeting that?
Core practices	CI/CD, infrastructure as code, automation, shared ownership, fast feedback	SLIs and SLOs, error budgets, toil reduction, incident response, blameless postmortems, capacity planning
Typical metrics	The five DORA metrics: change lead time, deployment frequency, failed deployment recovery time, change fail rate, deployment rework rate	SLO attainment, error budget burn rate, toil percentage, incident count and duration, pages per shift
Prescriptiveness	Principles; little detail on how to run services	Concrete rules, such as caps on toil and on-call time
Where the work sits	Pipelines, environments, platforms, developer workflow	Production systems, observability, on-call, reliability of user-facing services

DevOps owns how changes reach production; SRE owns how production behaves once they land.

What DevOps actually means

DevOps started as a reaction to a familiar split: developers were measured on shipping features, operations on keeping things stable, and every release became a fight. DevOps says one team should own the whole path from commit to production, with automation doing the repetitive work.

A common way to describe it is CALMS: Culture, Automation, Lean, Measurement and Sharing. In practice that means CI/CD pipelines, infrastructure as code, small and frequent changes, environments anyone can recreate, and developers who can see and act on how their code behaves in production.

"DevOps engineer" is the job title that grew out of this. In most companies a DevOps engineer builds and runs the delivery platform: cloud infrastructure, CI/CD, containers and Kubernetes, secrets, and the tooling that lets developers ship on their own.

What SRE actually means

SRE is what you get when you ask software engineers to run operations. Its central idea is that 100% reliability is the wrong target. Users cannot tell 99.99% from 100%, and chasing the last fraction slows every release. So SRE makes reliability an explicit, measured number:

SLI (service level indicator): what you measure, such as the share of requests that succeed in under 300 ms.
SLO (service level objective): the target for that indicator over a window, such as 99.9% over 30 days.
Error budget: the failure you are allowed. A 99.9% SLO over 30 days allows about 43 minutes of full downtime, or the equivalent in failed requests.

While budget remains, teams ship freely. When it runs out, the priority shifts to reliability work until the service is healthy again. That one rule turns the old dev vs ops argument into a shared, data-driven decision.

The error budget turns a reliability target into a clear rule for when to ship and when to fix.

SRE also sets hard limits on operational load. Google's SRE teams aim to keep toil, the manual, repetitive, automatable work that "scales linearly as a service grows", below 50% of each engineer's time. Of the remaining operational time, no more than 25% should be on-call.

How much downtime does each SLO allow?

SLO	Allowed downtime per 30 days	Per year
99%	7 hours 12 minutes	about 3.65 days
99.5%	3 hours 36 minutes	about 1.8 days
99.9%	43.2 minutes	about 8.8 hours
99.95%	21.6 minutes	about 4.4 hours
99.99%	4.3 minutes	about 53 minutes

"class SRE implements interface DevOps"

Google's own SRE workbook frames the relationship this way. DevOps is the interface: a set of principles. SRE is one concrete implementation of those principles, with specific practices for each of them:

Reduce organizational silos: SREs share ownership of production with developers and use the same tools.
Accept failure as normal: error budgets and blameless postmortems.
Implement gradual change: small changes, canary releases, fast rollback.
Tooling and culture are interrelated: SREs use the same tools as developers, and the toil cap forces automation.
Measure everything: SLOs and error budgets drive decisions.

Each DevOps principle has a concrete SRE practice behind it.

The workbook also names the differences: DevOps is broader and targets silos across the whole organization, while SRE is narrower, more prescriptive, and focused on services and their users.

Responsibilities side by side

Area	DevOps engineer	Site reliability engineer
CI/CD	Builds and owns pipelines, build caching, release automation	Adds safety: canaries, automated rollback, release gates tied to SLOs
Infrastructure	Provisions cloud, networks, clusters with Terraform or similar	Capacity planning, load testing, failure-mode design, disaster recovery drills
Observability	Installs and maintains the monitoring and logging stack	Defines SLIs and SLOs, alerting on symptoms and burn rate, dashboards for incidents
Incidents	Often on the rotation, fixes infrastructure causes	Leads incident response, runs postmortems, tracks follow-ups to completion
Developer experience	Self-service environments, templates, internal platform	Production readiness reviews, reliability guidance for service owners
Code	Infrastructure code, scripts, pipeline config	Automation and tooling, sometimes changes to application code for reliability

In small companies these roles overlap heavily, and one senior engineer often does both. The split becomes real once you have enough services and traffic that reliability needs a dedicated owner.

Skills: where they overlap and where they don't

Shared core: Linux, networking, a major cloud provider, containers and Kubernetes, infrastructure as code, CI/CD, a scripting or programming language, and monitoring.

Leans DevOps: pipeline design, build systems, Terraform module design, developer tooling, platform engineering, cost-aware infrastructure.

Leans SRE: distributed systems failure modes, SLO design, statistics for latency and error rates, performance and capacity analysis, incident command, writing production-grade software in Go, Python or Java.

SRE vs DevOps salary

The two pay similarly, with SRE slightly ahead in most surveys. In the 2024 Stack Overflow Developer Survey, median yearly salaries were:

Role	United States median	Global median
SRE	$166,500	$99,099
DevOps specialist	$145,000	$72,611
Cloud infrastructure engineer	$165,000	$96,666

The 2025 survey no longer breaks out SRE, but reports a US median of $165,000 for "DevOps engineer or professional". Seniority, location and company size move pay far more than the title does. Treat the gap between the two roles as small.

Which does your startup need?

Stage	What you need	Why
Pre-product or MVP, 1 to 10 engineers	DevOps practices, owned by developers, with outside help for the foundation	You need CI/CD, infrastructure as code and basic monitoring. Reliability targets would be guesses.
Early customers, 10 to 30 engineers	A first DevOps or platform engineer, plus simple SLOs on your one or two critical user journeys	Delivery speed is the bottleneck. A light SLO keeps reliability from being argued by gut feel.
Growth, 30 to 100 engineers	A platform team and your first SRE, or SRE practices embedded in the platform team	Incidents, on-call load and enterprise customers' uptime expectations make reliability a full-time job.
Scale, 100+ engineers or strict SLAs	A dedicated SRE function alongside platform engineering	Error budgets, capacity planning and incident management across many teams need clear ownership.

Two practical rules:

Hire for the bottleneck. If releases are slow and painful, you need DevOps capability. If releases are fine but outages, pages and customer complaints are the problem, you need SRE capability.
Don't build a 24/7 rotation too early. Google's guidance puts the minimum for a single-site 24/7 rotation at eight engineers. Most startups get further with business-hours paging, good runbooks and clear severity rules. Our guide to designing an on-call rotation for a small team covers the math.

Let the problem you feel most decide which capability to hire first.

How DevOps and SRE work together

In a healthy setup, the platform or DevOps team builds the paved road (pipelines, environments, infrastructure modules), and SRE makes sure what runs on that road meets its reliability targets. Concretely:

Product teams own their services and their SLOs.
The platform team makes the right thing the easy thing: standard deploys, built-in metrics, safe rollbacks.
SREs set the SLO framework, review production readiness, lead major incidents and push systemic fixes back into the platform.
Error budget policy decides, in advance, what happens when a service runs out of budget.

A practical first step for any team is cleaning up alerting so that pages map to user impact. Our article on handling alert fatigue shows how, and the observability stack guide covers what to run at each team size.

Common mistakes

Renaming the ops team "SRE". Without SLOs, error budgets and a toil cap, it is the same ops team with a new title.
Hiring a "DevOps engineer" to be the only person who can deploy. That rebuilds the silo DevOps was meant to remove.
Setting SLOs at 99.99% by default. Each extra nine costs a lot. Start from what users notice and what your dependencies can deliver.
Measuring only uptime. Latency and correctness matter to users as much as availability.
Treating it as either/or. Most teams need DevOps practices first and SRE practices as they grow, not one instead of the other.

FAQ

Is SRE better than DevOps?

Neither is better; they solve different problems. DevOps improves how you build and ship software. SRE improves how reliably it runs. SRE is often described as one specific implementation of DevOps principles, so a mature organization usually uses both.

Is DevOps a part of SRE, or SRE a part of DevOps?

SRE is generally seen as an implementation of DevOps. Google's phrase is "class SRE implements interface DevOps": DevOps defines the principles, and SRE is a concrete, prescriptive way to apply them to running production services.

Who earns more, SRE or DevOps?

SREs usually earn slightly more. In the 2024 Stack Overflow Developer Survey, the US median was $166,500 for SREs and $145,000 for DevOps specialists. Seniority, location and employer matter more than the title.

Can a DevOps engineer become an SRE?

Yes, it is one of the most common paths. Add stronger programming skills, learn SLO and error budget design, get comfortable leading incidents and postmortems, and go deeper on distributed systems failure modes and capacity planning.

Will AI replace SRE jobs?

AI tools are taking over parts of the work, such as summarizing incidents, correlating alerts and drafting runbooks and postmortems. The judgment parts remain: choosing reliability targets, deciding trade-offs during incidents and designing systems that fail safely. The role is shifting toward that work, not disappearing.

What is the difference between SRE and platform engineering?

Platform engineering builds the internal products developers use to ship: templates, pipelines, environments, an internal developer platform. SRE focuses on the reliability of the services running in production. Many startups combine them in one team until they grow.

Does a startup need an SRE?

Usually not as a first infrastructure hire. Most startups need DevOps capability first, with simple SLOs on critical user journeys. A dedicated SRE starts to pay off when incidents, on-call load or customer uptime commitments take a meaningful share of engineering time.

The short answer

DevOps gets your changes to production quickly and safely. SRE keeps production as reliable as your users need, and no more. Start with DevOps practices, add SLOs early and cheaply, and bring in dedicated SRE when reliability becomes a job of its own.

Need senior help with either? MeteorOps provides DevOps consulting and SRE and on-call support for startups, from first pipelines to SLOs and incident response.

How to Choose a Staff Augmentation Company: A CTO's Guide

Michael Zion — Fri, 30 Oct 2026 15:32:25 +0000

This guide was written by Michael Zion, CEO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

To choose a staff augmentation company, judge the engineers, not the sales deck. Insist on interviewing the exact person who will join your team, ask how the vendor screens for your stack, and get clear answers on ramp-up, access, replacement terms, rate structure and how the engagement ends. A good provider answers all of that specifically and in writing before you sign. A weak one talks about its "talent pool", sends a polished CV and asks for a long commitment.

Staff augmentation is the right model when you know what needs doing, you have someone internal who can direct the work, and you are short on senior hands. It is the wrong model when nobody owns the problem: then you need a consultancy or a managed service that takes responsibility for the outcome, not extra capacity that waits for instructions.

This guide is written for CTOs, VPs of Engineering and engineering managers at startups and scale-ups who are buying engineering capacity, with a focus on DevOps, platform and cloud work, where a bad hire can touch production on day three. It covers how staff augmentation compares with managed services and consulting, the questions to ask (with what good and bad answers sound like), how pricing works and how to compare quotes, the contract clauses that matter, red flags, a scorecard, and a first-30-days plan.

What a staff augmentation company actually sells

A staff augmentation company supplies engineers who work inside your team, under your direction, in your tools and your processes. You decide what they work on. They attend your standups, open pull requests in your repositories and follow your review rules. The vendor handles sourcing, employment, payroll, benefits and replacement if the person leaves or is not a fit.

That is the whole product: vetted people plus the commercial wrapper around them. The quality difference between providers comes almost entirely from three things: how good their engineers really are, how honestly they match a person to your problem, and what happens when something goes wrong.

Staff augmentation vs managed services vs consulting

Most buyers searching for a staff augmentation company are also deciding between this model and two others. The difference is who owns the outcome.

	Staff augmentation	Managed services	Consulting / project delivery
Who directs the work	You	The provider, against an agreed scope or service levels	The provider, against a defined project goal
Who owns the outcome	You	The provider	Shared; the provider owns delivery of the agreed scope
Typical billing	Hourly or monthly per engineer	Monthly fee tied to service levels	Fixed price or time and materials per project
Best for	A known backlog and a gap in senior capacity	Ongoing operations you do not want to run yourself	A specific change you lack the expertise to design
Your management effort	High: priorities, reviews, onboarding	Low day to day, high at contract time	Medium: decisions and acceptance
Knowledge stays with	Your team, if you insist on documentation	Mostly the provider	Your team, if handover is in the contract
Main risk	Paying for capacity nobody directs	Lock-in and slow change requests	A design your team cannot operate after handover

In practice the lines blur. Many DevOps providers offer an engineer who is embedded in your team and also backed by an architect who reviews their work. That hybrid is often the best fit for a startup without a senior platform lead, because you get capacity plus a second opinion on design decisions. If you want a deeper comparison of provider types, see our breakdown of DevOps agencies, consultancies and services companies.

When staff augmentation is the right call (and when it is not)

Use staff augmentation when most of these are true:

You can name the work: a Kubernetes upgrade, a Terraform cleanup, CI/CD (continuous integration and continuous delivery) that takes 40 minutes, an observability gap, a cloud migration.
Someone on your side can set priorities, review pull requests and accept the work, even if only for a few hours a week.
You need senior skills faster than you can hire. A senior DevOps or platform hire commonly takes months from opening the role to a productive first week.
The need is real but its long-term size is uncertain, so a permanent headcount is premature.
You want the knowledge to stay in your team, not in a provider's runbooks.

Choose something else when:

Nobody internal owns the area. An augmented engineer with no direction drifts toward whatever ticket is loudest. Buy an assessment or a consulting engagement that produces a plan first.
You want to stop operating something entirely. If the goal is "we never want to think about our database backups again", that is a managed service with service levels, not a person.
The real problem is organisational. Unclear product priorities, no agreed architecture owner or a team that does not review code will not be fixed by adding people. Extra capacity will expose those problems faster.
You need one decision, not ongoing work. A few hours with a senior engineer to review an architecture or choose between two approaches is cheaper and faster than a monthly seat.

Staff augmentation pays off only when someone owns the work and the work is known and ongoing.

Pros and cons of staff augmentation

Pros	Cons
Senior capacity in days or weeks instead of months	You still have to manage, onboard and review the work
Scale up or down without hiring and layoffs	Higher hourly cost than an equivalent employee's salary
Access to specialists you only need for a while (Kubernetes, security, data platforms)	Knowledge leaves with the engineer unless documentation is enforced
Full control over priorities and technical direction	Quality varies hugely between providers and between engineers at the same provider
The vendor carries recruitment, payroll and replacement risk	Weak vendors rotate people, which resets your onboarding investment

Write a one-page brief before you contact any vendor

The fastest way to separate good providers from weak ones is to send them all the same short brief and compare what comes back. It also forces you to decide what you are buying. Keep it to one page:

The problem in one paragraph. "Releases need a senior engineer to babysit them, and our EKS clusters are two minor versions behind" beats "we need a DevOps engineer".
The stack. Cloud provider, orchestration, IaC (infrastructure as code) tooling, CI/CD system, observability tools, languages of the services being deployed.
The first three outcomes you would be happy to see in 60 to 90 days.
Who directs the work on your side and how much time they really have.
Working pattern. Required time-zone overlap, on-call expectations, meeting cadence.
Expected volume. A few hours a week, half-time or full-time, and for roughly how long.
Constraints. Compliance (SOC 2, ISO 27001, HIPAA), data residency, background checks, device policy.

If you are unsure how much DevOps capacity you need in the first place, size it before talking to vendors: our guide to calculating your company's required DevOps capacity walks through the inputs.

Questions to ask a staff augmentation company, and what good answers sound like

These questions are grouped by the risk they test. For each one, the good answer is specific and checkable. The bad answer is vague, deflects to process or promises something nobody can promise.

Good answers are specific and checkable; red flags are vague or ask you to trust the vendor.

About the engineers

1. "Can I interview the exact engineer who will work with us?"

Good answer: yes, before you commit, with a technical conversation led by your own engineer, and the person you interview is the person who starts. Red flag: "our team lead will represent the engineer", "we will assign the best available resource after signing", or a different name on day one.

2. "How do you screen engineers for this specific stack?"

Good answer: a description of a real technical evaluation (hands-on exercises, production debugging scenarios, reviews by senior engineers in that discipline) and how often people fail it. Red flag: certifications and years of experience as the main filter. Certifications show someone passed an exam, not that they can safely upgrade a production cluster.

3. "Is this person an employee or a subcontractor of yours?"

Good answer: a straight answer, either way, plus how they handle confidentiality and IP (intellectual property) assignment for both cases. Red flag: hesitation, or discovering later that your "vendor engineer" was sourced through a second agency, which adds margin and removes accountability.

4. "What has this engineer actually run in production?"

Good answer: concrete examples in their own words during the interview. "I migrated 30 services from ECS (Elastic Container Service) to EKS (Elastic Kubernetes Service) and here is what broke" is worth more than any CV bullet. Ask what they would do differently. Engineers who have operated real systems always have an answer.

About fit and ramp-up

5. "What will the first two weeks look like?"

Good answer: an access plan, a system walkthrough, a review of your repositories and pipelines, and one or two small, low-risk changes shipped in week one. Red flag: "two weeks of discovery" with no output, or the opposite, a plan to rebuild your platform before they have read it.

6. "Who reviews this engineer's work besides us?"

Good answer: a named senior or architect at the provider who is available for design reviews and hard problems, at no extra cost or at a clearly stated one. This matters most when your own team cannot review infrastructure changes in depth. Red flag: the engineer works alone and the vendor's involvement ends at invoicing.

7. "How do you handle time-zone overlap and communication?"

Good answer: a stated number of overlapping working hours, response expectations in your chat tool, and written updates. For DevOps work, a minimum of three to four hours of overlap is a reasonable bar if the engineer touches anything that can page someone. Red flag: "we are flexible" with no numbers.

About security and access

8. "What access will the engineer need, and how do you handle it?"

Good answer: they expect to use your identity provider, your SSO (single sign-on) and individual named accounts, start with read-only or non-production access, and get production access through your normal approval path. They should also mention offboarding: what gets revoked on the last day. Red flag: asking for shared admin credentials or long-lived cloud access keys "to move faster".

9. "What do you require from engineers on security?"

Good answer: device security requirements, a signed NDA (non-disclosure agreement) and IP assignment that flows down to the individual, willingness to follow your policies and complete your security training, and support for background checks if you need them. If you are SOC 2 audited, ask whether they have supported customer audits before.

About commercials and exit

10. "What happens if the engineer is not a fit?"

Good answer: a defined replacement window (for example, a replacement candidate within one to two weeks), no charge for the overlap or ramp-up of the replacement, and a short notice period. Red flag: replacement only after a formal performance process, or a long minimum term with no fit clause.

11. "What is the minimum commitment and notice period?"

Good answer: month to month or a short initial term, with notice measured in weeks. Long lock-ins are a sign the provider is pricing in churn risk. Red flag: annual contracts with auto-renewal for a first engagement.

12. "If we want to hire this engineer, what does it cost?"

Good answer: a clear conversion fee or a period after which conversion is free. Some providers prohibit conversion entirely, which is their right, but you should know before you sign. Red flag: the clause is only discovered when you ask to make an offer.

13. "How will we know this is working?"

Good answer: they propose measures tied to your brief, such as deployment lead time for a named service, manual release steps removed, alert volume, or a migration checklist completed, plus time logs that show where hours went. Red flag: hours logged and tickets closed as the only proof of value.

How staff augmentation pricing works

Almost every staff augmentation company bills by time, but the structure around that time varies a lot. The headline rate is rarely the number that decides what you actually pay.

Pricing model	How it works	Good for	Watch out for
Hourly, billed monthly	You pay for hours worked, with no minimum or a small one	Uneven or uncertain demand, advisory work	Availability: a part-time engineer may have other clients when you need them
Hours bank	You prepay a block of hours and draw it down	A defined project with an uneven pace	Expiry dates on unused hours
Monthly capacity	A committed monthly volume (for example, full-time) at a rate tied to that volume	Ongoing embedded work and ownership	Minimum terms and what happens when the engineer is on leave
Fixed monthly fee per seat	One price per engineer per month regardless of hours	Predictable budgets	Unclear hours: you may be getting less than full time
Cost plus markup	The engineer's cost plus a disclosed margin	Large, long engagements where transparency matters	Rare in practice; verify what "cost" includes

How to compare quotes from different providers

Rates for DevOps and platform engineers vary several times over between regions, seniority levels and provider types, so comparing headline numbers tells you little. Compare the cost of useful, senior output instead:

Seniority you actually get. A cheaper engineer who needs a week to do what a senior does in a day is more expensive. Ask what the engineer has done before, not what level they are labelled.
Ramp-up cost. Who pays for the first week of onboarding, and for the ramp-up of a replacement?
Included support. Is architect review, specialist backup or holiday cover included or billed separately?
Minimums and lock-in. A lower rate with a six-month minimum can cost more than a higher rate you can stop after a month.
Your management time. A provider that needs you to write detailed tickets for every task consumes your most expensive hours.

A simple way to compare: estimate the total cost of the first three months (rate times expected hours, plus any fees and minimums), then divide it by the outcomes you expect in that period. Two quotes that differ by 30% per hour often end up close, or reversed, on that measure.

The cheaper hourly rate is not always the cheaper quote once fees, minimums and outcomes are counted.

Also compare against hiring. An employee's salary is not their full cost: benefits, payroll taxes, equipment, recruiting fees and management time add a substantial amount on top, and an open senior role costs you months of the work not being done. Augmentation usually costs more per hour than an employee, and less than an empty seat.

Contract clauses to check before you sign

Clause	What you want
Named engineer	The engineer you interviewed is named, and substitution requires your approval
IP assignment	All work product belongs to you, and the assignment flows down to individual engineers and any subcontractors
Confidentiality	An NDA that covers the vendor and each individual, with a defined survival period
Replacement	A replacement timeline and no charge for the replacement's ramp-up
Term and notice	Short initial term, notice in weeks, no silent auto-renewal into a long term
Rate changes	Rates fixed for a stated period, with notice before any increase
Conversion	A stated fee or free conversion after a period, if you may want to hire
Non-solicitation	Reasonable scope and duration, mutual, and not blocking conversion you have paid for
Security and compliance	Your policies apply, access is individual and revocable, and the vendor supports audits
Offboarding	Handover of documentation and credentials, and access revoked on the last day
Liability and insurance	Professional liability coverage appropriate for engineers who touch production

None of these are exotic. A provider that resists most of them is telling you how it expects the relationship to go.

Red flags that should end the conversation

You cannot interview the engineer. This is the single most reliable warning sign.
Bait and switch. A strong engineer in the interview and a different, weaker one on day one.
"We have someone for every technology." Breadth claims with no depth behind them usually mean the vendor matches keywords on CVs.
No questions about your system. A provider that quotes before asking about your stack, team and constraints is selling hours, not outcomes.
Pressure toward a long commitment before any work has been delivered.
Requests for shared or permanent admin access. Good engineers ask for the least access they need.
Vague answers about subcontracting or about who actually employs the person.
No references from similar companies, or references they will not let you speak to.

A scorecard to compare staff augmentation companies

Score each provider from 1 to 5 on each criterion after the interviews, multiply by the weight and add up. Adjust the weights to your situation; these suit a startup buying senior DevOps capacity.

Criterion	Weight	What a 5 looks like
Engineer's technical depth (from your own interview)	30%	Has run your stack in production and explains tradeoffs clearly
Fit with your problem and team	15%	Asked good questions about your brief and proposed sensible first steps
Senior backup and review	15%	Named architect or senior available for design reviews
Communication and overlap	10%	Clear overlap hours, written updates, fast responses during evaluation
Security practices	10%	Individual access, least privilege, clean offboarding, audit support
Commercial flexibility	10%	Short notice, fit clause, transparent rates, clear conversion terms
References	10%	Similar companies who confirm the engineers' quality, not just the sales experience

The engineer's interview carries the most weight on purpose. A great provider with the wrong person for your problem is still the wrong choice.

The first 30 days with an augmented engineer

Most failed engagements fail in the first month, usually because access was slow, priorities were unclear or nobody reviewed the work. A simple plan prevents all three.

Before day one: accounts in your identity provider, repository access, chat channels, read-only cloud and cluster access, and a short list of documents and dashboards to read. Name the internal owner who sets priorities.
Week 1: system walkthrough with someone who knows the history, review of infrastructure code and pipelines, and one or two small changes shipped through your normal review path. Ask for a written list of the top risks they noticed.
Week 2: the first meaningful item from your brief, such as removing a manual release step, tuning noisy alerts or fixing a flaky pipeline stage. Production access granted through your normal approval process if the work needs it.
Week 3: a larger piece of work with a written plan and rollback path, reviewed by your owner and, ideally, the provider's architect.
Week 4: a review meeting. Compare progress with the brief, read the time logs, agree on the next month's priorities and decide whether to continue, change scope or stop.

Most failed engagements fail in month one, so plan access, priorities and review before day one.

Keep a weekly written update: what shipped, what is blocked, what risks appeared and what is next. It takes the engineer fifteen minutes and gives leadership what it needs without extra meetings.

How to measure whether staff augmentation is paying off

Measure outcomes your team would care about even without a vendor. For DevOps and platform work, the DORA metrics are a good backbone: deployment frequency, change lead time, change fail rate, failed deployment recovery time and rework rate. Add measures specific to your brief:

Manual steps removed from the release process.
Share of infrastructure changes made through reviewed IaC rather than the console.
Pages per week and the share that needed action.
Time for a new developer to get a working environment.
Migration or upgrade checklist items completed.
Documentation produced: runbooks, architecture decision records and README updates your team actually uses.

Avoid measuring hours, tickets closed or meetings attended. They show activity, not whether your systems became safer or faster to change.

Notes specific to DevOps and platform staff augmentation

DevOps work carries more risk than most augmented roles, because the engineer works on the systems everything else depends on. A few extra checks are worth it:

Infrastructure changes go through code review. Terraform plans, Helm changes and pipeline edits should be reviewed pull requests, never console changes, from the first week.
On-call is explicit. Decide whether the engineer joins your rotation, what the response expectations are, and how it is billed. Do not let it happen by accident after the first incident.
Bus factor is tracked. If only the augmented engineer understands the new deployment pipeline, you have built a dependency on a vendor. Pair them with an internal engineer on anything critical.
Breadth matters. DevOps covers Kubernetes, cloud networking, IAM (identity and access management), CI/CD, observability and cost. One engineer will not be deep in all of it, which is why access to specialist backup at the provider is valuable. If you are still defining the role, read how to build a DevOps team.

FAQ

What is a staff augmentation company?

A staff augmentation company provides engineers who join your team temporarily and work under your direction. The provider employs or contracts the engineers and handles sourcing, payroll and replacement, while you set priorities and own the outcome.

What is the difference between staff augmentation and managed services?

With staff augmentation you direct the work and own the result; the provider supplies people. With managed services the provider owns an agreed outcome or service level and decides how to deliver it. Augmentation gives you control and keeps knowledge in-house; managed services reduce your management effort at the cost of control.

How much does staff augmentation cost?

Most providers bill hourly or per month per engineer, and rates vary several times over with seniority, specialism and region. Compare total cost against expected outcomes over the first few months, including minimums, ramp-up and included support, rather than headline hourly rates.

Is staff augmentation better than hiring a full-time engineer?

It is better when you need senior skills quickly, the long-term need is uncertain, or the work is specialised and temporary. Hiring is better when the need is permanent and central to your product. Many companies use augmentation to cover the gap while they hire, and some convert the augmented engineer.

How long does it take to onboard an augmented engineer?

A senior engineer with good access and a clear brief should ship small changes in the first week and meaningful work by the second. If it takes longer, the cause is usually slow access approvals or unclear priorities on the client side.

What should a staff augmentation contract include?

At minimum: the named engineer, IP assignment that flows down to individuals, confidentiality, replacement terms, term and notice period, rate change rules, conversion terms, security obligations and offboarding. Short terms and clear exit clauses protect both sides on a first engagement.

What are the risks of staff augmentation?

The main risks are paying for capacity nobody directs, knowledge leaving with the engineer, vendor rotation of people, and over-broad access on critical systems. Each one is managed by an internal owner, enforced documentation, a named-engineer clause and least-privilege access.

The short version

Pick a staff augmentation company by the engineer you will actually get, not by the company's pitch. Write a one-page brief, send the same brief to each provider, interview the exact person, check the contract for replacement, notice and IP terms, and run the first month against a written plan. If the answers are specific and the first weeks produce visible work, the model works. If they are vague, keep looking, or reconsider whether you need capacity or someone to own the problem.

If you are looking for senior DevOps, SRE or platform engineers who work inside your team with an architect behind them, see how our DevOps staff augmentation works, or compare buying options.

The CTO DevOps Handbook: Simple Principles and Examples

Michael Zion — Thu, 29 Oct 2026 15:32:16 +0000

This handbook was written by Michael Zion, CEO at MeteorOps, and first published on the MeteorOps blog.

Nail the DevOps part as your company's CTO.

The goal of this handbook is to give you clarity on DevOps:

Understand what’s DevOps (in simple words)
Know what’s possible with DevOps (in simple goals)
Get simple “when-to-do-what” DevOps guidelines

I added a bonus at the bottom of the article.

It's a production-ready setup example you could take inspiration from.

Who this article is for

You might be a founder who wishes to get started with DevOps the right way.

You might be a CTO of a 1,000 employees company who wishes to get simple principles.

Or, maybe you’re a Software Engineer, and you want to understand if your company’s DevOps approach is good.

If you’re looking for a simple DevOps playbook, this is it.

Understand the desired result

Two things your company needs to be able to do

Serve its product to customers
Build and improve the product

Abilities you need to build, improve, and serve software

Run experiments and test changes

DevOps has a simple meaning

Developers and Operators have shared responsibility for building and improving the system.

In practice:

Developers are responsible to “Operate”
DevOps Engineers are responsible to enable to “Operate” AND do some of it themselves

Operate = provision, monitor, secure, configure, deploy, scale.

Choose a balance: Enabler, Doer, or Automator

The DevOps role will end up as a balance between:

Enabler: Provides the tools and knowledge to fulfill the DevOps goals
Doer: Does the tasks that fulfill the DevOps goals
Automator: Automates any repeating operation

The DevOps role ends up as a balance between Enabler, Doer and Automator. Operate = provision, monitor, secure, configure, deploy, scale.

Know what things you should enable, do, or automate

Provision infrastructure
Secure the system
Deploy workloads
Monitor the system
Recover from issues
Scale up or down
Track & test changes
Automate processes

Choose the right tools

Has state management = Saves time automating state-aware processes (e.g., Terraform)
Has a big community & good docs = Saves time dealing with common issues (e.g., Kubernetes)
Has multiple interface types: API, CLI, UI = Saves time integrating with the existing system (e.g., Vault)

You can also read about choosing tools here.

Set useful goals

There are DevOps goals that adopting them will focus you on the right direction:

One-Click Environments: makes e2e tests easy and quick
Atomic Commits: provides confidence that a tested change will work in production
Separate the Shared & Env-Specific Parts: enables e2e tests as the company scales up

If you want to learn about more useful DevOps goals, feel free to book a free consultation here.

Enablers: Choose the Tools-to-Knowledge Balance

Developers can either have the knowledge or the tools to do something.

More knowledge-reliance: if you want the developers to contribute to the DevOps efforts
More tools-reliance: if you want to abstract the operations from the developers

If the balance between the two is not intentional, it’s accidental.

Doers: Have a good reason to do it

Is it a one-time task?
Does it teach you how the developers work?
Are you directly accountable for the results of the task?

If you answered “no” to the above questions, enable or automate it instead.

Doing more = Learning the system's use-cases

Doing too much = Not scalable, too-much knowledge-reliance

Automators: Have a good reason to automate it

Did it happen before?
Is it likely to happen again?
Will automating it take less time than doing it?
Will automating it teach you an important company process?

If you answered “yes” to 2 out of the 4 questions - automate it!

More automations = Less reliance on knowledge to operate the system.

Too much automations = No system awareness.

P.S. - you can also enable developers to automate it.

The two checklists on one page: do it only with a good reason, automate it when 2 of the 4 answers are yes.

Create available DevOps Capacity

The DevOps needs of a company have spikes.

One month you need 2 DevOps Engineers, and half of that the next month.

Switchovers between big efforts and small tasks are common.

This is true, especially for new companies.

Break the assumption: “DevOps tasks must be done by a DevOps Engineer”.

There are 3 types of DevOps capacity

Non-Flexible: A full-time DevOps Engineer on the team
Semi-Flexible: Key developers that can contribute to the DevOps goals
Fully-Flexible: A flexible DevOps Services company or freelancer

DevOps needs spike month to month. Non-flexible capacity covers the base, semi-flexible and fully-flexible capacity cover the spikes.

You can read more about calculating the DevOps capacity your company needs here.

When to focus on what: Common Dilemmas

When: You work alone, and the system is simple

Focus: On simplifying the development - Dockerize your apps, Create a post-commit pipeline that runs tests

When: You need to be able to create new environments quickly (for development, or for clients)

Focus: On implementing “One-Click Environments”: Using IaC (e.g., Terraform) + Deployment tool (Depends on the platform).

When: You want to e2e test every code modification, but there are many code modifications

Focus: On splitting the “One-Click Env” into a “base” with shared resources, and “env” with env-specific resources

When: You want to unify & standardize how you deploy, monitor, scale, configure, and secure your workloads

Focus: On implementing an orchestrator such as Kubernetes

When: You want you have many moving parts and wish to be certain a tested change will work

Focus: On implementing GitOps and consider a Monorepo (the sooner the better)

When: You want the DevOps efforts to be done by the dev team

Focus: On using “actual” IaC tools (Pulumi Typescript/Python), Full “how to operate” (see above) documentation

Never: - Invest lots of time in new tech without a strong reason

Always:

Have your code in Git
Monitor the basic stuff: CPU, Memory, Disk, Network, App Logs, Cloud Costs
Architect for high-availability
Test before you deploy

The common dilemmas in the order they usually arrive, with the one thing to focus on at each.

BONUS: An example setup for a CTO approaching Production

2 AWS Accounts

One for development and staging
Another for production

Monorepo in Github

Docker-Compose for local development

2 Infrastructure-as-Code projects: 'base' & 'apps'

base = shared resources (e.g., VPC, RDS, ECS Cluster, EKS Cluster)
apps = env-specific resources (e.g., Lambda Functions, ECS Services, Kubernetes Namespaces)
config file per environment

Github Actions Workflow: Development workflow

Checkout branch and locally develop + test changes
Create a Pull Request: Deploys a Pull-Request ‘apps’ environment on the ‘development’ environment ‘base’
On merge to main: Deploys from the ‘main’ branch an ‘apps’ environment onto the ‘development’ environment ‘base’
Manual: Deploy from the ‘main’ branch onto the ‘staging’ / ‘production’ environment ‘base’

Setup Notes

Avoid mentioning an environmnent's name in the code for conditional resources deployment
Use each environment’s config file to declare if a resource should be created
Could be implemented using Terraform, Terragrunt, Pulumi, CDK, and other IaC tools
Production should have 2-instances of every workload for high-availability

If you’d like to see this setup in your startup, click here to book a call.

P.S. - I'll be updating this page occasionally, so you might want to visit again.

Another Bonus: DevOps Dictionary for Human Beings

Term	Definition	Tools
Environment	A working instance of the entire system
CI (Continuous Integration)	Enable developers to collaborate by agreeing on a single source-of-truth (master/main)	Jenkins, Github Actions, GitlabCI
CD (Continuous Delivery)	Create an artifact that’s ready for production (tested, tagged)	JFrog Artifactory, Nexus, AWS ECR
CD (Continuous Deployment)	Every available deliverable (artifact) gets deployed automatically	ArgoCD, Jenkins, AWS CodeDeploy
Monitoring / Observability	Collect metrics/traces/logs from apps and infrastructure, analyze them, and display them, and setup alerts	Prometheus, Jaeger, Elasticsearch, Fluentd, OpenTelemetry
Infrastructure	The resources on which the workloads run, in which the data is stored, and through which the network flows	Servers, Databases, Network Routers & Switches
Cloud Infrastructure	Same as the above, but specifically in the cloud	AWS EC2, AWS RDS, GCP Compute Engine, Azure Virtual Machines
Cloud	Computing & Data services served from remote locations for you to build your system	AWS, Azure, GCP
Containerization & Virtualization	Technologies utilizing Kernel & OS features to create virtual machines, or isolate process (AKA run containers)	Docker, vSphere, KVM
Secrets Management	Storing and retrieving sensitive configurations (e.g., tokens, passwords)	Hashicorp Vault, AWS Secrets Manager, SealedSecrets
Configuration Management	Usually refers to preparing servers for workloads (e.g., creating directories & files, starting processes)	Ansible, Chef, Puppet
Version Control	Saving the code in a versioned way (Git)	Github, Gitlab
GitOps	Making the system is the same as it’s described in Git	Flux, ArgoCD, Jenkins
Monorepo	All of the company’s code is in one Git Repository	NX, Turborepo
Polyrepo	Multiple Git repositories for different components
IaC (Infrastructure-as-Code)	Creating Cloud infrastructure with idempotent code and state management	Terraform, Pulumi, CDK, Crossplane
Deployment	Execute, serve, or install the artifacts	ArgoCD, Jenkins, AWS CodeDeploy, Scripts (Bash, Python, etc.)
Orchestrator	Dynamically allocating workloads to a pool of nodes	Kubernetes, Nomad, AWS ECS
Authentication & Authorization	Making sure each person, workload, or resource, has access only to what’s necessary (other workloads and resources)	AWS IAM, OpenID, OpenVPN, Twingate, Istio
Service Discovery	Exposing available workloads using DNS	Consul, CoreDNS

GitOps Benefits, Challenges, and When to Use It

Michael Zion — Wed, 28 Oct 2026 15:31:43 +0000

This guide was written by Michael Zion, CEO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

GitOps is a way of running deployments where the desired state of your systems lives in Git, and an agent inside your environment continuously pulls that state and makes reality match it. The real benefits are concrete: every production change is a reviewed commit, rollback is a git revert, manual drift gets corrected automatically, your CI system no longer needs admin credentials to your clusters, and a lost cluster can be rebuilt from a repository.

The real costs are just as concrete. You have to solve secrets, environment promotion and repository layout yourself; tools like the Horizontal Pod Autoscaler (HPA) will fight your sync controller until you configure around them; and debugging a failed sync adds a layer between a developer and their running pod. GitOps pays off for teams running several services on Kubernetes across more than one environment. It is usually not worth it for a single app on a managed platform, or for a team without anyone who will own the setup.

This guide covers what GitOps actually is (using the OpenGitOps principles, not vendor definitions), each benefit and what it takes to get it, the challenges teams hit in practice, a decision table for when to use it and when not to, Argo CD vs Flux, a worked example, an adoption path, and how to measure whether it worked.

What GitOps is: the four OpenGitOps principles

"GitOps" has been stretched by marketing to mean almost anything involving Git. The useful definition comes from OpenGitOps, a CNCF working group that published four principles (version 1.0.0):

Declarative. "A system managed by GitOps must have its desired state expressed declaratively." You describe what should exist (Kubernetes manifests, Helm values, Kustomize overlays), not the commands to create it.
Versioned and Immutable. "Desired state is stored in a way that enforces immutability, versioning and retains a complete version history." In practice this is Git, though OCI registries also qualify.
Pulled Automatically. "Software agents automatically pull the desired state declarations from the source." The agent runs next to the target system and fetches changes; nothing outside pushes into it.
Continuously Reconciled. "Software agents continuously observe actual system state and attempt to apply the desired state." This is the part most "GitOps" pipelines skip.

The last two principles are what separate GitOps from "we keep our YAML in Git and CI runs kubectl apply". A pipeline that pushes manifests on merge gives you versioning, but it applies changes once and then stops looking. If someone edits a Deployment by hand an hour later, nothing notices. A GitOps agent (Argo CD or Flux, in almost every real setup) notices and either reports the drift or reverts it.

Pushing manifests on merge applies them once; only continuous reconciliation catches drift afterwards.

GitOps vs DevOps vs CI/CD

DevOps is a culture and set of practices for shared ownership of delivery and operations. CI/CD (continuous integration and continuous delivery) is the automation that builds, tests and ships code. GitOps is one specific way to implement the deployment half of CD: CI still builds and tests your code and pushes an image; GitOps takes over from "the new image exists" to "the cluster runs it". It does not replace your CI system and it is not an alternative to DevOps.

The real benefits of GitOps (and what each one requires)

Most benefit lists are written as if you get everything for free once you install Argo CD. You do not. Each benefit below comes with the condition that makes it real.

1. An audit trail you get for free

Every change to production is a commit with an author, a timestamp, a pull request, reviewers and a diff. When someone asks "what changed before the incident at 14:05?", the answer is git log on the environment path, not a search through CI logs and Slack. For teams going through SOC 2 or ISO 27001, this gives auditors change management evidence (who approved what, when) without a separate ticketing ritual.

What it requires: branch protection with required reviews on the environment repository or path, and no other way to change production. If engineers still have write access with kubectl, your audit trail has a hole exactly where incidents happen.

2. Rollback is `git revert`

Because the cluster follows Git, reverting the commit that bumped an image tag or changed a config value rolls the environment back through the same reviewed, logged path as any deployment. There is no special rollback runbook to remember during an incident.

What it requires: deployments must be expressed as a single, self-contained change (one image tag bump, one values change). If a release is spread across several commits in different repositories, a revert is no longer one action. Also note that Argo CD's own UI rollback is disabled for Applications with automated sync; the Argo CD docs state that "Rollback cannot be performed against an application with automated sync enabled." Git revert is the rollback. Database migrations are not rolled back by reverting a manifest, so design migrations to be backward compatible.

3. Drift detection and correction

Drift is the gap between what you think is running and what is running: a hotfixed ConfigMap, a replica count someone scaled by hand, a resource deleted by mistake. A reconciling agent detects this continuously. With Argo CD's selfHeal or Flux's default behaviour of re-applying on every interval, the agent reverts it. This is the single biggest operational difference between GitOps and push-based CD.

What it requires: deciding what the agent should own and what it should ignore (see the HPA challenge below). Drift correction that fights a legitimate controller is worse than no drift correction.

4. A better security model: pull-based delivery

In push-based CD, your CI system (GitHub Actions, GitLab CI, Jenkins) holds credentials that can modify production clusters. CI runners execute arbitrary code from pull requests and third-party actions, so they are a high-value target. In pull-based GitOps, the agent runs inside the cluster with in-cluster permissions, and CI only needs permission to push an image and, in some setups, to open a commit or pull request against the config repository. The cluster API server does not need to be reachable from your CI provider at all.

With pull-based delivery, the credentials that can change production never leave the cluster.

What it requires: actually removing the old CI credentials and human admin access. Teams often add GitOps alongside their existing kubectl pipeline "temporarily" and end up with two attack surfaces instead of one. Also remember that write access to the config repository now effectively equals deploy access, so protect it like production.

5. Self-service for developers, with guardrails

Once the platform team has defined how apps are laid out, a developer ships a new service or changes a resource limit by opening a pull request against a known path. Review rules, policy checks in CI (schema validation, Open Policy Agent or Kyverno policies) and CODEOWNERS files decide what needs platform approval. This is how a small platform team supports many product engineers without becoming a ticket queue.

What it requires: a repository structure and templates that developers can understand. If adding a service means touching six files across three directories, self-service will not happen.

6. Disaster recovery and cluster rebuilds

If everything that runs in a cluster is declared in Git, rebuilding a cluster is: create the cluster, install the GitOps agent, point it at the repository, wait. The same mechanism lets you spin up a new region or a full preview environment, which is the foundation of the one-click environment idea.

What it requires: really everything in Git, including cluster add-ons (ingress controller, cert-manager, external-dns, monitoring). Persistent data (databases, volumes) is not restored by GitOps; you still need backups. Test the rebuild before you need it, because the first attempt usually exposes a resource someone created by hand two years ago.

7. Faster, more frequent, less scary deployments

When deploying is a small pull request and rollback is a revert, teams deploy smaller changes more often. That is the mechanism by which GitOps moves DORA metrics, discussed at the end of this guide. GitOps does not make your tests better or your services more reliable; it removes friction and fear from the deployment step.

GitOps challenges: what teams actually struggle with

These are the problems that show up in the first three to six months of running GitOps in production. None of them is a reason to avoid GitOps, but each needs a deliberate decision.

Secrets management

You cannot commit plaintext secrets to Git, yet GitOps says the desired state lives in Git. There are three common answers:

Reference, do not store: External Secrets Operator or the Secrets Store CSI Driver. Git holds a pointer ("this secret comes from AWS Secrets Manager key X"); an in-cluster controller fetches the value. This is usually the best default.
Encrypt in Git: Sealed Secrets, or SOPS with age or a cloud KMS key. Flux decrypts SOPS natively through the Kustomization spec.decryption field.
Inject at render time with a plugin such as argocd-vault-plugin. The Argo CD secret management docs "strongly caution against this style", partly because "Argo CD stores generated manifests in plaintext in its Redis cache", and recommend managing secrets on the destination cluster instead.

Environment promotion

GitOps tells you how to make an environment match Git. It does not tell you how a change moves from dev to staging to production. Common patterns are a directory per environment with Kustomize overlays (promotion is a pull request that copies a tag from one overlay to the next), a values file per environment for Helm, or automation that opens the promotion pull request after staging checks pass. Avoid a long-lived Git branch per environment: branches drift apart, merges carry unrelated changes, and nobody can easily answer "what is different between staging and prod?". Directories make that question a diff.

Repository structure

Monorepo or one config repo per team? App code and manifests together or separate? There is no universal answer, but two rules hold. First, keep deployable configuration separate from application source when CI writes back to it, otherwise image-bump commits trigger new CI builds and pollute history. Second, decide early how cluster add-ons, shared platform config and app config are separated, because moving hundreds of resources between paths later means carefully orchestrated pruning.

Drift vs controllers that legitimately change state (HPA and friends)

The HPA changes spec.replicas on your Deployment. If Git also sets replicas: 3 and self-heal is on, the GitOps agent and the HPA fight, and you get replica counts snapping back on every sync. The fix is to stop declaring replicas for autoscaled workloads, and tell the agent to ignore the field. In Argo CD that is ignoreDifferences on /spec/replicas, plus the RespectIgnoreDifferences=true sync option so the ignored field is also left alone during sync, not only in the diff (both documented in the Argo CD sync options). The same applies to fields mutated by admission webhooks, operators and service meshes. We cover the autoscaling side in how to configure Kubernetes HPA without causing scaling flaps.

Stop declaring replicas for autoscaled workloads and tell the agent to ignore the field.

Debugging sync failures

Before GitOps, a failed deploy showed up red in the CI job the developer was watching. After GitOps, CI is green (it pushed an image and a commit), and the failure happens asynchronously in the cluster: a schema error, a missing CRD, a failing health check, an immutable field change. Developers need to know where to look. Wire the agent's notifications (Argo CD Notifications or the Flux notification-controller) into the team's channel and into commit statuses, and give developers read access to the Argo CD UI or Flux status, otherwise every failed sync becomes a platform team ticket.

Separating CI from CD

The clean split is: CI builds, tests, scans and pushes an image, then records the new tag in the config repository (directly or through a pull request). The GitOps agent deploys. Teams get into trouble when CI keeps doing deployment-shaped work (running helm upgrade "just for this one service", running migrations with cluster credentials), because that reintroduces push access and splits the source of truth.

CI ends at a commit to the config repo; everything after that belongs to the agent.

Image update automation

Someone has to write the new image tag into Git. Options, from simplest to most automated:

CI commits the tag (for example with kustomize edit set image or yq) to the dev overlay after a successful build. Simple, explicit, and works with any agent.
Flux image automation: ImageRepository, ImagePolicy and ImageUpdateAutomation (API image.toolkit.fluxcd.io/v1) scan a registry and commit tag updates to lines marked with a comment like # {"$imagepolicy": "flux-system:podinfo"}.
Argo CD Image Updater, a separate project. Its docs describe it as under active development and invite users to test it on non-critical environments, so treat it accordingly for production.

Multi-cluster and fleet management

One cluster is easy. Ten clusters across regions and environments raise new questions: one central Argo CD managing all clusters (a single pane of glass, but a single, highly privileged control plane) or an agent per cluster (more isolation, more to operate)? How do you template the same add-on across clusters with small differences? Argo CD answers the templating question with ApplicationSets; Flux with Kustomize overlays per cluster path and, increasingly, the Flux Operator's ResourceSets. Pick one model deliberately before cluster number three.

Ordering, CRDs and bootstrap

A CRD must exist before resources that use it; a namespace before what lives in it; cert-manager before Certificates. Flux expresses this with dependsOn between Kustomizations; Argo CD with sync waves and sync phases. Expect to spend real time on bootstrap ordering the first time you rebuild a cluster from scratch.

When to use GitOps and when not to

GitOps is most valuable when there is a lot of state to keep consistent and more than a handful of people changing it. Use this table as a starting point.

Situation	Recommendation	Why
1 to 3 engineers, one app on a PaaS (Vercel, Heroku, Cloud Run, App Runner)	Skip GitOps	The platform already gives you declarative deploys and rollback. A Kubernetes GitOps stack would be overhead without a cluster to reconcile.
Small team, one Kubernetes cluster, a few services	Adopt, but keep it minimal	One agent, one config repo, plain manifests or Kustomize. You get audit, rollback and DR cheaply. Avoid ApplicationSets and fancy promotion until you need them.
5 to 50 engineers, multiple services, dev/staging/prod on Kubernetes	Strong fit	This is where drift, "who changed what" and slow manual deploys start to hurt. Self-service and promotion via pull requests pay back quickly.
Multiple clusters, regions or tenants; platform team serving product teams	Strong fit, invest in fleet tooling	Templated add-ons, cluster bootstrap from Git and centralized visibility become essential.
Regulated environment (SOC 2, ISO 27001, PCI)	Strong fit	Reviewed commits are change-management evidence, and removing CI cluster credentials shrinks the attack surface.
Mostly serverless or managed services, little or no Kubernetes	Use Infrastructure as Code with PR-based plans instead	Terraform or OpenTofu with plan-on-PR and apply-on-merge gives most of the audit benefit. Continuous reconciliation is possible (for example via Crossplane) but is rarely the first problem to solve.
No one available to own the setup	Wait	A half-configured GitOps setup (secrets unsolved, drift fights, nobody reading sync errors) is worse than a simple, well-understood pipeline.

A useful smell test: if your team can't answer "what exactly is running in production right now, and who approved it?" in under a minute, GitOps will help. If you can, and deploys are not painful, it may not be your highest-leverage project. Our guide to calculating your company's required DevOps capacity helps decide whether you have the people to own it.

Can GitOps manage cloud infrastructure, not just Kubernetes apps?

Yes, if the infrastructure is expressed as Kubernetes resources that a controller reconciles: Crossplane, or cloud-specific controllers such as AWS Controllers for Kubernetes and Google Config Connector. See deploying AWS resources using Crossplane on Kubernetes. For most startups, Terraform with pull-request plans is the pragmatic choice for cloud resources, with GitOps handling what runs inside the clusters.

Argo CD vs Flux: which GitOps tool to choose

Both are CNCF graduated projects (Flux graduated in November 2022, Argo in December 2022), both are production grade, and both implement the four principles. The difference is mostly in operating model and user experience.

	Argo CD	Flux
Core model	An `Application` resource points at a source (Git, Helm repo or OCI) and a destination cluster and namespace.	A set of controllers: sources (`GitRepository`, `OCIRepository`, `HelmRepository`) plus appliers (`Kustomization`, `HelmRelease`).
UI	Rich built-in web UI showing resource trees, diffs, sync status and logs. A big reason teams pick it.	No UI in core Flux. The Flux Operator ships a Flux Web UI (status dashboard with OIDC and RBAC); many teams use the CLI and notifications.
Multi-cluster	Commonly one central instance managing many clusters; ApplicationSets generate Applications per cluster, per directory or per pull request.	Commonly one Flux per cluster, each bootstrapped from its own path in Git. Can also target remote clusters via kubeconfig.
Multi-tenancy	AppProjects restrict which repos, clusters, namespaces and resource kinds a team can deploy, plus SSO and Argo CD RBAC.	Relies on Kubernetes RBAC: each Kustomization can impersonate a service account scoped to a tenant's namespaces.
Helm	Renders charts with `helm template` and applies the output; Helm release history is not used.	`HelmRelease` performs real Helm installs and upgrades, with Helm hooks, tests and rollback remediation.
Secrets	Bring your own (External Secrets, Sealed Secrets, CSI driver).	Native SOPS decryption in Kustomizations, or bring your own.
Image automation	Argo CD Image Updater (separate project, still maturing).	Built-in image automation controllers.
Change detection	Polls repos every 3 minutes by default (120s plus up to 60s jitter), plus webhooks.	Per-object `interval`, plus webhook receivers via notification-controller.
Progressive delivery	Argo Rollouts (canary, blue-green).	Flagger.

Our default advice: if developers will interact with deployments and you want them to see what is happening without learning kubectl, pick Argo CD. If you want a lightweight, Kubernetes-native agent per cluster, heavy Helm usage with real Helm lifecycle, native SOPS and built-in image automation, pick Flux. Both are fine choices; the bigger risk is spending weeks choosing. For canary and blue-green on top of Argo CD, see zero-downtime Kubernetes releases with Helm and Argo Rollouts.

A worked GitOps example: repo layout and manifests

Here is a minimal, realistic setup for one service with dev and production environments, using Kustomize overlays. The same layout works with either tool.

platform-config/
  apps/
    checkout-api/
      base/
        deployment.yaml
        service.yaml
        hpa.yaml
        kustomization.yaml
      overlays/
        dev/
          kustomization.yaml      # image tag for dev, smaller resources
        production/
          kustomization.yaml      # image tag for prod, prod resources
  clusters/
    dev/
      checkout-api.yaml           # Argo CD Application or Flux Kustomization
    production/
      checkout-api.yaml

The production overlay pins the image tag. Promotion to production is a pull request that changes newTag here, and rollback is reverting that pull request:

# apps/checkout-api/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: ghcr.io/acme/checkout-api
    newTag: "1.42.0"

Option A: Argo CD Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-api-production
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/acme/platform-config.git
    targetRevision: main
    path: apps/checkout-api/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: checkout
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas          # owned by the HPA, not Git
  syncPolicy:
    automated:
      prune: true                 # delete resources removed from Git
      selfHeal: true              # revert manual changes in the cluster
    syncOptions:
      - CreateNamespace=true
      - RespectIgnoreDifferences=true

Notes: prune and selfHeal are both off by default in Argo CD, so drift correction only happens if you turn them on. In real setups, replace project: default with a dedicated AppProject that limits which repos and namespaces this team can deploy to.

Option B: Flux GitRepository and Kustomization

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: platform-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/acme/platform-config.git
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: checkout-api
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/checkout-api/overlays/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: platform-config
  targetNamespace: checkout
  wait: true
  timeout: 3m

Notes: Flux re-applies the desired state on every interval, which corrects drift. For a private repository, add a secretRef on the GitRepository. For the HPA case, the simplest Flux-side fix is to leave replicas out of the Deployment manifest entirely, so the HPA owns it. Add dependsOn if this app needs, for example, cert-manager to be ready first.

Whichever tool you use, add health checks that mean something. A sync that "succeeds" with pods in a restart loop is a false green; see how to configure Kubernetes liveness probes without causing restart loops.

A practical GitOps adoption path: from first app to fleet

Pick one non-critical service and one environment. Install Argo CD or Flux, point it at a config repo, and deploy that service with automated sync. Goal: the team sees a merge turn into a deployment in a few minutes.
Solve secrets before the second service. Choose External Secrets Operator, Sealed Secrets or SOPS and document the pattern. This is the decision teams most regret postponing.
Move cluster add-ons into Git. Ingress controller, cert-manager, external-dns, metrics and logging agents. Now the cluster itself is reproducible.
Add environments and promotion. Overlays per environment, CI writes the dev tag, promotion to staging and production by pull request. Add notifications and commit statuses so developers see sync results.
Turn on self-heal and remove manual access. Once teams trust the flow, enable self-heal, remove CI cluster credentials and reduce human write access to break-glass roles with audit logging.
Prove the rebuild. Create a fresh cluster from Git and measure how long it takes to serve traffic. Fix whatever was missing.
Scale to the fleet. ApplicationSets or per-cluster Flux paths, policy checks in CI, preview environments per pull request if useful, progressive delivery for critical services.

Each step builds the trust and coverage the next one depends on.

Common GitOps mistakes and how to fix them

Keeping kubectl write access "for emergencies" for everyone. Fix: a break-glass role that is audited and rarely used; emergencies go through a fast-tracked pull request.
A branch per environment. Fix: directories or overlays per environment on one branch.
Using the latest tag. Git does not change when the image does, so there is no audit trail and no revert. Fix: immutable tags or digests.
Declaring replicas for autoscaled workloads. Fix: remove them and ignore the field in the agent.
Turning on prune on day one without a review. A path mistake can delete a namespace's worth of resources. Fix: enable prune after the layout is stable, and protect critical resources (Argo CD supports a Prune=false sync option per resource).
Nobody watching sync status. Fix: notifications to the owning team's channel and failing commit statuses.
Secrets via render-time plugins. Fix: move to destination-cluster secret management as Argo CD recommends.

How to measure GitOps value with DORA metrics

GitOps should move delivery metrics, and you should measure before and after. DORA currently defines five software delivery metrics:

Change lead time: time from commit to running in production. Watch whether deployment queues and manual steps disappear.
Deployment frequency: how often you deploy. Smaller, routine pull request deploys should increase it.
Failed deployment recovery time: how long it takes to restore service after a failed deployment. Revert-based rollback should cut this sharply.
Change fail rate: share of deployments that need urgent fixing (rollback or hotfix).
Deployment rework rate: share of deployments that were unplanned and caused by a production incident.

The commit history in your config repository plus the agent's sync events give you most of the raw data: commit time, sync time, reverts and their timestamps. Add two GitOps-specific health signals: how often drift is detected (it should trend toward zero as manual access disappears) and how many syncs fail (it should be low and owned by the team that made the change, not the platform team).

GitOps implementation checklist

Desired state for apps and cluster add-ons is in Git, in a structure developers understand.
Branch protection and required reviews on environment paths; CODEOWNERS for platform-owned paths.
One secrets approach, documented, with no plaintext secrets in Git.
Immutable image tags; CI writes tags, the agent deploys.
Promotion between environments is a pull request.
Fields owned by other controllers (HPA replicas, webhook-injected fields) are ignored by the agent.
Sync notifications reach the owning team; developers can see sync status and logs.
Self-heal and prune enabled deliberately; critical resources protected.
CI has no cluster admin credentials; human write access is break-glass only.
A cluster rebuild from Git has been tested and timed.
DORA baseline captured before rollout.

When outside GitOps help makes sense

Most teams can install Argo CD or Flux in an afternoon. What takes months is the design around it: repository structure, secrets, promotion, multi-cluster, and migrating existing services without downtime. Bringing in experienced engineers makes sense when you are migrating a large number of existing services, moving to multiple clusters or regions, preparing for a compliance audit, or when the people who would own GitOps are already fully booked on product work. The goal of any good engagement should be a setup your own team can run and extend afterwards, not a dependency.

GitOps FAQ

What are the main benefits of GitOps?

A complete audit trail of production changes, rollback by reverting a commit, automatic detection and correction of drift, a safer pull-based security model where CI holds no cluster credentials, developer self-service through pull requests, and the ability to rebuild clusters from Git. Each benefit depends on removing other ways to change production.

What is the biggest disadvantage of GitOps?

The added complexity around the edges: secrets, environment promotion, repository structure and debugging asynchronous sync failures all need deliberate design. For a small team with one app on a managed platform, that overhead can outweigh the benefits.

What are the four principles of GitOps?

According to OpenGitOps: the desired state is declarative; it is versioned and immutable; it is pulled automatically by software agents; and it is continuously reconciled against the actual state.

Is GitOps only for Kubernetes?

The principles are not Kubernetes specific, but the mature tooling (Argo CD, Flux) is built for Kubernetes. Cloud resources can be managed the GitOps way through Kubernetes-based controllers such as Crossplane, while many teams use Terraform with pull-request plans for infrastructure instead.

Is GitOps the same as CI/CD?

No. CI builds and tests code and produces artifacts. GitOps is a way to do the deployment part of CD, where an agent pulls the desired state from Git and reconciles the environment. You still need CI.

Should I use Argo CD or Flux?

Choose Argo CD if you want a strong UI and a central control plane for many clusters and teams. Choose Flux if you prefer a lightweight per-cluster agent, native Helm lifecycle, built-in SOPS decryption and image automation. Both are CNCF graduated and production ready.

How do you roll back with GitOps?

Revert the commit (or pull request) that introduced the change, and the agent syncs the environment back. In Argo CD, UI rollback is disabled for Applications with automated sync, so Git revert is the intended path. Database schema changes need to be backward compatible because reverting manifests does not undo migrations.

How do you handle secrets in GitOps?

Either store a reference in Git and let External Secrets Operator or the Secrets Store CSI Driver fetch the value from a secret manager, or store encrypted secrets with Sealed Secrets or SOPS. Avoid render-time injection plugins, which Argo CD's own docs caution against.

If you are weighing GitOps for your stack and want a second opinion on structure, secrets and tool choice, you can book a free consultation with a senior DevOps engineer.

Terraform Kubernetes Provider: Deployment Guide and Examples

Michael Zion — Tue, 27 Oct 2026 15:30:18 +0000

This guide was written by Arthur Azrieli, CTO at MeteorOps, and first published on the MeteorOps blog.

Bottom line

You can deploy and manage Kubernetes resources with Terraform using the official hashicorp/kubernetes provider. It gives you typed resources such as kubernetes_deployment_v1, kubernetes_service_v1 and kubernetes_ingress_v1, a generic kubernetes_manifest resource for anything else (including custom resources), and it pairs with the hashicorp/helm provider for charts. Terraform plans the change, shows you a diff, and keeps everything in state next to the cloud resources it depends on.

The harder question is where Terraform should stop. Our short answer, from running this at startups and scale-ups: use Terraform for the cluster and the platform layer it needs to be useful (namespaces, RBAC, storage classes, the ingress controller, cert-manager, external-dns, and the GitOps controller itself), and hand application workloads to a GitOps tool such as Argo CD or Flux. Use Terraform for app Deployments only when the app is small, rarely changes, or is tightly coupled to cloud resources Terraform already creates.

This guide covers that decision first, then gives you copy-paste-correct HCL (HashiCorp Configuration Language) for provider authentication on EKS, GKE and AKS, a fully annotated kubernetes_deployment_v1 with the container and port blocks, Service and Ingress, YAML-from-file patterns, helm_release, and the errors you will actually hit. Everything is checked against the current provider releases: hashicorp/kubernetes v3.4.0 and hashicorp/helm v3.3.0 as of October 2026.

Should you manage Kubernetes resources with Terraform?

Kubernetes already has its own reconciliation loop. Terraform has a different one that only runs when someone runs a plan and apply. Putting both on the same object works, but you need to be deliberate about which tool owns what. This table is the boundary we use.

Layer	Examples	Recommended owner	Why
Cloud and cluster	VPC, EKS/GKE/AKS cluster, node pools, IAM roles for service accounts	Terraform	Cloud APIs, long-lived, needs plan and review
Cluster bootstrap	Namespaces, RBAC, StorageClasses, priority classes, quotas, the Argo CD or Flux install	Terraform	Changes rarely, must exist before anything else, often wired to cloud IDs (IAM role ARNs, bucket names)
Platform add-ons	Ingress controller, cert-manager, external-dns, Karpenter, metrics-server	Terraform with helm_release, or GitOps once it is bootstrapped	Either works; pick one owner per add-on and stick to it
Application workloads	Deployments, Services, Ingresses, HPAs, ConfigMaps for your own services	GitOps (Argo CD, Flux) with Helm or Kustomize	Deployed many times a day by CI, needs fast rollbacks and continuous drift correction
One-off glue	A ConfigMap holding Terraform outputs (database endpoint, queue URL) for apps to read	Terraform	The values only exist in Terraform; this is the cleanest hand-off

Terraform builds the foundation; GitOps owns what changes every day.

Signs Terraform is the right tool for an in-cluster object:

It is created once per environment and changes a few times a year.
It needs values that only Terraform knows (an IAM role ARN for a ServiceAccount annotation, an RDS endpoint, a VPC ID).
You want a new environment to come up from one command. We wrote about that goal in the one-click environment.
There is no GitOps controller yet, and a single small service does not justify one.

Signs it is the wrong tool:

The image tag changes on every merge. Running a Terraform apply per deploy is slow, needs cloud credentials in CI, and serializes deploys on the state lock.
Something else mutates the object (HPA, VPA, a service mesh injector, an operator). You will fight drift on every plan.
You need progressive delivery (canary, blue/green). That lives in tools like Argo Rollouts; see zero-downtime releases with Helm and Argo Rollouts.
Developers need to change their own manifests without touching infrastructure state.

Terraform Kubernetes provider setup and authentication

Pin both providers. The hashicorp/kubernetes provider v3 requires Terraform v1.0 or later, and the provider functions used later in this guide need Terraform 1.8 or later.

terraform {
  required_version = ">= 1.8.0"

  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 3.4"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 3.3"
    }
  }
}

Local development: kubeconfig

The provider does not read the KUBECONFIG environment variable. Point it at a file explicitly with config_path (or KUBE_CONFIG_PATH), and pin the context so a stray kubectl config use-context never sends an apply to production.

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = "kind-dev"
}

EKS: exec authentication

EKS tokens are short-lived. An exec block fetches a fresh token on every run, which avoids the classic "Unauthorized" error on long applies that you get when you pass a static token from aws_eks_cluster_auth. Do not mix exec with token or client certificates in the same provider block; the provider docs call that undefined behavior.

data "aws_eks_cluster" "this" {
  name = var.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", var.cluster_name]
  }
}

The IAM principal that runs Terraform also needs Kubernetes permissions: an EKS access entry (or an aws-auth mapping on older clusters) bound to an appropriate access policy. Without it, authentication succeeds and every request is rejected.

GKE and AKS

# GKE: requires gke-gcloud-auth-plugin on the machine running Terraform
data "google_container_cluster" "this" {
  name     = var.cluster_name
  location = var.region
}

provider "kubernetes" {
  host                   = "https://${data.google_container_cluster.this.endpoint}"
  cluster_ca_certificate = base64decode(data.google_container_cluster.this.master_auth[0].cluster_ca_certificate)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "gke-gcloud-auth-plugin"
  }
}

# AKS with local accounts: credentials from the cluster data source
data "azurerm_kubernetes_cluster" "this" {
  name                = var.cluster_name
  resource_group_name = var.resource_group
}

provider "kubernetes" {
  alias                  = "aks"
  host                   = data.azurerm_kubernetes_cluster.this.kube_config[0].host
  client_certificate     = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].client_certificate)
  client_key             = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].client_key)
  cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.this.kube_config[0].cluster_ca_certificate)
}

For AKS clusters with Microsoft Entra ID integration and local accounts disabled, use an exec block that calls kubelogin instead of client certificates.

The chicken-and-egg problem: cluster and resources in one root module

The most common failure with this provider is creating the cluster and the in-cluster resources in the same root module. The provider block is configured before the cluster exists, so plans fail or, worse, the provider silently falls back to a default and targets the wrong cluster. The official provider docs warn against exactly this and recommend that the cluster and the Kubernetes resources are managed with separate apply operations.

Split cluster creation and in-cluster resources into separate states applied in order.

Practical options, from most to least robust:

Two root modules (two states). infra/cluster creates the cluster. infra/cluster-bootstrap reads it with data "aws_eks_cluster" (or the GKE and AKS equivalents) and configures the provider from that. Terragrunt or your CI runs them in order. This is what we default to; see our Terragrunt starter for GCP for the layout.
HCP Terraform Stacks with deferred changes. Stacks can defer the Kubernetes component until the cluster outputs are known, so one deployment creates both. This is a Stacks feature, not plain Terraform CLI.
Targeted first apply. terraform apply -target=module.eks then a full apply. It works for a demo, but -target is meant for exceptions, and you will forget it when rebuilding an environment.

kubernetes_deployment_v1: a complete, annotated example

Use the _v1 resources. In provider v3.0.0 every unversioned resource (kubernetes_deployment, kubernetes_service, kubernetes_namespace and the rest) was marked deprecated in favor of the version-suffixed name. The schema is the same, so examples written for kubernetes_deployment work after renaming. How to migrate existing state is covered in the FAQ below.

This example is what a production-ready Deployment looks like in HCL: namespace, named container port, env from literal values, a Secret and a ConfigMap, resource requests and limits, three probes, and a rolling update strategy.

resource "kubernetes_namespace_v1" "app" {
  metadata {
    name = "checkout"
    labels = {
      "app.kubernetes.io/part-of" = "shop"
    }
  }
}

locals {
  app_labels = {
    "app.kubernetes.io/name"     = "checkout-api"
    "app.kubernetes.io/instance" = "checkout-api"
  }
}

resource "kubernetes_deployment_v1" "api" {
  metadata {
    name      = "checkout-api"
    namespace = kubernetes_namespace_v1.app.metadata[0].name
    labels    = local.app_labels
  }

  # Default is true: Terraform waits until the rollout completes,
  # so a broken image fails the apply instead of passing silently.
  wait_for_rollout = true

  spec {
    replicas = 3

    selector {
      match_labels = local.app_labels   # must match template labels
    }

    strategy {
      type = "RollingUpdate"
      rolling_update {
        max_surge       = "25%"
        max_unavailable = "0"
      }
    }

    template {
      metadata {
        labels = local.app_labels
      }

      spec {
        service_account_name = "checkout-api"

        container {
          name  = "api"                        # required
          image = "ghcr.io/acme/checkout-api:1.14.2"

          # One port block per port. container_port is required.
          port {
            name           = "http"            # referenced by probes and the Service
            container_port = 8080
            protocol       = "TCP"             # default; UDP also valid
          }

          port {
            name           = "metrics"
            container_port = 9090
          }

          # One env block per variable
          env {
            name  = "LOG_LEVEL"
            value = "info"
          }

          env {
            name = "DATABASE_PASSWORD"
            value_from {
              secret_key_ref {
                name = "checkout-db"
                key  = "password"
              }
            }
          }

          # Import every key of a ConfigMap as env vars
          env_from {
            config_map_ref {
              name = "checkout-config"
            }
          }

          resources {
            requests = {
              cpu    = "250m"
              memory = "256Mi"
            }
            limits = {
              memory = "512Mi"
            }
          }

          startup_probe {
            http_get {
              path = "/healthz"
              port = "http"
            }
            period_seconds    = 5
            failure_threshold = 30   # up to 150s to boot
          }

          readiness_probe {
            http_get {
              path = "/ready"
              port = "http"
            }
            period_seconds = 10
          }

          liveness_probe {
            http_get {
              path = "/healthz"
              port = "http"
            }
            period_seconds    = 10
            failure_threshold = 3
          }
        }
      }
    }
  }
}

Notes on the parts people get wrong:

selector.match_labels must be a subset of template.metadata.labels, and it is immutable after creation. Changing it forces a replacement of the Deployment.
Probes on http_get take port as a string, so it accepts either a number (8080) or the port name ("http"). Using names means you change the port in one place.
Setting a memory limit without a CPU limit is deliberate: CPU limits cause throttling, memory limits protect the node. See resource requests and limits for the reasoning.
A liveness probe that checks dependencies (database, downstream APIs) turns a partial outage into a restart storm. Keep it shallow; our guide on liveness probes without restart loops goes deeper.

Container port block syntax reference (and other nested blocks)

The provider maps every Kubernetes list to a repeated block, and every Kubernetes map to an HCL map attribute with =. That one rule explains most syntax errors: port and env are blocks (no equals sign, repeat them), while labels, requests and limits are maps (equals sign, braces).

One mapping rule explains most HCL syntax errors in this provider.

Kubernetes YAML	Terraform HCL	Notes
`ports: - containerPort: 8080`	`port { container_port = 8080 }`	Repeat the `port` block per port. Optional: `name`, `protocol`, `host_port`, `host_ip`
`env: - name: X value: y`	`env { name = "X" value = "y" }`	Repeat per variable. Use `value_from` for Secret and ConfigMap keys
`envFrom: - configMapRef:`	`env_from { config_map_ref { name = ... } }`	Or `secret_ref`; optional `prefix`
`resources: requests: cpu`	`resources { requests = { cpu = "250m" } }`	`requests` and `limits` are maps of strings
`livenessProbe: httpGet:`	`liveness_probe { http_get { path = "/" port = "http" } }`	Same shape for `readiness_probe` and `startup_probe`; also `tcp_socket`, `exec`, `grpc`
`volumeMounts:`	`volume_mount { name = ... mount_path = ... }`	Pair with a `volume` block in the pod spec
`imagePullSecrets:`	`image_pull_secrets { name = ... }`	In the pod spec, not the container

The schema reference for every field is on the kubernetes_deployment_v1 registry page. When you need many ports or env vars from a variable, use a dynamic block:

variable "ports" {
  type = map(number)
  default = {
    http    = 8080
    metrics = 9090
  }
}

# inside container { ... }
dynamic "port" {
  for_each = var.ports
  content {
    name           = port.key
    container_port = port.value
  }
}

Service and Ingress with kubernetes_service_v1 and kubernetes_ingress_v1

Reference the Deployment's labels and port names instead of repeating literals. target_port is a string, so it can be the container port name.

resource "kubernetes_service_v1" "api" {
  metadata {
    name      = "checkout-api"
    namespace = kubernetes_namespace_v1.app.metadata[0].name
  }

  spec {
    selector = local.app_labels
    type     = "ClusterIP"

    port {
      name        = "http"
      port        = 80
      target_port = "http"   # the named container port
    }
  }
}

resource "kubernetes_ingress_v1" "api" {
  metadata {
    name      = "checkout-api"
    namespace = kubernetes_namespace_v1.app.metadata[0].name
    annotations = {
      "cert-manager.io/cluster-issuer" = "letsencrypt"
    }
  }

  spec {
    ingress_class_name = "nginx"

    rule {
      host = "checkout.example.com"
      http {
        path {
          path      = "/"
          path_type = "Prefix"
          backend {
            service {
              name = kubernetes_service_v1.api.metadata[0].name
              port {
                name = "http"
              }
            }
          }
        }
      }
    }

    tls {
      hosts       = ["checkout.example.com"]
      secret_name = "checkout-api-tls"
    }
  }
}

For a LoadBalancer Service or an Ingress that provisions a cloud load balancer, wait_for_load_balancer = true makes Terraform wait for an address, which is useful when a DNS record in the same apply depends on it.

kubernetes_manifest and deploying YAML files with Terraform

kubernetes_manifest takes any Kubernetes object as an HCL map and applies it with server-side apply. Use it for custom resources (cert-manager ClusterIssuer, Karpenter NodePool, Argo CD Application) and for kinds the typed resources do not cover. Two requirements matter in practice:

It needs API access at plan time. The provider asks the cluster for the resource schema during plan, so the cluster must already exist and be reachable. This is another reason to split cluster creation into its own state.
The CRD must exist before the plan. If the CustomResourceDefinition is installed in the same apply (for example by a helm_release of cert-manager), the plan fails with "API did not recognize GroupVersionKind from manifest (CRD may not be installed)". depends_on does not help, because the failure happens at plan time. Install CRDs in an earlier stage.

The failure happens at plan time, so the fix is a separate earlier stage.

A single YAML file

resource "kubernetes_manifest" "issuer" {
  manifest = yamldecode(file("${path.module}/manifests/cluster-issuer.yaml"))
}

Terraform's built-in yamldecode accepts exactly one YAML document; a file that contains several documents returns an error. For multi-document files, use the provider functions.

Multi-document YAML with provider functions (Terraform 1.8+)

locals {
  docs = provider::kubernetes::manifest_decode_multi(
    file("${path.module}/manifests/monitoring.yaml")
  )
}

resource "kubernetes_manifest" "monitoring" {
  # Key by kind/namespace/name so reordering the file does not recreate objects
  for_each = {
    for m in local.docs :
    "${m.kind}/${try(m.metadata.namespace, "_")}/${m.metadata.name}" => m
  }

  manifest = each.value
}

Key for_each by a stable identity, never by list index. With count, inserting one document at the top of the file shifts every index and Terraform plans to destroy and recreate everything below it.

Stable keys keep reordering a YAML file from touching existing objects.

A directory of YAML files

locals {
  files = fileset("${path.module}/manifests", "*.yaml")
  objects = flatten([
    for f in local.files :
    provider::kubernetes::manifest_decode_multi(file("${path.module}/manifests/${f}"))
  ])
}

Then feed local.objects into the same keyed for_each as above.

When to use the kubectl provider instead

The community kubectl provider takes raw YAML (yaml_body) and does not need the CRD schema at plan time, which is why many teams use it for CRD-heavy bootstraps. The original gavinbunney/kubectl has not released since January 2025; the actively maintained fork is alekc/kubectl.

terraform {
  required_providers {
    kubectl = {
      source  = "alekc/kubectl"
      version = "~> 2.4"
    }
  }
}

data "kubectl_file_documents" "crds" {
  content = file("${path.module}/manifests/crds.yaml")
}

resource "kubectl_manifest" "crds" {
  for_each  = data.kubectl_file_documents.crds.manifests
  yaml_body = each.value
}

The trade-off: you lose the field-level diff that kubernetes_manifest shows, and you add a third-party provider to your supply chain.

helm_release: installing charts with the Terraform Helm provider

For third-party software that ships as a Helm chart, helm_release is almost always better than translating the chart into kubernetes_* resources. Note the Helm provider v3 syntax: kubernetes in the provider and set in the release are now attributes with =, not blocks. Most examples online still show the v2 block syntax and fail on v3.

provider "helm" {
  kubernetes = {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
    exec = {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", var.cluster_name]
    }
  }
}

resource "helm_release" "argocd" {
  name             = "argocd"
  namespace        = "argocd"
  create_namespace = true

  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"
  version    = "10.10.5"   # always pin; unpinned installs the latest chart

  values = [file("${path.module}/values/argocd.yaml")]

  set = [
    {
      name  = "configs.params.server\\.insecure"
      value = "true"
    }
  ]
}

Prefer a values file over long set lists: it is reviewable, and it is the same file you would use with the Helm CLI. Installing Argo CD or Flux this way, and then letting it own the application layer, is the cleanest Terraform to GitOps hand-off we know.

Comparison: typed resources vs kubernetes_manifest vs kubectl vs Helm vs GitOps

Approach	Best for	Reads schema from cluster at plan	CRDs	Drift handling	Main drawback
`kubernetes_*_v1`	Core objects: namespaces, RBAC, Services, small Deployments	No	No	Diff on next plan	Verbose HCL; not every kind exists
`kubernetes_manifest`	Custom resources, any kind, YAML you already have	Required	Yes, if the CRD already exists	Diff on next plan (server-side apply)	Fails at plan when the cluster or CRD is missing
`kubectl_manifest` (alekc)	CRD-heavy bootstraps, raw YAML	No	Yes, same apply	Diff on next plan (less detailed)	Community provider
`helm_release`	Third-party charts (ingress, cert-manager, Argo CD)	No	Yes, chart installs them	Release-level diff	Chart internals are opaque in the plan
Argo CD or Flux	Application workloads	Not applicable	Yes, ordered with sync waves or dependencies	Continuous reconciliation	Another system to run; needs a bootstrap

Common errors and how to fix them

Unauthorized or forbidden

"Unauthorized" partway through a long apply: a static token from aws_eks_cluster_auth expired. Switch to an exec block.
Authenticated but every call is "forbidden": the CI role has no Kubernetes RBAC. On EKS, add an access entry for the role Terraform assumes.
Terraform targets the wrong cluster or localhost: the provider got empty values (cluster not created yet, or KUBECONFIG assumed). Split the states and set config_context explicitly.

API did not recognize GroupVersionKind (CRD may not be installed)

A kubernetes_manifest references a custom resource whose CRD is not in the cluster at plan time. Install the CRD (or the chart that ships it) in a previous stage, or use kubectl_manifest for that object.

Perpetual diff on replicas because of an HPA

When a HorizontalPodAutoscaler scales the Deployment, the next plan wants to reset replicas to the value in code. Tell Terraform to stop managing it after creation:

resource "kubernetes_deployment_v1" "api" {
  # ...
  lifecycle {
    ignore_changes = [spec[0].replicas]
  }
}

Set the initial replica count to your HPA minimum so the first apply does not scale down. If the HPA itself flaps, see configuring HPA without scaling flaps.

Drift from controllers, annotations and restarts

Admission webhooks, service meshes and kubectl rollout restart add annotations and labels. Ignore them at the provider level with regex lists, or per resource for the pod template:

provider "kubernetes" {
  # ...
  ignore_annotations = ["^service\\.beta\\.kubernetes\\.io\\/aws-load-balancer.*"]
}

resource "kubernetes_deployment_v1" "api" {
  # ...
  lifecycle {
    ignore_changes = [
      spec[0].template[0].metadata[0].annotations["kubectl.kubernetes.io/restartedAt"],
    ]
  }
}

Provider-level ignore_annotations and ignore_labels only apply to top-level metadata, not to pod templates. For kubernetes_manifest, fields the API server mutates go in computed_fields; metadata.labels and metadata.annotations are included by default.

Provider produced inconsistent result after apply

Usually a mutating webhook changed a field you set in a kubernetes_manifest. Add that field path to computed_fields.

Apply hangs, then times out waiting for rollout

wait_for_rollout defaults to true, so a crash-looping pod or an image that cannot be pulled holds the apply until it times out. Check kubectl describe pod and events. Do not disable the wait to make the error go away; it is telling you the deploy failed.

Field is immutable / forces replacement

Selectors, container names and Service cluster_ip cannot change in place. The plan will show a replacement. For Deployments that serve traffic, create the new object under a new name and switch the Service, rather than letting Terraform destroy first.

Production checklist

Providers pinned (~> 3.4 for kubernetes, ~> 3.3 for helm) and the lock file committed.
Cluster creation and in-cluster resources in separate states, applied in order.
Short-lived credentials through exec; no long-lived tokens in variables or state.
Only _v1 (or newer versioned) resource names; no deprecated unversioned ones in new code.
A clear owner per object: Terraform or GitOps, never both.
ignore_changes on replicas for every Deployment that has an HPA.
CRDs installed in an earlier stage than the custom resources that use them.
Helm charts pinned to an exact version, with values in files.
for_each keyed by kind, namespace and name for anything decoded from YAML.
Secrets sourced from a secret manager (External Secrets Operator, or a cloud secret store) rather than written into Terraform state in plain text.
A plan step in CI on every pull request, so reviewers see the Kubernetes diff before it lands.

FAQ

What is the difference between kubernetes_deployment and kubernetes_deployment_v1?

They have the same schema and both create an apps/v1 Deployment. Since provider v3.0.0, kubernetes_deployment is deprecated and kubernetes_deployment_v1 is the supported name. No removal date is announced, but new code should use the _v1 name.

How do I migrate kubernetes_deployment to kubernetes_deployment_v1 without recreating it?

Rename the resource in code, then remove the old address from state and import the existing object under the new one: terraform state rm kubernetes_deployment.api followed by terraform import kubernetes_deployment_v1.api checkout/checkout-api (namespace/name). Run the plan only after the import and confirm it shows no replacement. Provider v3.3.0 and v3.4.0 also support moved blocks from the old names for kubernetes_namespace and kubernetes_role.

How do I write the container port block in a Terraform kubernetes_deployment?

Inside spec.template.spec.container, add one port block per port with container_port (required) and optionally name and protocol: port { name = "http" container_port = 8080 }. It is a block, so there is no = after port, and you repeat it for multiple ports or generate it with a dynamic "port" block.

Can Terraform apply a Kubernetes YAML manifest from a file?

Yes. For a single document, use kubernetes_manifest with yamldecode(file(...)). For multi-document files, use provider::kubernetes::manifest_decode_multi with a keyed for_each, or the alekc/kubectl provider's kubectl_file_documents data source.

Should I use Terraform or Helm for Kubernetes?

They are not alternatives at the same layer. Helm packages Kubernetes manifests; Terraform orchestrates infrastructure. Use Terraform's helm_release to install platform charts during cluster bootstrap, and use Helm charts through Argo CD or Flux for your own applications.

Is Terraform or Argo CD better for deploying applications to Kubernetes?

For applications that deploy frequently, a GitOps controller like Argo CD is usually better: it reconciles continuously, corrects drift, and gives developers rollbacks without infrastructure credentials. Terraform is better for the cluster and the bootstrap layer, including installing Argo CD itself.

Why does kubernetes_manifest fail when I create the cluster in the same configuration?

It needs to contact the API server during plan to read the resource schema, and the cluster does not exist yet. Create the cluster in a separate state (or use HCP Terraform Stacks deferred changes) and run the Kubernetes configuration afterwards.

Can Terraform install Kubernetes itself?

Terraform creates managed clusters through the cloud providers (EKS, GKE, AKS) and can provision VMs for self-managed clusters, but it does not install Kubernetes on its own. For local testing, kind or minikube is faster than any Terraform setup.

Drawing the line between Terraform and GitOps is a decision you live with for years, and it is easier to get right before the first hundred services. If you want a second opinion on your cluster bootstrap or a review of your Terraform layout, book a free consultation with our engineers.

How to Reduce Cloud Costs Without Slowing Engineering

Michael Zion — Mon, 26 Oct 2026 15:31:40 +0000

This guide was written by Arthur Azrieli, CTO at MeteorOps, and first published on the MeteorOps blog.

The way to reduce cloud costs without slowing engineering is to take the decisions off engineers' plates. Make waste disappear through defaults and automation (lifecycle rules, log retention, after-hours schedules, environment TTLs), buy discounts on the stable baseline you already run, and show each team its own spend where it already works: in pull requests and in a weekly per-team report. Do not add approval gates, ticket queues, or a monthly "cost review" meeting that pulls engineers out of delivery.

In practice that means a specific order. First, get spend attributed to teams and kill obvious waste (idle resources, forgotten non-production environments, unbounded log retention). Second, rightsize and fix the platform defaults that create the waste (Kubernetes requests, node autoscaling, storage classes, NAT gateway traffic). Third, commit to Savings Plans once the baseline is clean. Fourth, put guardrails in code so the bill does not drift back.

This playbook is written for CTOs, VPs of Engineering, and platform or DevOps leads at startups and scale-ups (roughly 10 to 200 engineers, mostly on AWS, often on Kubernetes) whose cloud bill is growing faster than revenue. Every AWS fact below links to AWS documentation; where we give an opinion, it comes from running these changes in production for teams like yours.

Why cost work usually slows engineering (and how to avoid it)

Cost initiatives slow teams down when they turn spending into a permission problem. The common failure patterns are familiar:

Approval gates. A ticket to request a bigger instance or a new environment. Engineers wait, then over-provision when they finally get approval so they never have to ask again.
Finance-shaped dashboards. A chart that says "EC2 is up 18%" gives nobody a decision to make. The engineer still has to dig through clusters, accounts, and services to find the owner.
One-off cleanup sprints. A team spends two weeks deleting things, the bill drops, and six months later it is back because nothing changed in how resources are created.
Risky savings. Aggressive rightsizing or Spot adoption without the reliability groundwork causes an incident, and the organization learns that "cost work breaks production."

The alternative is to treat cost like any other non-functional requirement: encode it in platform defaults, measure it per service, and give owners fast feedback. The FinOps Foundation describes the cycle as three phases, Inform, Optimize, and Operate: see the data, find improvements, then make them part of how you run. The rest of this guide follows that cycle with concrete AWS and Kubernetes steps.

The 30/60/90 day plan to reduce cloud costs

Here is the sequence we recommend. It front-loads changes that need no product decisions and no engineer time from feature teams.

Phase	Goal	Actions	Who does the work
Days 1 to 30	Visibility and zero-risk waste	Activate cost allocation tags, turn on Cost Anomaly Detection and budgets, delete unattached volumes and old snapshots, set log retention, migrate gp2 to gp3, schedule non-prod off after hours	Platform team (1 to 2 people)
Days 31 to 60	Fix the defaults that create waste	Rightsize Kubernetes requests, enable node consolidation, add Spot for tolerant workloads, add S3 gateway endpoints, add lifecycle rules, cut log and metric volume	Platform team, with service owners reviewing requests
Days 61 to 90	Lock in rates and prevent drift	Buy Savings Plans on the cleaned baseline, add cost estimates to pull requests, enforce tags and TTLs with policy as code, start per-team showback	Platform team plus engineering leadership

The order matters. If you buy commitments first, you lock in a discount on waste you were about to delete. If you add guardrails before visibility, nobody knows why the guardrail exists.

Commit to discounts only after the waste is gone, then lock in the result with guardrails.

Days 1 to 30: visibility and quick wins

Make spend attributable to an owner

Unowned spend becomes permanent spend. Before anything else, every dollar should map to a team and a service. The two strongest tools are account structure and tags.

Separate accounts per environment (at minimum production and non-production) through AWS Organizations. Account boundaries are the most reliable cost split because they need no tagging discipline.
A small, mandatory tag set. Keep it to three or four keys, for example team, service, env, and optionally ttl for ephemeral resources. More keys means lower compliance.
Activate the tags for billing. User-defined tags do not appear in billing data until you activate them as cost allocation tags. AWS notes it can take up to 24 hours for tag keys to appear and up to another 24 hours to activate, so do this on day one.
Kubernetes needs its own allocation. On EKS, the bill shows EC2 nodes, not pods. AWS offers split cost allocation data for EKS in its billing data, and Kubecost or OpenCost (a CNCF incubating project) break costs down by namespace, workload, and label.

Tag in Terraform with provider-level default_tags so engineers do not have to remember:

provider "aws" {
  region = "us-east-1"

  default_tags {
    tags = {
      team    = "payments"
      service = "checkout-api"
      env     = "staging"
    }
  }
}

Turn on anomaly detection and budgets

The cheapest overspend is the one you catch on day two instead of at the end of the month. AWS Cost Anomaly Detection uses machine learning models on your net unblended cost, runs roughly three times a day, and can alert by email or SNS (which you can route to Slack). Expect a delay: it relies on Cost Explorer data that can lag up to 24 hours. Set it up in Terraform so it lives next to the rest of your infrastructure:

resource "aws_ce_anomaly_monitor" "services" {
  name              = "per-service-monitor"
  monitor_type      = "DIMENSIONAL"
  monitor_dimension = "SERVICE"
}

resource "aws_ce_anomaly_subscription" "daily" {
  name      = "cost-anomalies-daily"
  frequency = "DAILY"

  monitor_arn_list = [aws_ce_anomaly_monitor.services.arn]

  subscriber {
    type    = "EMAIL"
    address = "platform-team@example.com"
  }

  threshold_expression {
    dimension {
      key           = "ANOMALY_TOTAL_IMPACT_ABSOLUTE"
      match_options = ["GREATER_THAN_OR_EQUAL"]
      values        = ["100"]
    }
  }
}

Add AWS Budgets on each account with forecast-based alerts. Budgets can also run budget actions such as applying an IAM policy or SCP, or stopping specific EC2 and RDS instances. Use actions for sandbox accounts only; on production, alerts should go to a human who owns the service.

Delete idle resources

Idle resources are the safest savings because nobody uses them. AWS Cost Optimization Hub consolidates idle-resource, rightsizing, Savings Plans, and Reserved Instance recommendations across accounts and Regions, so start there. Then check the usual suspects directly:

# Unattached EBS volumes (paying for storage nobody mounts)
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query 'Volumes[].{ID:VolumeId,SizeGiB:Size,Created:CreateTime}' \
  --output table

# Elastic IPs not associated with anything
aws ec2 describe-addresses \
  --query 'Addresses[?AssociationId==`null`].PublicIp' \
  --output table

# CloudWatch log groups with no retention (kept forever by default)
aws logs describe-log-groups \
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' \
  --output table

Also look for old EBS snapshots and AMIs from past migrations, load balancers with no healthy targets, idle RDS instances in non-production, and NAT gateways in VPCs nobody uses. Snapshot before you delete anything you are not sure about; snapshots are cheap insurance compared with a surprise outage.

Set log retention and switch gp2 to gp3

Two changes that are almost always safe:

Log retention. CloudWatch log groups keep data indefinitely unless you set a retention period. Pick a default (for example 30 days for application logs, longer only where compliance requires it) and apply it with aws logs put-retention-policy --log-group-name NAME --retention-in-days 30 or in Terraform.
gp2 to gp3. AWS states gp3 volumes cost up to 20% less per GB than gp2 and deliver a baseline of 3,000 IOPS and 125 MB/s at any size. The migration is an online volume modification: aws ec2 modify-volume --volume-id vol-0123456789abcdef0 --volume-type gp3. Check volumes that need more than the gp3 baseline before converting them, and update your Kubernetes StorageClass so new volumes default to gp3.

Turn non-production off after hours

Development and staging environments that run 24/7 but are used 50 hours a week are pure waste. AWS's own Instance Scheduler on AWS starts and stops tagged EC2 and RDS instances on a schedule, and AWS cites up to 70% savings for instances only needed during business hours (168 weekly hours reduced to 50). On Kubernetes, scale workloads to zero with a KEDA cron trigger and let the node autoscaler remove the empty nodes:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: checkout-api-business-hours
  namespace: staging
spec:
  scaleTargetRef:
    name: checkout-api
  minReplicaCount: 0
  triggers:
    - type: cron
      metadata:
        timezone: Europe/London
        start: "0 7 * * 1-5"
        end: "0 20 * * 1-5"
        desiredReplicas: "2"

Make the schedule opt-out, not opt-in, and give engineers a one-command way to wake an environment (a Slack command or a CI job). The guardrail that does not slow anyone down is the one with an easy override.

Days 31 to 60: rightsizing and platform defaults

Rightsize Kubernetes requests and enable bin-packing

On Kubernetes, you pay for nodes, and nodes are sized by the sum of pod requests, not actual usage. If every service requests 1 CPU and uses 0.1, your cluster is roughly 90% air no matter how good your autoscaler is. This is usually the single biggest lever for Kubernetes cost optimization. (If the difference between requests and limits is fuzzy, start with our guide to Kubernetes resource requests and limits.)

You pay for what pods request, so savings appear only when requests shrink and nodes are removed.

Measure. Compare requested CPU and memory with actual p95 usage per workload, using Prometheus, Kubecost, or OpenCost.
Get recommendations without risk. Run the Vertical Pod Autoscaler in recommendation-only mode so it suggests requests but never restarts pods:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-api
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  updatePolicy:
    updateMode: "Off"

Apply through pull requests to each service's manifests or Helm values, reviewed by the owner. Lower CPU requests are low risk (CPU is compressible); be more careful with memory, because a memory limit that is too low means OOM kills.
Let the cluster shrink. Rightsized requests only save money if nodes are removed. With Karpenter, consolidation does this: WhenEmptyOrUnderutilized (the default) replaces or removes nodes whose pods fit elsewhere, and disruption budgets cap how many nodes move at once.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
    budgets:
      - nodes: "10%"

Consolidation moves pods, so it is only safe when your workloads tolerate it: PodDisruptionBudgets, multiple replicas, and correct readiness probes. Our guides on draining nodes without evicting critical workloads and Kubernetes autoscaling without cost surprises cover that groundwork.

Use Spot where interruptions are cheap

AWS advertises Spot Instances at up to 90% off On-Demand prices. The trade-off is interruption: EC2 issues an interruption notice two minutes before it stops or terminates a Spot Instance, on a best-effort basis. Good candidates are CI runners, batch and queue workers, stateless services with several replicas, and most non-production environments. Poor candidates are single-replica stateful services and anything that takes longer than two minutes to shut down cleanly. Diversify across many instance types and Availability Zones so a single capacity pool going away does not take all your nodes with it, and keep an On-Demand floor for critical services.

Rightsize instances and databases, and try Graviton

Outside Kubernetes, use AWS Compute Optimizer (surfaced in Cost Optimization Hub) to find EC2, RDS, Lambda, and Fargate resources that are larger than their measured usage. Then consider Graviton: AWS states Graviton-based instances cost up to 20% less than comparable x86-based EC2 instances. For interpreted languages (Node.js, Python, Ruby) and the JVM, the move is often just a multi-architecture container build plus a node pool change. Native dependencies and third-party agents are what usually block it, so test one service end to end before planning the fleet.

Cut data transfer and NAT gateway costs

Data transfer is the line item teams understand least. The most common surprise is the NAT gateway. AWS charges an hourly charge per NAT gateway plus a per-GB data processing charge for every gigabyte processed through it, regardless of source or destination (AWS's example rate in US East (Ohio) is $0.045 per hour and $0.045 per GB). So a private-subnet service pulling container images or reading S3 through a NAT gateway pays a processing fee on every byte.

Add gateway VPC endpoints for S3 and DynamoDB. AWS states there are no data processing or hourly charges for gateway-type VPC endpoints, and S3 traffic then bypasses the NAT gateway. This is usually the highest return-on-effort change in the whole network.
Use a pull-through cache or ECR in the same Region for container images rather than pulling from public registries through NAT on every node start.
Watch cross-AZ traffic. Traffic between Availability Zones carries data transfer charges. Chatty services and replicas spread across zones can generate a surprising bill. Check the DataTransfer-Regional-Bytes usage types in Cost Explorer before optimizing, so you fix what actually costs money.

A gateway endpoint takes S3 traffic off the NAT gateway and its per-GB processing fee.

Storage tiers and lifecycle policies

S3 buckets that only grow are a slow leak. Apply lifecycle rules to anything with a predictable access pattern (logs, backups, build artifacts) and use S3 Intelligent-Tiering where access is unpredictable. Intelligent-Tiering moves objects to an Infrequent Access tier after 30 days without access and to Archive Instant Access after 90, for a small per-object monitoring charge; objects under 128 KB are not monitored or tiered, so it does little for buckets full of tiny files.

resource "aws_s3_bucket_lifecycle_configuration" "logs" {
  bucket = aws_s3_bucket.logs.id

  rule {
    id     = "logs-tiering-and-expiry"
    status = "Enabled"

    filter {
      prefix = "logs/"
    }

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    expiration {
      days = 365
    }

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

Reduce observability and log costs without losing visibility

Observability is often the second or third largest line on a startup's bill once you add the vendor invoice (Datadog, New Relic, Grafana Cloud, or CloudWatch itself). The goal is not less visibility; it is paying only for data someone will actually look at. Work through these in order:

Find the top emitters. In most systems a handful of services and log lines produce most of the volume: health checks, debug logs left on, per-request access logs, and retry loops. Rank by volume per service and per log pattern before changing anything.
Drop or demote noise at the source. Set production log levels to INFO or WARN, stop logging successful health checks, and sample high-volume success logs (for example keep 1 in 100 successful requests, all errors).
Filter before indexing. Most vendors let you ingest without indexing. In Datadog, exclusion filters discard logs from indexes while still letting them feed log-based metrics and archives. Convert "count of X" log searches into metrics, which are far cheaper to keep.
Tier by access. CloudWatch's Infrequent Access log class has a lower ingestion price per GB but drops features such as Live Tail, metric filters, and subscription filters, and a log group's class cannot be changed after creation. Use it for audit and forensic logs you query rarely, and archive to S3 for long-term retention.
Sample traces intelligently. Tail sampling keeps every error and slow trace while sampling the boring ones. With the OpenTelemetry Collector:

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: sample-the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

Control metric cardinality. Custom metrics tagged with user IDs, request IDs, or pod names explode series counts. Drop high-cardinality labels in the collector or agent before they reach the vendor.

To prove you did not lose visibility, keep a short list of the queries and alerts your on-call actually used in the last quarter and confirm each one still works after every change.

Days 61 to 90: Savings Plans vs Reserved Instances vs Spot

Commitments are the largest discount available and the easiest to get wrong. Buy them last, on the baseline that remains after cleanup and rightsizing. AWS defines four Savings Plans types, all with 1-year or 3-year terms and All Upfront, Partial Upfront, or No Upfront payment:

Option	Maximum discount (per AWS)	Flexibility	Best for
Compute Savings Plans	Up to 66% off On-Demand	Any EC2 family, size, Region, OS; also Fargate and Lambda	Most startups; teams still changing instance types, moving to Graviton, or shifting to Fargate
EC2 Instance Savings Plans	Up to 72% off On-Demand	One instance family in one Region; any size within it	Very stable fleets you will not migrate during the term
Database Savings Plans	Up to 35%	Aurora, RDS, DynamoDB, ElastiCache, DocumentDB, OpenSearch and others, across engines and Regions	Steady database spend, especially if you may change engines or instance generations
Reserved Instances	Varies by service and term	Tied to specific attributes	Services not covered by a Savings Plan you need, or capacity reservations
Spot	Up to 90% off On-Demand	Can be interrupted with a two-minute notice	Interruption-tolerant workloads; not a commitment at all

Our guidance for a growing startup:

Prefer Compute Savings Plans. You give up a few percentage points versus EC2 Instance plans and keep the freedom to change architecture. For a team that is still evolving, that freedom is worth more.
Cover the floor, not the average. Commit to the lowest sustained hourly compute spend over recent months, not the average. Under-committing costs you a small missed discount; over-committing costs you money for nothing, and AWS states the terms cannot be changed after purchase.
Ladder purchases. Buy smaller plans every quarter rather than one large plan, so expiries are staggered and you can adjust as you grow.
Know what is not covered. On EKS, Savings Plans cover the underlying EC2 instances but not the EKS cluster charge itself.

Commit to the lowest sustained spend so every committed dollar is actually used.

Cost Explorer generates Savings Plans purchase recommendations from your history; treat them as a starting point and adjust for anything you know is about to change (a migration, a big customer, a planned Graviton move).

Guardrails that do not slow engineers down

A good guardrail is automatic, explains itself, and has an obvious escape hatch. A bad one is a ticket. Here are the guardrails we install, roughly in order of value:

Guardrail	How to implement	Why it does not slow anyone
Default tags	Terraform provider `default_tags`, Kubernetes labels set in shared Helm charts	Engineers get tagging for free
TTLs on ephemeral environments	A `ttl` tag or label plus a scheduled cleanup job; preview environments destroyed on PR merge or close	Nobody has to remember to clean up
Auto-stop non-prod after hours	Instance Scheduler on AWS, KEDA cron, or scaling node groups to zero	Opt-out tag and a wake-up command
Policy as code in audit mode first	Kyverno or OPA Gatekeeper in Kubernetes, tag policies and SCPs in AWS Organizations	Warns before it blocks; blocks only what is clearly wrong
Cost estimate in every infrastructure PR	Infracost or similar in CI	Feedback arrives in the review the engineer is already doing
Anomaly alerts routed to owners	Cost Anomaly Detection per account or cost category, sent to the owning team's channel	Only fires when something is actually unusual
Sandbox budget actions	AWS Budgets actions applying an SCP when a sandbox account exceeds its budget	Applies only to experiments, never production

Start policies in audit mode, look at what they would have blocked, fix the false positives, then enforce. A Kyverno policy that requires a team label on Deployments, in audit mode, looks like this:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-team-label
spec:
  background: true
  rules:
    - name: check-team-label
      match:
        any:
          - resources:
              kinds:
                - Deployment
      validate:
        failureAction: Audit
        message: "Every Deployment needs a team label for cost allocation."
        pattern:
          metadata:
            labels:
              team: "?*"

The biggest guardrail of all is a self-service path that is cheap by default. If spinning up an environment through your platform gives you right-sized, scheduled, tagged, TTL-bound resources automatically, engineers will use it because it is faster, and cost control comes along for free. That is the idea behind a one-click environment.

How to give engineers cost visibility in their workflow

Engineers care about cost when they can see the effect of their own changes, quickly, in a place they already look. Three layers work well together:

Before deploy: cost in the pull request. Infracost estimates costs from Terraform, CloudFormation, and AWS CDK and can comment on each pull request with the cost difference, policy violations, and tagging issues introduced by the change, with integrations for GitHub Actions, GitLab CI, and Azure DevOps. A reviewer seeing "+$1,400/month" next to a database instance change is the cheapest cost control you will ever add.
After deploy: per-team and per-service dashboards. Built from your cost allocation tags (Cost Explorer, Cost Categories, or a CUR-based dashboard) and, for Kubernetes, from Kubecost or OpenCost by namespace. The test of a good dashboard: can a team lead answer "what did my services cost last week and why did it change" in under a minute?
Weekly: a short showback message. A Slack post per team with last week's spend, the change versus the prior week, and the top three movers. No action required unless something looks wrong. Showback (reporting cost to owners) builds the habit without the overhead of chargeback (billing it to their budget).

Engineers see cost at review time, after release, and in a weekly summary, without leaving their workflow.

Where possible, express cost as a unit metric the team cares about: cost per 1,000 requests, per active customer, per build, or per GB processed. "Our bill went up 20%" is alarming; "cost per customer went down 8% while customers grew 30%" is a success, and a unit metric tells you which one you are looking at.

How to get engineering buy-in for cost reduction

Most engineers do not resist saving money. They resist being blamed, being slowed down, and being handed toil. Get buy-in by addressing those directly:

Frame it as engineering quality, not frugality. Over-provisioned requests, unbounded logs, and forgotten environments are bugs in how the system is run. Fixing them is the same craft as fixing a slow query.
The platform team does the first round. Quick wins in the first 30 days should cost feature teams zero hours. Showing a real reduction before asking anyone for anything earns the right to ask later.
Give ownership, not orders. Each team sees its own spend and decides how to act. Leadership sets a target (for example, a unit cost target or a cap on cloud spend as a percentage of revenue), not a list of instance types.
Never trade reliability for savings silently. Every change that touches production capacity (Spot, consolidation, lower requests) goes through the normal review and rollout process. One cost-driven outage will kill the program.
Celebrate the wins publicly. Share savings in engineering all-hands with the name of the team that found them. It costs nothing and turns cost into something people want to be associated with.

If you are an engineering leader structuring this for the first time, our CTO DevOps handbook covers how to set ownership boundaries between platform and product teams.

Cloud cost optimization tools by category

You do not need a paid platform to start. AWS native tools cover the first 90 days for most startups; third-party tools earn their place when you need Kubernetes allocation, multi-cloud views, or engineer-facing workflows. Capabilities below are as described in each product's own documentation; check current pricing directly with the vendor.

Category	AWS native	Third-party or open source	Use it for
Spend analysis and allocation	Cost Explorer, cost allocation tags, Cost Categories, Cost and Usage Reports	CloudZero, Vantage, Kubecost, OpenCost	Who spends what, and unit costs
Recommendations	Cost Optimization Hub, Compute Optimizer, Trusted Advisor	Most commercial FinOps platforms	Idle resources, rightsizing, commitments
Alerts and budgets	Cost Anomaly Detection, AWS Budgets and budget actions	Commercial platforms with Slack routing	Catching spikes in days, not at month end
Kubernetes cost	Split cost allocation data for EKS	Kubecost, OpenCost; Karpenter for consolidation	Namespace and workload cost, bin-packing
Cost in the developer workflow	None built in	Infracost	Cost diffs on pull requests
Scheduling non-prod	Instance Scheduler on AWS	KEDA cron scaler, kube-downscaler	Turning environments off after hours
Policy and governance	Tag policies, SCPs in AWS Organizations, AWS Config rules	Kyverno, OPA Gatekeeper, Cloud Custodian	Enforcing tags, TTLs, and allowed resource types
Observability cost	CloudWatch log classes, retention settings	OpenTelemetry Collector (sampling, filtering), vendor exclusion filters	Reducing log, metric, and trace volume

When you evaluate a paid cost platform for a team that is mostly engineers, ask three questions. Does it attribute Kubernetes spend to teams without a tagging project? Does it deliver insights where engineers already work (pull requests, Slack, the IDE) rather than in another dashboard? And does it need agents or ongoing maintenance from your platform team? A tool that adds operational overhead is just another cost line.

Common cloud cost reduction mistakes

Buying Savings Plans before cleanup. You lock in a discount on waste. Clean up and rightsize first, then commit to the floor.
Rightsizing nodes instead of requests. On Kubernetes, smaller nodes do nothing if pod requests stay inflated. Fix requests, then let consolidation shrink the fleet.
Setting memory limits too tight. CPU over-request is cheap to fix; memory under-provisioning causes OOM kills and restarts. Lower memory requests gradually and watch.
Moving everything to Spot at once. Without PodDisruptionBudgets, multiple replicas, and graceful shutdown, Spot interruptions become incidents.
Ignoring the NAT gateway. Teams optimize compute for weeks while S3 and image traffic quietly pay per-GB processing fees. Add gateway endpoints early.
Cutting logs blindly. Dropping logs without checking which queries and alerts depend on them saves money until the next incident. Inventory what on-call uses first.
Too many mandatory tags. Ten required tag keys means nobody complies. Three or four, applied by default, beats ten applied by hand.
Making it a project instead of a habit. A one-time cleanup decays. Anomaly alerts, PR cost diffs, and weekly showback keep it from coming back.

Cloud cost reduction checklist

Visibility

Production and non-production in separate AWS accounts
3 to 4 mandatory tags applied through default_tags and shared Helm charts
Cost allocation tags activated in the Billing console
Kubernetes cost allocation by namespace (split cost allocation data, Kubecost, or OpenCost)
Cost Anomaly Detection and budgets on every account, routed to owners

Quick wins

Unattached EBS volumes, unused Elastic IPs, and old snapshots removed
Retention set on every CloudWatch log group
gp2 volumes migrated to gp3; StorageClass defaults to gp3
Non-production scheduled off outside working hours
S3 and DynamoDB gateway endpoints in every VPC with a NAT gateway

Structural changes

Pod requests rightsized from measured usage; VPA in recommendation mode
Node consolidation enabled with disruption budgets and PDBs in place
Spot used for CI, batch, and stateless multi-replica services
S3 lifecycle rules on log, backup, and artifact buckets
Log levels, sampling, and exclusion filters reviewed for the top emitters
Compute Savings Plans covering the sustained floor, purchased in quarterly ladders

Guardrails

Cost diff on every infrastructure pull request
TTLs on preview and ephemeral environments
Policy as code for tags, running in audit mode before enforce
Weekly per-team showback with unit cost trends

FAQ: reducing cloud spend without slowing engineering

How do engineering teams reduce cloud spend without slowing delivery?

They move cost control into defaults and automation instead of approvals. The platform team removes waste (idle resources, log retention, after-hours schedules), rightsizes Kubernetes requests, and buys Savings Plans on the cleaned baseline, while feature teams only see a cost diff in their pull requests and a weekly report on their own services. Nobody waits on a ticket to ship.

How do engineering teams reduce log volume and cost without losing visibility?

Start by ranking log volume by service and pattern, because a few sources usually dominate. Drop health-check and debug noise, sample high-volume success logs while keeping all errors, convert count-style log searches into metrics, and use tail sampling for traces. Before and after each change, confirm the queries and alerts on-call actually used still work.

How do I create FinOps guardrails that don't slow down engineers?

Make every guardrail automatic and give it an escape hatch. Examples: default tags in Terraform, TTLs on preview environments, non-production that auto-stops after hours with a one-command wake-up, policies that run in audit mode before they enforce, and anomaly alerts that only fire on unusual spend. Avoid manual approval steps for routine resources.

How do I get developers to care about cloud costs without making it a burden?

Show them the cost of their own changes, in tools they already use, and do not blame them for the past. A cost estimate in the pull request and a weekly per-team spend summary create awareness with no extra work. Express cost as a unit metric (per request, per customer) so teams can see efficiency improve even while the total bill grows with the business.

What is the best way for engineering teams to take ownership of cloud cost reduction?

Give each team its own spend data, attributed through accounts and tags, and a target such as a unit cost or a share of revenue, then let them decide how to hit it. The platform team owns shared levers (commitments, cluster consolidation, network endpoints); service teams own their requests, environments, and logging. Showback before chargeback keeps it collaborative.

Should a startup use Savings Plans or Reserved Instances?

For most startups, Compute Savings Plans are the better default because they apply across instance families, Regions, Fargate, and Lambda, so you can keep changing your architecture. EC2 Instance Savings Plans offer a higher maximum discount but lock you to one instance family in one Region. Buy only after cleanup, cover your lowest sustained usage, and stagger purchases.

What should I look for in a cloud cost platform if my team is mostly engineers?

Look for Kubernetes-aware allocation, feedback inside the engineering workflow (pull requests, Slack), and low operational overhead. Check whether it needs agents or ongoing tuning from your platform team. Many teams get most of the value from AWS native tools plus Infracost and OpenCost before a paid platform is justified.

How much can you realistically save?

It depends entirely on your starting point, so be wary of anyone quoting a percentage before looking at your bill. The verifiable ceilings come from AWS itself: up to 66% off On-Demand with Compute Savings Plans, up to 90% with Spot, and up to 70% on instances that only need to run during business hours. The biggest real-world savings usually come from waste you can see in the first week of analysis.

Where to start this week

If you do only three things this week: activate cost allocation tags, turn on Cost Anomaly Detection for every account, and run the three idle-resource queries above. You will have attribution starting to flow, an early warning system, and probably a first round of savings, without taking a single hour from your feature teams. If you want senior engineers to run this playbook with your team, MeteorOps offers a free consultation to review your setup and find the biggest levers.

Kubernetes Requests and Limits: Set CPU and Memory Right

Michael Zion — Fri, 23 Oct 2026 17:40:03 +0000

This guide was written by Arthur Azrieli, CTO at MeteorOps, and first published on the MeteorOps blog.

Set a CPU request and a memory request on every container, sized from real usage: the 95th percentile of CPU and the peak memory working set over one to two weeks, plus headroom. Set the memory limit equal to the memory request. For latency-sensitive services, leave the CPU limit off (or set it well above the request) unless you have a specific reason to cap CPU. That combination keeps the scheduler honest, avoids Completely Fair Scheduler (CFS) throttling, and makes out-of-memory (OOM) kills predictable instead of random.

The rest of this guide explains why each of those rules exists, shows how to measure the numbers with PromQL and kubectl, covers the exceptions (multi-tenant clusters, batch jobs, quotas, Guaranteed pods), and ends with runtime settings for the JVM, Go and Node.js, a troubleshooting table, and a checklist. If you only need the definitions, the Kubernetes resource requests and limits glossary entry covers them in two minutes. This page is about choosing the numbers and changing them safely in production.

What requests and limits actually do (four different mechanisms)

Most confusion about requests and limits comes from treating them as one feature. In reality four separate components read those two fields, and each one does something different with them.

Field	Who reads it	What happens
CPU request	Scheduler, then the kernel	The scheduler only places the pod on a node whose unreserved CPU covers the sum of requests. At runtime the request becomes the container's CPU weight, so under contention CPU is shared in proportion to requests.
CPU limit	Kernel CFS bandwidth control	The container gets a quota of CPU time per 100 ms period (the kubelet default `cpuCFSQuotaPeriod`). Once the quota is spent, every thread in the container waits for the next period. That wait is throttling, and it happens even if the node is idle.
Memory request	Scheduler and the kubelet eviction manager	Used for placement, and later to decide who gets evicted first when the node runs low on memory: pods using more than their request go first.
Memory limit	Kernel cgroup memory controller	A hard ceiling. When the container exceeds it and the kernel cannot reclaim memory, the kernel OOM killer terminates it and the container shows `OOMKilled` with exit code 137.

Each field is read by a different component, at placement time or at runtime.

Two consequences matter for sizing:

The scheduler never looks at actual usage. A node can be "full" at 15% real CPU because requests are inflated, or overcommitted at 100% real usage because requests are tiny. Requests are a reservation, and the cluster is only as efficient as those reservations are accurate.
CPU is compressible, memory is not. Too little CPU makes a process slow. Too little memory kills it. That is why the recommended policies for CPU and memory are different, and why copying the same pattern for both is the most common mistake.

One detail that surprises people: if you set a limit and no request, Kubernetes copies the limit into the request (unless a LimitRange default applies). A manifest with only limits.cpu: "2" reserves two full CPUs on the node.

QoS classes and eviction order: Guaranteed, Burstable, BestEffort

From the requests and limits you set, Kubernetes derives a Quality of Service (QoS) class for each pod. You never set the class directly, and in-place resize cannot change it later, so pick it deliberately.

QoS class	How you get it	OOM score adjustment	When to use it
Guaranteed	Every container has CPU and memory requests equal to limits	-997 (killed last)	Databases, stateful systems, anything that needs exclusive cores with the static CPU manager policy
Burstable	At least one container has a request or limit, but it does not meet the Guaranteed rule	Between 2 and 999, lower for pods with larger memory requests	Most stateless services, especially with memory request = limit and no CPU limit
BestEffort	No requests or limits on any container	1000 (killed first)	Almost never in production. Throwaway jobs on dedicated nodes at most

When a node runs short of memory, the kubelet evicts pods in this order: first BestEffort and Burstable pods whose usage exceeds their requests (ranked by priority, then by how far over their request they are), and last Guaranteed pods and Burstable pods that stay under their requests. Read that again, because it is the strongest argument for accurate memory requests: a pod that stays within its memory request is in the last group to be evicted, regardless of QoS class.

Eviction order under node memory pressure. Staying within the memory request puts a pod in the last group.

The default Linux hard eviction threshold is memory.available<100Mi. If the kernel OOM killer acts before the kubelet notices pressure, it picks the container with the highest OOM score, which is why BestEffort pods and Burstable pods with small requests die first.

Should you set CPU limits in Kubernetes?

This is the most searched and most argued part of the topic, so here is a clear position: set CPU requests everywhere; drop CPU limits for latency-sensitive services by default; keep them where you have a concrete reason.

Why CPU limits cause throttling at low average usage

The CFS quota is measured in CPU time per 100 ms period, summed across all threads in the container. A container with limits.cpu: "1" gets 100 ms of CPU time per 100 ms period. If a request arrives and the application runs 8 threads in parallel (a garbage collector, a thread pool, a Node.js libuv pool), those threads can burn the whole 100 ms of quota in about 12.5 ms of wall-clock time. Every thread then sleeps for the remaining 87.5 ms of the period. Your dashboard shows 30% average CPU, and your p99 latency shows an extra 80 ms that nobody can explain.

With limits.cpu: "1", eight busy threads spend the whole period's quota in 12.5 ms and then wait for the next period.

That is why "my pod is throttled but CPU usage is low" is not a contradiction. Averages over 30 seconds or a minute hide bursts that happen inside a 100 ms window.

Has the kernel fixed this?

Partly. A real kernel bug made throttling much worse on multi-core machines: per-CPU slices of quota expired unused, so highly threaded workloads were throttled while using a fraction of their limit. The fix ("sched/fair: Fix low cpu usage with high throttling by removing expiration of cpu-local slices") landed in 2019 around Linux 5.4 and was backported to stable kernels, so any modern node image has it. What remains is not a bug: a hard quota per period is how CPU limits are supposed to work, and cgroup v2 (where the limit lives in cpu.max) enforces the same quota and period model. Kubernetes does not expose the cgroup v2 burst setting, so you cannot ask the kernel to absorb short spikes above the limit.

What happens without a CPU limit

Without a limit, a container can use idle CPU on the node. When the node is busy, CPU is divided by weight, and the weight comes from the CPU request. A pod with a 500m request competing against a pod with a 1000m request gets one third of the contended CPU. So removing limits is safe only when every pod on the node has an honest CPU request. A pod with no request at all gets the minimum weight and starves first, which is the right outcome.

When to keep CPU limits

Situation	Recommendation
Latency-sensitive API or web service on a cluster where all pods set requests	No CPU limit, or a limit at 2x to 4x the request as a runaway guard
Multi-tenant cluster or untrusted workloads	Keep limits, so one tenant cannot consume idle CPU others expect to burst into
Batch jobs, CI runners, video or ML preprocessing that will use every core it sees	Keep limits, or isolate on dedicated node pools
Databases or workloads that need dedicated cores with the static CPU manager	Guaranteed QoS with integer CPU requests equal to limits
Load tests and capacity benchmarks	Limits equal to requests, so results reflect what you will reserve in production
Namespace has a ResourceQuota on `limits.cpu`	Every pod must set a CPU limit (or receive one from a LimitRange default). Prefer quotas on `requests.cpu` instead

A practical middle ground that works well for teams that are nervous about removing limits entirely: keep a generous CPU limit (for example 4x the request) and alert on throttling. You keep protection against a runaway loop and lose almost all of the latency penalty.

Memory requests and limits: why request should equal limit

For memory, the common advice is to set the memory request equal to the memory limit. It is good advice, and the reason is the eviction and OOM mechanics above.

Memory cannot be taken back. If a pod is allowed to grow from a 512Mi request to a 2Gi limit, the scheduler placed it as if it needed 512Mi. When several neighbors do the same, the node runs out of real memory, and the kubelet or the kernel kills something. Often not the pod that grew.
Request = limit makes failures local. The container can only hurt itself. If it needs more, it is OOMKilled with a clear signal instead of triggering node pressure that evicts unrelated pods.
It keeps you in the "evicted last" group. A pod using at most its request is never in the first eviction group.

A gap between memory request and limit is overcommit. Setting them equal keeps the failure inside the container that grew.

The exception is a workload with a rare, short, known memory spike (a nightly report, a large startup cache) on nodes with spare memory. A gap between request and limit can be acceptable there, but treat it as overcommit you have chosen, and watch node memory pressure.

On cgroup v2 nodes, the kubelet by default sets memory.oom.group on container cgroups (the singleProcessOOMKill kubelet setting defaults to false), so when the OOM killer fires, all processes in the container are killed together. On cgroup v1, only the single largest process was killed, which could leave a container half alive with its main worker gone. If you run multi-process containers (for example a process manager with workers), expect the whole container to restart on cgroup v2.

Kubernetes also has an opt-in MemoryQoS feature gate that uses cgroup v2 memory.high to throttle memory allocation before the limit for Burstable pods. It is off by default; do not size around it unless your platform team has enabled it.

How to size requests and limits from real metrics, step by step

Guessing produces either waste or incidents. Here is the method we use when tuning production clusters. It assumes Prometheus with cAdvisor metrics (scraped through the kubelet) and kube-state-metrics, which most clusters already have through kube-prometheus-stack or a managed equivalent.

Step 1: Collect at least 7 to 14 days of data

The window must include peak traffic days, deploys, cron jobs, cache warmups and any weekly batch. A day of data underestimates almost everything. For brand-new services, start with a conservative guess, run for a week, then apply this method.

Step 2: Find CPU usage at p95 and p99

# p95 of 5-minute CPU usage per container over 14 days (in cores)
quantile_over_time(0.95,
  sum by (namespace, pod, container) (
    rate(container_cpu_usage_seconds_total{namespace="production", container!="", container!="POD"}[5m])
  )[14d:5m]
)

Change 0.95 to 0.99 to see the tail. To get one number per workload across all of its replicas, wrap the expression in max by (namespace, container) (...).

Step 3: Measure throttling

# Fraction of CFS periods in which the container was throttled
sum by (namespace, pod, container) (
  increase(container_cpu_cfs_throttled_periods_total{namespace="production", container!=""}[5m])
)
/
sum by (namespace, pod, container) (
  increase(container_cpu_cfs_periods_total{namespace="production", container!=""}[5m])
)

These counters only exist for containers with a CPU limit. As a rule of thumb, a latency-sensitive service throttled in more than a few percent of periods during peak is worth fixing, and anything above 20% is almost certainly hurting tail latency. Correlate with your p99 latency before deciding. container_cpu_cfs_throttled_seconds_total gives total time spent throttled if you prefer seconds to periods.

Step 4: Find peak memory working set

# Peak memory working set per container over 14 days (bytes)
max by (namespace, container) (
  max_over_time(container_memory_working_set_bytes{namespace="production", container!="", container!="POD"}[14d])
)

Use the working set, not RSS or total usage: it is the number the kubelet uses for eviction decisions and what kubectl top shows. Use the max, not a percentile, because memory sizing must survive the worst case.

Step 5: Compare against what you requested

# Requested CPU per container (cores), from kube-state-metrics
sum by (namespace, pod, container) (
  kube_pod_container_resource_requests{namespace="production", resource="cpu"}
)

# Containers whose last termination was an OOM kill
kube_pod_container_status_last_terminated_reason{namespace="production", reason="OOMKilled"} == 1

Dividing the p95 usage from step 2 by the request from step 5 gives a utilization ratio per container. Below 0.3 means you are paying for capacity you do not use. Above 1.0 means the pod regularly relies on spare CPU it never reserved.

Step 6: Apply the sizing rules

The sizing method on one page: p95 CPU for the CPU request, peak working set plus headroom for memory.

Setting	Starting rule	Notes
CPU request	p95 of 5-minute usage over 14 days	Use p99 for services with strict latency SLOs. If HPA scales on CPU, see the HPA section: the request is also your scaling denominator
CPU limit	None, or 2x to 4x the request	Equal to request only for Guaranteed workloads and benchmarks
Memory request	Peak working set + 20% to 30%	More headroom for runtimes with large off-heap usage (JVM) and for workloads that grow with data
Memory limit	Equal to the memory request	Raise both together. Never set the limit below the observed peak
Ephemeral storage	Request what you write to local disk (logs, temp files, emptyDir)	Pods that exceed an ephemeral-storage limit are evicted

These are starting points, not laws. The point is to have a rule, apply it consistently, and revisit it on a schedule, instead of tuning each service from memory during an incident.

Step 7: Roll out like any other risky change

Changing resources on a Deployment triggers a rollout. Lowering memory is the dangerous direction, so lower it in steps (no more than 20% to 25% at a time), canary it on one service first, and watch OOM kills and restarts for a full traffic cycle. Make sure the rollout cannot take capacity below what you need: a sensible maxUnavailable, a PodDisruptionBudget, and readiness probes that do not flap under the new CPU budget (see how to configure liveness probes without causing restart loops, because a CPU-starved pod failing its liveness probe is a classic self-inflicted outage).

Copy-paste YAML for common workloads

Latency-sensitive stateless service (Burstable, no CPU limit)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry.example.com/api:1.42.0
          resources:
            requests:
              cpu: 500m
              memory: 768Mi
            limits:
              memory: 768Mi   # memory limit = request; no CPU limit

Same service with a runaway guard on CPU

          resources:
            requests:
              cpu: 500m
              memory: 768Mi
            limits:
              cpu: "2"        # 4x the request: protects the node, rarely throttles
              memory: 768Mi

Guaranteed pod for a database or pinned workload

          resources:
            requests:
              cpu: "4"        # integer CPUs are required for exclusive cores with the static CPU manager
              memory: 16Gi
            limits:
              cpu: "4"
              memory: 16Gi

Batch job that should not hog the node

apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-export
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: export
          image: registry.example.com/export:2.3.1
          resources:
            requests:
              cpu: "1"
              memory: 2Gi
            limits:
              cpu: "2"
              memory: 2Gi

LimitRange and ResourceQuota: namespace defaults and guardrails

Individual manifests drift. Namespace policies make sure nothing reaches production as BestEffort and that one team cannot reserve the whole cluster.

A LimitRange fills in defaults for containers that omit requests or limits, and rejects values outside a min and max:

apiVersion: v1
kind: LimitRange
metadata:
  name: container-defaults
  namespace: production
spec:
  limits:
    - type: Container
      defaultRequest:      # applied when a container sets no request
        cpu: 100m
        memory: 256Mi
      default:             # applied when a container sets no limit
        memory: 256Mi      # no CPU default limit on purpose
      min:
        cpu: 10m
        memory: 32Mi
      max:
        cpu: "8"
        memory: 16Gi

A ResourceQuota caps the total a namespace can request:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: production
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    limits.memory: 80Gi
    pods: "200"

Notice what is missing: limits.cpu. Once a quota tracks a resource, every new pod in the namespace must declare that value or the API server may reject it. A quota on limits.cpu therefore forces CPU limits on every pod, which quietly undoes a "no CPU limits" policy. Quota on CPU requests, and on memory requests and limits, is usually what you want.

Keep LimitRange defaults small and treat them as a safety net. A default of 100m CPU for a service that really needs two cores makes it schedulable and then slow; the real fix is still a measured request in the manifest.

Vertical Pod Autoscaler and in-place pod resize

Use VPA in recommendation mode first

The Vertical Pod Autoscaler (VPA) watches usage and computes recommended requests. Its updateMode controls what it does with them:

updateMode	Behavior
`Off`	Only writes recommendations to the VPA status. Nothing changes on pods
`Initial`	Sets requests when pods are created, never afterwards
`Recreate`	Evicts pods whose requests differ significantly from the recommendation (respects PodDisruptionBudgets)
`InPlaceOrRecreate`	Resizes running pods in place, falls back to eviction if that fails
`InPlace`	Only resizes in place and never evicts; retries later if a resize is not possible
`Auto`	Deprecated; currently behaves like `Recreate`

Running VPA with updateMode: "Off" across a namespace is the cheapest way to get a second opinion on every workload, and it feeds the sizing method above:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        minAllowed:
          cpu: 50m
          memory: 64Mi
        maxAllowed:
          cpu: "4"
          memory: 8Gi

kubectl describe vpa api -n production   # see Target, Lower Bound and Upper Bound per container

Do not let VPA and the Horizontal Pod Autoscaler (HPA) act on the same metric. The VPA project explicitly warns against using both on CPU or memory for the same workload; they fight each other. VPA on memory with HPA on CPU, or HPA on custom metrics, is supported.

In-place pod resize is now stable

In-place pod resize lets you change CPU and memory on a running pod without recreating it. It was introduced as alpha in Kubernetes v1.27, became beta (enabled by default) in v1.33, and is stable since v1.35. Resizes go through the pod's resize subresource, which needs kubectl v1.32 or later:

kubectl patch pod api-7d9c6b5f4-x2k8p -n production --subresource resize --patch \
  '{"spec":{"containers":[{"name":"api","resources":{"requests":{"cpu":"800m"}}}]}}'

# Check progress: look for PodResizePending (Infeasible or Deferred) or PodResizeInProgress
kubectl get pod api-7d9c6b5f4-x2k8p -n production -o jsonpath='{.status.conditions}'

Per container, resizePolicy decides whether a change needs a restart:

          resizePolicy:
            - resourceName: cpu
              restartPolicy: NotRequired
            - resourceName: memory
              restartPolicy: RestartContainer   # most runtimes size their heap at startup

Limits worth knowing before you build automation on it: the QoS class cannot change through a resize; decreasing a memory limit without a restart is best effort and is skipped if current usage is above the new limit; Windows pods, non-restartable init containers, and pods under the static CPU or memory manager cannot be resized in place. Also remember that resizing a pod does not change its Deployment: the next rollout brings back the template values. Use resize for emergencies and for VPA's in-place modes, and keep the source of truth in Git.

Requests and HPA: utilization is a ratio of the request

An HPA with a CPU utilization target computes usage divided by request. That has two consequences that cause real incidents.

HPA without resource requests does nothing. If any container in the pod lacks a request for the metric, the utilization is undefined and the HPA takes no action for it. Either set requests on every container (sidecars included), use a ContainerResource metric that targets only the application container, or switch to an AverageValue target expressed in absolute cores.
Changing requests changes scaling. If you cut a CPU request in half during a cost cleanup, the same traffic now reads as double the utilization, and the HPA adds replicas. Retune the HPA target in the same change as the request.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: ContainerResource
      containerResource:
        name: cpu
        container: api          # ignores sidecars with missing or odd requests
        target:
          type: Utilization
          averageUtilization: 70

For the scaling side of this, see how to configure HPA without scaling flaps and how to set up Kubernetes autoscaling without cost surprises. Node autoscalers such as Cluster Autoscaler and Karpenter also provision nodes from pending pods' requests, so inflated requests turn directly into extra nodes on the bill.

Runtime settings: JVM, Go and Node.js inside resource limits

Containers do not hide the host from the process. Each runtime decides how much heap and how many threads to use, and if it guesses from the node instead of the container, your carefully chosen limits will not hold.

Runtime	What it does by default	What to set
JVM (JDK 10+ and 8u191+)	`UseContainerSupport` is on, so the JVM reads the container memory and CPU limits. The default max heap is only 25% of the container memory (`MaxRAMPercentage=25`)	`-XX:MaxRAMPercentage=70` to 75 for heap-heavy services, leaving the rest for metaspace, thread stacks, direct buffers and the code cache. Without a CPU limit, consider `-XX:ActiveProcessorCount` so thread pools match the CPU request
Go 1.25+	`GOMAXPROCS` defaults to the cgroup CPU limit when it is lower than the node's CPU count, and updates if the limit changes. CPU requests are ignored. The heap has no limit by default	Set `GOMEMLIMIT` to about 90% of the memory limit so the GC works harder before the OOM killer acts. Without a CPU limit, set `GOMAXPROCS` from the CPU request. On Go older than 1.25, use `go.uber.org/automaxprocs`
Node.js	Recent versions size the default V8 heap from the container memory limit, but the fraction and the cap vary by version	Set `--max-old-space-size` explicitly (in MiB) to about 75% of the memory limit. Size `UV_THREADPOOL_SIZE` to your CPU request if you lean on the libuv pool

The downward API lets you feed the container's own requests and limits into these settings so they never drift from the manifest:

          env:
            - name: GOMAXPROCS
              valueFrom:
                resourceFieldRef:
                  resource: requests.cpu   # rounded up to a whole number of CPUs
                  divisor: "1"
            - name: GOMEMLIMIT
              value: "690MiB"              # about 90% of a 768Mi memory limit

A Go service on a 64-core node with no CPU limit and no GOMAXPROCS will run 64 scheduler threads against a 500m request. It works until the node is busy, then latency falls apart. A JVM with no heap setting in a 4Gi container uses a 1Gi heap and gets OOMKilled only if off-heap usage grows, which makes the failure look random. Both are configuration problems, not sizing problems.

Troubleshooting: OOMKilled, CPU throttling, Pending pods and evictions

Symptom	Likely cause	How to confirm	Fix
Container restarts with `OOMKilled`, exit code 137	Memory limit below real peak, a leak, or a runtime heap larger than the limit	`kubectl describe pod <pod>` (Last State: Terminated, Reason: OOMKilled); `container_oom_events_total`	Raise memory request and limit together; set runtime heap limits; if usage climbs steadily between restarts, fix the leak
High p99 latency, low average CPU	CFS throttling from a tight CPU limit	Throttled periods ratio query above; `kubectl top pod --containers` looks fine	Remove the CPU limit or raise it to 2x to 4x the request; reduce thread count to match CPU
Pod stuck `Pending` with `0/12 nodes are available: Insufficient cpu` (or memory)	Sum of requests on every node leaves no room, even if real usage is low	`kubectl describe node <node>` and read "Allocated resources"	Right-size inflated requests elsewhere, let the node autoscaler add capacity, or lower this pod's request if it is overestimated
Pods `Evicted` with "The node was low on resource: memory"	Node memory pressure from Burstable pods using more than their requests	`kubectl get events -A --field-selector reason=Evicted`; node conditions show `MemoryPressure`	Set memory request = limit on the pods that grow; reserve system memory on nodes
Noisy neighbor: one service slows when another deploys or bursts	Neighbors with no or tiny CPU requests using idle CPU, or BestEffort pods on the node	`kubectl top pod -A --sort-by=cpu`; compare usage to requests	Enforce requests with a LimitRange, add CPU limits to batch workloads, isolate them on separate node pools
HPA shows `<unknown>` for CPU and never scales	A container in the pod has no CPU request, or metrics-server is missing	`kubectl describe hpa <name>` shows "missing request for cpu"	Add requests to all containers or use a `ContainerResource` metric
Cluster is "full" at 20% real utilization	Requests set from guesses or old peaks	Step 5 utilization ratio per container	Apply the sizing method; run VPA in `Off` mode to confirm

A few commands that answer most questions quickly:

# Requests and limits for every container in a namespace
kubectl get pods -n production -o custom-columns='POD:.metadata.name,CONTAINER:.spec.containers[*].name,CPU_REQ:.spec.containers[*].resources.requests.cpu,CPU_LIM:.spec.containers[*].resources.limits.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,MEM_LIM:.spec.containers[*].resources.limits.memory'

# QoS class of each pod
kubectl get pods -n production -o custom-columns='POD:.metadata.name,QOS:.status.qosClass'

# How much of a node is already reserved by requests
kubectl describe node &lt;node&gt; | grep -A 8 "Allocated resources"

# Live usage per container (needs metrics-server)
kubectl top pod -n production --containers

When an eviction storm is caused by a node drain rather than pressure, the playbook is different: see how to drain Kubernetes nodes without evicting critical workloads.

Kubernetes resource requests and limits checklist

Every container, including sidecars and init containers, has a CPU request and a memory request.
Memory limit equals memory request for nearly every workload.
CPU limits are absent or generous for latency-sensitive services, and present for batch, multi-tenant and benchmark workloads.
Requests come from at least 7 to 14 days of metrics: p95 CPU, peak memory working set plus headroom.
A dashboard or alert tracks throttled periods, OOM kills, restarts, and request-to-usage ratio per workload.
JVM, Go and Node.js settings are aligned with the container limits (heap, GOMAXPROCS, GOMEMLIMIT).
Every production namespace has a LimitRange with sane defaults and a ResourceQuota on requests (not on limits.cpu unless you want CPU limits everywhere).
VPA runs in Off mode as a second opinion, and never on the same metric as an HPA.
HPA targets are retuned whenever CPU requests change.
Resource changes roll out gradually, memory decreases in small steps, with a PodDisruptionBudget in place.
Sizing is reviewed on a schedule (monthly or quarterly), not only during incidents.

FAQ

What is the difference between Kubernetes requests and limits?

A request is what the scheduler reserves for the container on a node and what determines its share of CPU under contention. A limit is a runtime ceiling enforced by the kernel: CPU above the limit is throttled, memory above the limit gets the container OOMKilled. Requests decide placement, limits decide behavior at the edge.

Should I set CPU limits in Kubernetes?

For latency-sensitive services on a cluster where every pod has accurate CPU requests, usually not, because CFS quotas throttle bursty and multi-threaded applications even when the node has idle CPU. Keep CPU limits for batch jobs, multi-tenant clusters, Guaranteed pods that need dedicated cores, and benchmarks. A limit at 2x to 4x the request is a reasonable compromise.

Why is my pod CPU throttled when CPU usage is low?

Throttling is measured per 100 ms period, while dashboards average over much longer windows. A multi-threaded process can use its whole quota in a few milliseconds and then wait for the rest of the period. Check container_cpu_cfs_throttled_periods_total against container_cpu_cfs_periods_total, then raise or remove the CPU limit.

Should memory requests equal memory limits?

In most cases, yes. Memory cannot be reclaimed from a running process, so a gap between request and limit is overcommit that ends in an OOM kill or a node eviction, often of a different pod. Setting them equal keeps failures local to the container that grew.

How do I adjust Kubernetes resources safely?

Measure first (p95 CPU and peak memory over one to two weeks), change one workload at a time, lower memory in steps of 20% to 25% or less, retune any HPA targets in the same change, and watch restarts, OOM kills and latency for a full traffic cycle. On Kubernetes v1.33 and later you can also resize CPU on a running pod in place for urgent fixes, then commit the value to the manifest.

Can HPA work without resource requests?

Not with a utilization target: if any container in the pod lacks a request for the metric, utilization is undefined and the HPA takes no action. Use an AverageValue target in absolute units, a ContainerResource metric for the main container, or custom metrics instead.

What does exit code 137 OOMKilled mean?

Exit code 137 means the process received SIGKILL (128 + 9). With the reason OOMKilled, the kernel killed the container because it exceeded its memory limit or the node ran out of memory. Compare the peak working set to the limit and check whether the runtime's heap setting fits inside it.

Is in-place pod resize production ready?

Yes. In-place resize of CPU and memory is stable since Kubernetes v1.35 (beta and on by default since v1.33), and VPA can use it through the InPlaceOrRecreate and InPlace update modes. It cannot change a pod's QoS class, and memory decreases without a restart are best effort.

Getting resource sizing right across a whole cluster

Requests and limits look like a per-service setting, but the payoff is cluster-wide: accurate requests let the scheduler pack nodes tightly, let autoscalers make the right call, and turn OOM kills and throttling into rare, explainable events. Start with the services that page you most or cost the most, apply the measurement method above, and put the guardrails in place so new services start from good defaults. If you want senior engineers to review your cluster's sizing, autoscaling and node pools with you, book a free production DevOps consultation.

Why & When to use GitOps: Use-Cases & Principles

Michael Zion — Tue, 11 Mar 2025 21:43:41 +0000

Optimizing Kubernetes Management with GitOps and CD Tools.

We’ve come a long way from the days of deploying releases to individual machines, whether manually or by an automated process.

The de-facto standard today is to deploy and manage the application stack using an orchestrator, usually Kubernetes.

However, Kubernetes carries its own challenges. Although you can manage your stack in one centralized place, you still have to manage deployments, updates, rollbacks, and the overall infra.

When Kubernetes was in its early years, services were deployed either manually or through CI/CD using kubectl or Helm.

There was nothing to describe the desired state and no easy way to keep track of changes or revert them. To solve these issues, GitOps was added to the mix.

Single Source of Truth with Increased Visibility and Operability

The idea behind GitOps is a git repo that describes the desired state of the infrastructure and the application stack that runs on it.

Combined with CI/CD, GitOps allows for fast and efficient infra provisioning and application lifecycle management.

In the context of GitOps and Kubernetes, any change to the source code triggers a change in the cluster or application stack.

Changes are tracked and releases are tagged with a version and can be rolled back as fast as they can be deployed.

Best of all, no one is running manual commands which can be very error prone. In essence, GitOps centralises all operations related to deployments and infra management while allowing various teams to both collaborate and work independently.

Continuous Delivery Tools to Enhance GitOps

However, GitOps and CI/CD for Kubernetes still lack something in terms of really being able to manage the infra and application stack.

Managing deployments and infra becomes easier with GitOps thanks to declarative continuous delivery tools for Kubernetes such as Argo CD or Flux CD.

These tools act as a wrapper around your codebase which translates everything declarative into actual state. Your codebase holds all of your Kubernetes manifests, Helm charts, and configuration files. Argo CD then takes it, evaluates it, and applies it to the cluster.

Such tools also offer intuitive and convenient UI to visualize and manage everything that is going on in the cluster.

It’s good to be able to revert a deployment or fix a configuration issue through the UI and on the spot rather than going through the process of committing code and waiting for it to sync. This is especially true in case of an emergency or when developing and debugging.

The Real Benefits of Combining GitOps and CD Tools

When put into the context of everyday work of developers and DevOps, the real benefits of GitOps and CD tools come into light. What GitOps and CD tools offer is a nice package of intuitive and secure easy-to-use framework for management and collaboration.

Ease of Use and Visibility into the State of the Application and Cluster

The truth is that although not too difficult, setting up a GitOps codebase with CD tools can be a complex task. However, once in place, they grant ease of use with increased security.

Reading Kubernetes manifests to understand the complex relationship of services and components is not easy. It’s even harder to debug and troubleshoot issues with no centralized control dashboard.

It’s easier with CD tools because they visualize everything that the code represents.
It’s easier to discern the components of a service or the various CRDs that have been added to the cluster when they appear in front of you with all their dependencies and associations.

CD tools have robust control, logging, and debugging features which allow developers and DevOps to quickly assess the status of the cluster and respond accordingly. There’s no need to run kubectl commands or tail pod logs. Everything is intuitively visualized in the CD tool.

Secure Operations

Having everything visualised and controllable from outside the cluster, does away with the need to access the cluster itself.

‍Developers and DevOps can be given roles through RBAC to determine which resources they have access to and what they can do in the CD tool itself rather than in the cluster.

Although access is ultimately given to components of the cluster and application, it’s still better than having to manage service accounts and having dozens of developers running exec into pods.

For additional security, CD tools sit inside the cluster and communicate with native API so their function takes place inside the cluster and not outside. There’s no need to open the cluster to external CI/CD pipelines.

Quick Recovery, Consistency, and Guardrails

Recovery is built into CD Tools. CD tools continuously monitor the state of the cluster and compare with what the code says.

In case of a mismatch, the CD tool attempts to reconcile between the two. If it fails, it will retry several times before sending an alert and even pausing sync.

Retries are especially important in complex systems with vast networks of dependencies between services and components.

Even if the code dictates service dependencies and the order of deployment, dependencies could still fail.

By retry, CD tools give all components of the system more time and more chances to go online before failing the deployment and pausing progress.

Since there are no imperative commands running without a state, GitOps prevents the occurrence of hanging resources. If you remove a manifest file or parts of it, the CD tools will sync immediately, without any CI/CD pipeline, and remove the corresponding resources and components.

Management of Multiple Environments and Applications

With inherent security, ease of use, and built-in recovery, GitOps and CD tools are easily expanded to support the configuration and management of multi-env, multi-app stacks. It becomes much easier to separate environments:

Different environments for development and production.
Regional stacks for compliance or disaster recovery.
Separate applications for business units in the organization.

Since the initial setup is most of what it takes to get GitOps and CD tools going, it’s incredibly easy to set up multiple stacks and manage both separately, as they are different stacks, and side-by-side in one place, as they are centrally managed.

The Application Needs GitOps and CD Tools

Keeping the application stack and cluster in order and in the desired state is the main function of GitOps and CD tools. There's no guessing how the cluster should be or what should be in and who or what put it there. Everything is conveniently visualized with a vast array of actions that can be performed right from within the UI. GitOps and CD tools give developers and DevOps the ability to work independently in a secure and safe way. Secure in terms of access that is restricted to the CD tools itself. Safe in terms of the ability to quickly revert changes and the self-healing properties of CD tools. It’s not the easiest task to set up GitOps with CD tools, but the agility, convenience, productivity and security that they offer allow for well-managed operations. Well-managed operations then allow for faster and safer development pipelines to further advance your application and product.

How to build a DevOps team

Michael Zion — Tue, 01 Oct 2024 22:27:55 +0000

If you're considering how to build a DevOps team the best way possible, this one's for you.

This blog post is the result of my advice for people building DevOps teams: CTOs, VPs of R&D, Team Leaders, and more.

TL;DR

I'll give you the bottom line.
These are the steps for building your DevOps team:

Polish your DevOps Philosophy
Understand the DevOps Lead responsibilities: Product, Project, People, Service, Architecture
Define your Team's Mission Statement
Set useful DevOps Goals & Practices
Set Guiding DevOps Principles
Define the DevOps team's roadmap & strategy
Set a definition of done
Common DevOps teams pitfalls
Calculate your team's required capacity
Hire the right DevOps Engineers

You can also apply these to existing DevOps teams and level-up your organization.
Let's get started!

Polish your DevOps Philosophy

You must first understand why organizations need DevOps, and how it can work in practice.

DevOps is an enabler role.

It's meant to enable the developers to build, improve, and take ownership over the system.

Let's break it down:

Build = Create something new. Could be a new microservice, a new database, or a new monitoring dashboard.
Improve = Introduce a change into something that exists. Could be fixing a bug, changing a database schema, or changing an alert threshold.
Take ownership = Take charge of problems that arise with what you built and improved. This means when something needs improvement, the owner is the one who does it.

The above are what the developers should do.

Now, DevOps should enable it.

Note, to enable is not to give full permissions and let them have fun.

To enable is to give developers what they need to build, improve, and own the system.

But, do it in a way that focuses the developers' energy in the right direction, and in a safe way.

Understand the DevOps Lead Responsibilities

You have 5 hats as a DevOps Lead:

Product (Platform)
Your clients are your company's developers.
Provide them with tools, knowledge, and automations.
Tools = Polished automations.
Automations = Automated knowledge.
Knowledge = Hard-earned information & insights.
Understand their requirements, and use Tools, Automations, and Knowledge to fulfill them.
Understand why they want what they want: Is it because of an underlying issue with the system? If yes, solve it before building anything.
If a tool or automation will save time, consider building or implementing it.
Service
Don't let your developers wait until you do something "the right way".
Sometimes they need immediate help to complete something.
Help them first, and invest time later in automation and tooling.
Project
Managing your DevOps team requires managing the work it does.
Turn the team's philosophy into a mission.
Turn the team's mission into goals.
Turn the team's goals into a roadmap.
Turn the team's roadmap into tasks.
Prioritize the tasks.
Set simple roles and responsibilities.
Hold your team members accountable to progression.
Make it easy to inform relevant team members on updates.
Make it easy to consult with a teammate on a subject of their expertise.
People
Each DevOps team member has type of work and tasks they enjoy more.
One enjoys sharing knowledge, another enjoys being of service, and some enjoy building tools.
Different teammates are also interested in different subjects: Monitoring, Infrastructure, CI/CD, etc.
A team member that's working on what they enjoy is more fulfilled and productive.
Strive to overlap each teammate's goals with the company's goals.
Architecture
You have 2 architectures to worry about.
1) The company's product architecture (built by the developers).
2) The DevOps platform's architecture (built by the DevOps team).
Help the developers understand and control the application's effect on the infrastructure.
Enable the developers to make informed decisions by providing context.
Finally, build the platform to support the developers requirements.
At every step, only limit decisions that damage the company.
Examples: High cost, Stability impairment, Restricts observability, etc.

Define Your DevOps Team's Mission Statement

Here's one for you:

"Enable the developers to build, improve, and own the system".

It's pretty minimal, so it's going to help your team stay focused.

A healthy sign it catches on: When the team debates a decision regarding a task, they ask "Does it help us enable the developers to build, improve, or own the system?".

That's when you win.

Set Useful DevOps Goals & Practices

Your company's success will be determined by its speed and its product's quality.

To make it happen, you should know programming is science, not maths - I'll explain.

People used to think programming will be a mathematical discipline.
They thought programmers will write functions and mathematically prove them.

Not what happened - Programming is a scientific discipline.

You write code, your test it, and you assume it's good - until one test fails.

In essence, you're experimenting.

You might ask: "wtf? how's this related to setting useful goals?"

The answer is that the first thing you want to enable is running experiments easily.

So here are some useful goals:

Developers can easily test their code in a consistent manner
Production and testing environments are identical (Production will benefit from the quality of the tests)
Developers can easily collaborate
Developers can understand the state of the system and the impact of changes

Let's translate those goals into smaller goals or practices that will achieve the goals:

Developers can create a testing/production environment with "One-Click"
There's a continuous integration process that enables the developers to collaborate by agreeing on the current up-to-date version of the system
Auto-create dashboards and alerts for new services

Set Guiding DevOps Principles

Some useful DevOps principles that help save your team time, improve the speed of delivery, and keep the system healthy:

PoC before doing things "the right way"
Make it work, then make it better
The entire system should be fully recoverable from Git
Use tools with a big community and well-documented interface
Equip key-developers with DevOps knowledge to be the first point-of-contact for their team members (super users)

Define the DevOps Team's Roadmap & Strategy

Roadmap = Goals * Strategy.

Once you set the goals (as mentioned above), you can start prioritizing.

♯1 - The DevOps Categories

Enabling developers requires a DevOps Engineer to handle and enable the following:

Provision infrastructure
Deploy workloads
Monitor the system
Recover from issues
Scale up and down
Track & test changes (Codebase Management)
Secure the system
Store & retrieve data
Configure the system

♯2 - Examine each goal through each category

Every DevOps goal you set should be examined through the lens of each category.

The reason is that together the categories cover each aspect of the building, improving, and owning of a software operation.

Let's do an example:

Goal: Create a "One-Click Environment Automation"
Categories to address:
- How should its infrastructure be provisioned?
- How should its workloads be deployed?
- How should its metrics and logs be sent, stored, and queried?
- ...

♯3 - Strategic Principles

Reach at least 50% capacity of working on the DevOps goals as soon as you responsibly can - Support the developers and teach them how to self-support to achieve that
Easy to modify > Perfect - When you do something that isn't perfect due to a lack of time, do it in such a way that modifying it later on to improve it is easy
Prerequisites first: Codebase Management -> Infrastructure -> Deployment -> Configuration -> Data Management.
At least moderate foundations quickly, reinforce later, but never weak foundations: If you don't use any boilerplates, or if you are not proficient in something early on, and it's a foundation (like infrastructure), then don't give up and build weak foundations, but also don't over-invest and build strong ones if it's too time consuming. Instead, moderately invest in the foundation, and revisit it later.

Set a Definition of Done

Also known as a Definition of done.

Ask the following questions for every single component in your system:

Monitoring: Are there metrics, logs, traces, and alerts setup in an actionable way?
Availability: Is there a mechanism to keep it alive during incidents?
Resiliency: Can it recover from an error quickly?
Recovery: Can it be fully restored to a previous state?
Testability: Is it possible to test changes to it?
Deliverability: Is there a process to release changes to it?
Persistency: Will its data persist if the system is hindered?
Integrable: Does it have a consistent and predictable interface allowing integration with it?
Security: Is it accessible only by the parts of the system that absolutely need it?
Dependencies: Are its dependencies fully tracked and managed?

Common DevOps Team Pitfalls

Pitfall: DevOps work blocks developers work
Indicator: Developers need to wait for DevOps team changes to complete before continuing work
Cause: The DevOps team doesn't utilize its own practices (AKA shoemaker's son always goes barefoot)
Pitfall: Only support developers and maintain the system
Indicator: No progress on any DevOps goal or task
Cause: Either there are no clear DevOps goals or repeating developers requests haven't been automated
Pitfall: Adopting 'Best' practices instead of 'Suitable' practices
Indicator: Introducing methodologies and adhering to principles that go against the company's goals
Cause: Prioritizing methodology over company goals, usually because of a disconnect from the company goals or due to lack of DevOps experience

Calculate Your DevOps Team's Required Capacity

Required DevOps Capacity = (Scale * Complexity) / Leverage.

Leverage = Level of DevOps Engineers * Company Resources * Team Focus
Scale = Number of instances of each component * Number of people in the organization
Complexity = Number of components * Number of teams

Hire the Right DevOps Engineers

The types of DevOps Engineers:

Barrels vs. Ammo
~ Ammo = People who can complete tasks but won't initiate them.
~ Barrel = People who understand what tasks are needed next but won't complete them.
Interviewer Tip: During the technical interview with a candidate ask about past DevOps accomplishments, and ask how the tasks were created and who did them.
Aspiration-Oriented vs. Prevention-Oriented
~ Aspiration-Oriented = Has goals, positive feedback encourages and focuses them while negative feedback discourages them and kicks them off track.
~ Prevention-Oriented = Avoids problems, positive feedback makes them lay back and reduce capacity while negative feedback focuses them and keeps them on track.
Interviewer tip: See if there's a common theme for projects the candidate did in the past. People focused on security tasks and deep attention to specific details are more likely to be prevention-oriented, while people who initiated many projects spanning multiple (DevOps) categories are more likely to be aspiration-oriented.

Working with service providers:

Set clear desired results
And let the DevOps service provider assist in exploring the goal and provide perspective from other companies.
Expect transparency on progress
And judge it against the latest plan.
Expect clear planning
And make sure it has clear goals, takes into consideration the risks, and has a strategy that adheres to your DevOps principles.

Summary

Lots of stuff covered over this one-pager, and still much was left outside.

If you take away 1 thing from it, let it be this: A simple DevOps team mission statement is the most significant thing you need.

It sounds over-simplistic and abstract, but without it there is not guiding principle for how DevOps in the organization should look like.

Hope you enjoyed, and send me an email at michael@meteorops.com if there's anything else you'd like to see in here.

Don't waste 6 months on hiring DevOps, take the 10 Hours DevOps Pill

Michael Zion — Thu, 04 Jan 2024 14:55:06 +0000

A practical playbook for founders and engineering managers to stop wasting time trying to get started with DevOps the right way

Is this article for you?

Maybe you're a CTO.
Maybe you're an engineering manager.
If you think about hiring you first DevOps Engineer - this article is for you.
It will save you 6 months of wasted mental capacity.

How? By dedicating 10 hours to planning.
I'll share with you here exactly what to do during those 10 hours.

You might know this story TOO well

~ "I need to focus on the product"

You are building your company’s product.
You start thinking about the development process and building or improving production.
Then, you read about DevOps somewhere and think to yourself: “This seems like what I need”.

~ "DevOps is expensive"

You start looking for a DevOps Engineer, but you find out the standard DevOps salary - “Oh, that’s pretty expensive for our budget”.
So you decide to wait. “Hiring a developer is more important right now” you tell yourself.

A month goes by, and you and your team are grinding away, building a kickass product.
But alas! The missing development automation hurts collaboration, the cloud infrastructure is a collection of manual changes - and you’re not sure if the system is built well.

So, you make a decision, and you post your first “DevOps Position”, congrats!

~ "I'll interview some candidates"

Candidates start streaming in, and you start interviewing.
After interviewing the 3rd candidate, you think to yourself “this is not it”.
Every candidate is missing a different piece of the required knowledge.
One doesn’t know AWS, the other isn’t familiar with "", and the last one only worked with on-prem before.

And so, you re-evaluate.

~ "I'll do it myself"

The realization is this: You can’t hire on knowledge for this position, you have to hire on skill and character.
BUT, you feel like the knowledge is crucial, at least at this point.
So you make another decision - I’ll invest some more time in this “DevOps” thing - I’ll learn.
You watch DevOps tutorials on Youtube.
You even subscribe to various DevOps Newsletters, something you didn’t do before.
Another month goes by.

~ "What do other startups do?"

After learning and implementing DevOps bits and pieces all over the place, it doesn’t feel right.
“Am I re-inventing the wheel and complicating the future?” you might ask yourself.
You take a step back, zoom-out, and ask other startup founders what they did at this stage.

That’s when you find out you’re not alone - “Oh! So it’s tough for other founders as well!”
But, each founder tells you something different.
One founder tells you to hire a freelancer - but that seems like just another hiring model and doesn’t solve the problem.
Another founder tells you to just hire a full-time DevOps Engineer - but your resources are limited, and you’d rather hire more developers to build the product.
Until one founder tells you to hire an agency: “Instead of a freelancer, get a company that’s built to get your DevOps right”.

~ "Maybe an agency?"

You’re convinced, and you start looking for agencies.
You get recommendations, you search online, and you start booking calls.

*1st call with a DevOps agency
*“Hi there! Yes we’re the solution for you! Let’s get started!”

2nd call with a DevOps agency
“Yes! We’ll provide you with a strong DevOps Engineer! Let’s get started!”

~ "I'll do it myself"

You choose not to go on the 3rd call and you pause the process.

Why? Because you realize: “These guys didn’t remove any uncertainty for me - how are they different from a freelancer?”
So you make another decision.
You’ll continue to do DevOps yourself, and maybe delegate some tasks to other team members.

That’s when you find out it was a trap! You got yourself into a loop!
‍

~ "Oh no! it's a loop!"

After just one month you realize it’s not sustainable, and you repeat the same process:

Open DevOps position
Realize each candidate is missing something else
Fear for the budget
Stop the hiring
Complete DevOps tasks yourself
Consider a freelancer
Drop the idea because it's similar hiring
Consider an agency
Interview a bunch of agencies
No extra certainty or confidence from an agency
Do DevOps work yourself
Repeat

This loop is sometimes 2 months long, and sometimes 12 months long.

Either way, it slows down your company!
It occupies your thoughts.
It takes valuable time from you.
It speeds up global warming.

Maybe the global warming part was an exaggeration.

It’s amazing how many founders went through this exact same process.

Break the loop

I’m here to tell you:

You can break the DevOps hiring loop
You can promote DevOps efforts quickly AND accurately
You can save 6-12 months of stumbling around

You need just 10 hours with a DevOps Expert

You need just 10 hours with a DevOps expert.
What you need most is perspective from an experienced individual.
You don’t yet have the DevOps perspective or experience? No problem.
Get a DevOps Expert to consult you on a hourly basis.
I call it "The 10 Hours DevOps Pill" 💊

Requirements for the DevOps expert

Not anyone will fit as your trusted DevOps expert.

Use someone that answers these criteria:

Helped 10+ companies with DevOps - Has a wide perspective
Consulted CTOs and VPs of R&D before - Understands the weight of culture in adopting DevOps
Consulted DevOps Group and Team Leaders before - Understands how to manage the work
Did lots of DevOps hands-on work before - Understands how to do the work
You understand them - They know where you stand and have simple explanations. (Run away from the buzzwords throwers)

Spoiler: The contents of the 10 hours

2h - Share everything about your startup
2h - Ask what’s possible + What others did
2h - Set DevOps goals
2h - Choose a strategy
2h - Create a DevOps roadmap Let’s dive into the playbook for each meeting!

1st + 2nd Consultation Hours: Share, share, share

Share the state of your company with the DevOps expert.
Share your infrastructure, your code, your work process, everything!
Don’t be shy, and don’t justify - simply explain your startup’s goals and your reasoning.
(Sign an NDA first, of course)

3rd + 4th Consultation Hours: Understand what’s possible + Gain perspective

Take notes or record this call!

Ask the DevOps expert: What’s possible?

How fast can changes reach production?
How fast can you create new environments?
How easily can your developers collaborate?
How confidently can you change the system?
What uptime can the system have?
How aware can your team be of the system’s state?

Ask the DevOps expert: What did other startups do at this stage?

What goals did they have?
What DevOps efforts did they promote?
Why did they decide those DevOps efforts?
Which startups performed best? Why?
Are there industry-unique considerations?

5th + 6th Consultation Hours: Set DevOps goals

You now know what’s possible, and what other companies do.
Time to set short-term, mid-term, and long term goals.
If you want to dive deep into useful DevOps principles, check out the CTO DevOps Handbook here 👈🏼.

7th + 8th Consultation Hours: Strategize

Based on your goals, strengths, and restrictions, choose a strategy.
Discuss the following options with your DevOps expert.

Possible Strengths

Developers with DevOps background
High budget
Available time to manage the DevOps efforts

Possible Restrictions

No budget
Low budget
DevOps needs are temporary for now
No time to manage the DevOps efforts

	DevOps Engineer	Freelancer	Agency	Winner
Budget	High	Medium	Medium	Freelancer / Agency
Management Effort Required	High	High	Low-Medium	Agency
Hourly Flexibility	Low	Medium	High	Agency
Ease of bridging knowledge gaps	Low-Medium	Low-Medium	Medium-High	Agency
Ease of building trust	High	Medium-High	Low-Medium	DevOps Engineer

The available strategies

Hire a DevOps Engineer, Boost with the DevOps expert

Create a DevOps roadmap with the DevOps Expert before hiring
Get the DevOps expert to interview the candidates
Get advice from the DevOps expert on what to examine during your interview with the candidates
Prioritize the DevOps efforts with the DevOps Expert
Plan the onboarding of the DevOps Engineer with the 6. DevOps expert
Hire the DevOps Engineer
Get advice on managing DevOps projects from the DevOps expert
Discuss the DevOps tasks implementation with the DevOps Engineer and DevOps expert
Occasionally or routinely consult with the DevOps expert on priorities, management, and engineering

Hire a DevOps freelancer, Boost with the DevOps expert

Exactly as the above strategy
Take advantage of the hourly flexibility when needed

Hire a DevOps agency, Boost with the DevOps expert

Create a DevOps roadmap with the DevOps Expert before choosing an agency
Share your requirements with the agency
Filter out any agency that doesn’t provide you with a plan BEFORE work starts
Get the plan reviewed by your DevOps expert: Does it make sense? Is the planned result desired?
Make sure the DevOps agency’s ability to provide support after the work is done
Asses the level of the DevOps agency’s DevOps Engineers with the help of the DevOps expert
Make sure the DevOps agency is flexible with its hourly capacity
Occasionally or routinely consult with the DevOps expert on priorities, management, and engineering

Advantages and disadvantages of the strategies

Option	Good when	Bad when
Full-time + Expert	- Your budget is high - You need flexible DevOps capacity - Your team has DevOps background	- Your budget is low
Freelancer + Expert	- Your budget is low - You need flexible DevOps capacity - You can't risk losing DevOps capacity short-mid-term
Agency + Expert	- Your budget is low - You need flexible DevOps capacity - You want an accurate plan before work starts - You frequently need to support new tools and technologies	- The agency doesn’t have flexible plans - The agency’s team isn’t experienced - The agency doesn’t create clarity before work starts

9th + 10th Consultation Hours: Create a DevOps roadmap + Execute your strategy

You’re ready to create a DevOps roadmap and start executing your strategy!
Creating a DevOps roadmap is the first step of each of the strategies described above.

You had to do a bunch of things first:

You needed a DevOps expert to help you break the vicious DevOps loop - Done.
You needed to set DevOps goals rooted in a deep DevOps understanding and perspective - Done.
You needed to get familiar with different strategies and choose the best one for you - Done!

You’re left with 2 things to do:

1 - Create the DevOps roadmap based on the goals and strategy you’ve set
2 - Start executing the strategy:

Share the DevOps goals with the team
Schedule the interviews with the Engineers/Freelancers/Agencies
Use the DevOps expert to stay on track
Set the milestones for the DevOps goals
Create the tasks
Complete the tasks

Prepare before starting the process

You want to do two things before jumping into the 10 hours:

Before the first 2 hours: Perfect your DevOps terminology (will support rich discussions)
Before the last 2 hours: List existing efforts that might slow down the DevOps roadmap (will help prioritizing)

Ask me anything

If you’re thinking about taking this “10-hours DevOps Pill”, feel free to reach out here.

Do it yourself

You now know the general process for breaking the DevOps loop.
You don’t NEED a DevOps expert.
Feel free to subscribe to my Newsletter and learn how to do it yourself here 👈🏼

The CTO DevOps Handbook: Simple Principles and Examples

Michael Zion — Mon, 25 Dec 2023 21:28:13 +0000

Nail the DevOps part as your company's CTO

The goal of this handbook is to give you clarity on DevOps:

Understand what’s DevOps (in simple words)
Know what’s possible with DevOps (in simple goals)
Get simple “when-to-do-what” DevOps guidelines ‍

I added a bonus at the bottom of the article.
It's a production-ready setup example you could take inspiration from.

Who this article is for

You might be a founder who wishes to get started with DevOps the right way.

You might be a CTO of a 1,000 employees company who wishes to get simple principles.

Or, maybe you’re a Software Engineer, and you want to understand if your company’s DevOps approach is good.

If you’re looking for a simple DevOps playbook, this is it.

Understand the desired result

Two things your company needs to be able to do

Serve its product to customers
Build and improve the product

Abilities you need to build, improve, and serve software

Run experiments and test changes

DevOps has a simple meaning

Developers and Operators have shared responsibility for building and improving the system.

In practice:

Developers are responsible to “Operate”
DevOps Engineers are responsible to enable to “Operate” AND do some of it themselves

Operate = provision, monitor, secure, configure, deploy, scale.

Choose a balance: Enabler, Doer, or Automator

The DevOps role will end up as a balance between:

Enabler: Provides the tools and knowledge to fulfill the DevOps goals
Doer: Does the tasks that fulfill the DevOps goals
Automator: Automates any repeating operation

Know what things you should enable, do, or automate

Provision infrastructure
Secure the system
Deploy workloads
Monitor the system
Recover from issues
Scale up or down
Track & test changes
Automate processes

Choose the right tools

Has state management = Saves time automating state-aware processes (e.g., Terraform)
Has a big community & good docs = Saves time dealing with common issues (e.g., Kubernetes)
Has multiple interface types: API, CLI, UI = Saves time integrating with the existing system (e.g., Vault)

You can also read about choosing tools here.

Set useful goals

There are DevOps goals that adopting them will focus you on the right direction:

One-Click Environments: makes e2e tests easy and quick
Atomic Commits: provides confidence that a tested change will work in production
Separate the Shared & Env-Specific Parts: enables e2e tests as the company scales up

If you want to learn about more useful DevOps goals, feel free to book a free consultation here.

Enablers: Choose the Tools-to-Knowledge Balance

Developers can either have the knowledge or the tools to do something.

More knowledge-reliance: if you want the developers to contribute to the DevOps efforts
More tools-reliance: if you want to abstract the operations from the developers

If the balance between the two is not intentional, it’s accidental.

Doers: Have a good reason to do it

Is it a one-time task?
Does it teach you how the developers work?
Are you directly accountable for the results of the task?

If you answered “no” to the above questions, enable or automate it instead.

Doing more = Learning the system's use-cases

Doing too much = Not scalable, too-much knowledge-reliance

Automators: Have a good reason to automate it

Did it happen before?
Is it likely to happen again?
Will automating it take less time than doing it?
Will automating it teach you an important company process?

If you answered “yes” to 2 out of the 4 questions - automate it!

More automations = Less reliance on knowledge to operate the system.

Too much automations = No system awareness.

P.S. - you can also enable developers to automate it.

Create available DevOps Capacity

The DevOps needs of a company have spikes.

One month you need 2 DevOps Engineers, and half of that the next month.

Switchovers between big efforts and small tasks are common.

This is true, especially for new companies.

Break the assumption: “DevOps tasks must be done by a DevOps Engineer”.

There are 3 types of DevOps capacity

Non-Flexible: A full-time DevOps Engineer on the team
Semi-Flexible: Key developers that can contribute to the DevOps goals
Fully-Flexible: A flexible DevOps Services company or freelancer

You can read more about calculating the DevOps capacity your company needs here.

When to focus on what: Common Dilemmas

When: You work alone, and the system is simple

‍Focus: On simplifying the development - Dockerize your apps, Create a post-commit pipeline that runs tests

When: You need to be able to create new environments quickly (for development, or for clients)

‍ Focus: On implementing “One-Click Environments”: Using IaC (e.g., Terraform) + Deployment tool (Depends on the platform).

When: You want to e2e test every code modification, but there are many code modifications

‍> Focus: On splitting the “One-Click Env” into a “base” with shared resources, and “env” with env-specific resources

When: You want to unify & standardize how you deploy, monitor, scale, configure, and secure your workloads

Focus: On implementing an orchestrator such as Kubernetes

When: You want you have many moving parts and wish to be certain a tested change will work

‍ Focus: On implementing GitOps and consider a Monorepo (the sooner the better)

When: You want the DevOps efforts to be done by the dev team

‍ Focus: On using “actual” IaC tools (Pulumi Typescript/Python), Full “how to operate” (see above) documentation‍

Never: Invest lots of time in new tech without a strong reason

Always:

Have your code in Git
Monitor the basic stuff: CPU, Memory, Disk, Network, App Logs, Cloud Costs
Architect for high-availability
Test before you deploy

BONUS: An example setup for a CTO approaching Production

2 AWS Accounts

One for development and staging
Another for production

Monorepo in Github

Docker-Compose for local development

2 Infrastructure-as-Code projects: 'base' & 'apps'

base = shared resources (e.g., VPC, RDS, ECS Cluster, EKS Cluster)
apps = env-specific resources (e.g., Lambda Functions, ECS Services, Kubernetes Namespaces)
config file per environment

Github Actions Workflow: Development workflow

Checkout branch and locally develop + test changes
Create a Pull Request: Deploys a Pull-Request ‘apps’ environment on the ‘development’ environment ‘base’
On merge to main: Deploys from the ‘main’ branch an ‘apps’ environment onto the ‘development’ environment ‘base’
Manual: Deploy from the ‘main’ branch onto the ‘staging’ / ‘production’ environment ‘base’

Setup Notes

Avoid mentioning an environmnent's name in the code for conditional resources deployment
Use each environment’s config file to declare if a resource should be created
Could be implemented using Terraform, Terragrunt, Pulumi, CDK, and other IaC tools
Production should have 2-instances of every workload for high-availability

If you’d like to see this setup in your startup, click here to book a call 👈🏼

P.S. - I'll be updating this page occasionally, so you might want to visit again

Another Bonus: DevOps Dictionary for Human Beings

Term	Definition	Tools
Environment	A working instance of the entire system
CI (Continuous Integration)	Enable developers to collaborate by agreeing on a single source-of-truth (master/main)	Jenkins, Github Actions, GitlabCI
CD (Continuous Delivery)	Create an artifact that’s ready for production (tested, tagged)	JFrog Artifactory, Nexus, AWS ECR
CD (Continuous Deployment)	Every available deliverable (artifact) gets deployed automatically	ArgoCD, Jenkins, AWS CodeDeploy
Monitoring / Observability	Collect metrics/traces/logs from apps and infrastructure, analyze them, and display them, and setup alerts	Prometheus, Jaeger, Elasticsearch, Fluentd, OpenTelemetry
Infrastructure	The resources on which the workloads run, in which the data is stored, and through which the network flows	Servers, Databases, Network Routers & Switches
Cloud Infrastructure	Same as the above, but specifically in the cloud	AWS EC2, AWS RDS, GCP Compute Engine, Azure Virtual Machines
Cloud	Computing & Data services served from remote locations for you to build your system	AWS, Azure, GCP
Containerization & Virtualization	Technologies utilizing Kernel & OS features to create virtual machines, or isolate process (AKA run containers)	Docker, vSphere, KVM
Secrets Management	Storing and retrieving sensitive configurations (e.g., tokens, passwords)	Hashicorp Vault, AWS Secrets Manager, SealedSecrets
Configuration Management	Usually refers to preparing servers for workloads (e.g., creating directories & files, starting processes)	Ansible, Chef, Puppet
Version Control	Saving the code in a versioned way (Git)	Github, Gitlab
GitOps	Making the system is the same as it’s described in Git	Flux, ArgoCD, Jenkins
Monorepo	All of the company’s code is in one Git Repository	NX, Turborepo
Polyrepo	Multiple Git repositories for different components
IaC (Infrastructure-as-Code)	Creating Cloud infrastructure with idempotent code and state management	Terraform, Pulumi, CDK, Crossplane
Deployment	Execute, serve, or install the artifacts	ArgoCD, Jenkins, AWS CodeDeploy, Scripts (Bash, Python, etc.)
Orchestrator	Dynamically allocating workloads to a pool of nodes	Kubernetes, Nomad, AWS ECS
Authentication & Authorization	Making sure each person, workload, or resource, has access only to what’s necessary (other workloads and resources)	AWS IAM, OpenID, OpenVPN, Twingate, Istio
Service Discovery	Exposing available workloads using DNS	Consul, CoreDNS

Get more practical advice

I post small nuggets of practical advice on the "MeteorOps Newsletter".
You can subscribe here 👈🏼