DEV Community: CloudSkill Consulting LLC

A Lambda That Writes Back to the Bucket That Triggers It Can Bill You in a Loop. Here's a Safe Setup

CloudSkill Consulting LLC — Thu, 29 Oct 2026 02:25:53 +0000

As of October 29, 2026, this post sums up the official pages linked here.

The problem

A 2019 post on Reddit's r/aws describes a bill nobody wants. A team had a Lambda function that listened for new files in an S3 bucket. It read each file, changed it, and saved the result to S3. But the source bucket and the output bucket were set to the same bucket. So every saved result was a new file, and every new file started the function again.

It ran like that for about three and a half days, over a weekend, in a new account with no cost alerts yet. The bill was over $6,800, mostly for S3 PUT requests. The team only found out when someone at AWS asked them about the traffic.

This is a common trap. AWS even warns about it in the Lambda docs. In this post we explain the root cause, show a setup that can't loop, add guard rails in case someone breaks it later, and show how to test all of it.

Root cause: the output matches the trigger

An S3 trigger is a rule on the bucket. It says, "When an object is created, call this function." AWS's Lambda docs put it plainly: if the function uploads an object to the same bucket, it "indirectly triggers itself."

Here is the loop:

A user uploads orders.csv. S3 calls the function.
The function saves orders.json to the same bucket.
That is a new object, so S3 calls the function again.
The function saves another file. Go to step 3.

Three things make it costly:

Each turn is billed. You pay for the Lambda run and for the S3 PUT. The Reddit bill was mostly PUT requests.
Lambda scales out. Each new file is its own event, so many copies can run at once.
It can be quiet. If the function overwrites the same key each time, the file count doesn't grow. Nothing looks wrong in the bucket.

A second, smaller bug often hides here too. The object key in an S3 event is URL encoded. AWS's example: red flower.jpg arrives as red+flower.jpg. If your code uses the raw key, it can't read the file, the run fails, and S3's async retry calls the function again.

Doesn't AWS stop loops now?

Partly. Lambda has recursive loop detection, and it is on by default. This came after that post. Today the docs list Amazon S3 as a supported service, with SQS, SNS and EventBridge custom buses. If one triggering event calls your function about 16 times in a chain, Lambda stops the next call and notifies you.

That helps a lot, but don't treat it as the fix:

It only works if the function uses a supported AWS SDK version (for Python, boto3 1.24.46 or later). Check what your runtime or package ships.
It allows about 16 runs per chain before it stops. Many uploads at once means many chains.
Loops through services it doesn't track (the docs name DynamoDB) are not stopped.
Anyone can switch a function to Allow.
The email and Health Dashboard alerts can take up to 3.5 hours.

So we design the loop out, then keep detection on as a safety net.

The fix: inputs and outputs never overlap

AWS's docs give two safe options: use two buckets, or trigger only on a prefix used for incoming objects. Two buckets is the simplest and the safest. When you must use one bucket, set it up so the function's output can never match the trigger. We use four layers:

Trigger filter. Fire only for incoming/*.csv. Write output to processed/*.jsonl. The filter never matches the output.
IAM. The function may read only incoming/* and write only processed/*. Even buggy code can't write back under the trigger prefix.
Code guards. Refuse to start if the two prefixes overlap. Skip any event outside incoming/.
Limits and alarms. Cap concurrency, keep loop detection on, and alarm on spikes.

The handler

# handler.py - Lambda (Python 3.12). Triggered by S3 ObjectCreated on incoming/*.csv.
# Reads the CSV, writes JSON Lines under processed/. Never writes under incoming/.
import csv
import io
import json
import os
from urllib.parse import unquote_plus

import boto3

IN_PREFIX = os.environ.get("IN_PREFIX", "incoming/")
OUT_PREFIX = os.environ.get("OUT_PREFIX", "processed/")

s3 = boto3.client("s3")


def output_key(in_key: str) -> str:
    """incoming/2026/orders.csv -> processed/2026/orders.jsonl"""
    rest = in_key[len(IN_PREFIX):]
    if rest.endswith(".csv"):
        rest = rest[:-4]
    return OUT_PREFIX + rest + ".jsonl"


def check_config() -> None:
    # Guard 1: the two prefixes must not overlap, or our own writes match the trigger.
    if OUT_PREFIX.startswith(IN_PREFIX) or IN_PREFIX.startswith(OUT_PREFIX):
        raise RuntimeError(f"OUT_PREFIX {OUT_PREFIX!r} overlaps IN_PREFIX {IN_PREFIX!r}")


def handler(event, context):
    check_config()
    written = []
    for record in event.get("Records", []):
        bucket = record["s3"]["bucket"]["name"]
        # Keys in S3 events are URL-encoded: a space arrives as "+".
        key = unquote_plus(record["s3"]["object"]["key"])

        # Guard 2: ignore anything outside the input prefix, even if a
        # trigger is set up wrong later.
        if not key.startswith(IN_PREFIX):
            print(json.dumps({"skip": key, "reason": "outside IN_PREFIX"}))
            continue

        body = s3.get_object(Bucket=bucket, Key=key)["Body"].read().decode("utf-8")
        rows = csv.DictReader(io.StringIO(body))
        lines = "".join(json.dumps(row) + "\n" for row in rows)

        out = output_key(key)
        s3.put_object(Bucket=bucket, Key=out, Body=lines.encode("utf-8"),
                      ContentType="application/x-ndjson")
        written.append(out)
    return {"written": written}

Why processed/ and not incoming/processed/? Because incoming/processed/x.jsonl still starts with incoming/. The trigger would match it, and the loop is back. Guard 1 catches that mistake before any write.

The Terraform

terraform {
  required_providers {
    aws     = { source = "hashicorp/aws", version = "~> 6.0" }
    archive = { source = "hashicorp/archive", version = "~> 2.0" }
  }
}

variable "bucket_name" {
  type = string
}

variable "alarm_topic_arn" {
  type        = string
  description = "SNS topic that gets the alarm emails"
}

locals {
  in_prefix  = "incoming/"
  out_prefix = "processed/"
}

resource "aws_s3_bucket" "data" {
  bucket = var.bucket_name
}

data "archive_file" "fn" {
  type        = "zip"
  source_file = "${path.module}/handler.py"
  output_path = "${path.module}/build/handler.zip"
}

resource "aws_iam_role" "fn" {
  name = "csv-to-jsonl"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

# Read only from incoming/, write only to processed/.
# Even a code bug can't put an object back under the trigger prefix.
resource "aws_iam_role_policy" "fn" {
  role = aws_iam_role.fn.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = "s3:GetObject"
        Resource = "${aws_s3_bucket.data.arn}/${local.in_prefix}*"
      },
      {
        Effect   = "Allow"
        Action   = "s3:PutObject"
        Resource = "${aws_s3_bucket.data.arn}/${local.out_prefix}*"
      },
      {
        Effect   = "Allow"
        Action   = ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"]
        Resource = "*"
      }
    ]
  })
}

resource "aws_lambda_function" "fn" {
  function_name    = "csv-to-jsonl"
  role             = aws_iam_role.fn.arn
  runtime          = "python3.12"
  handler          = "handler.handler"
  filename         = data.archive_file.fn.output_path
  source_code_hash = data.archive_file.fn.output_base64sha256
  timeout          = 60

  # A ceiling, not a fix: a loop can't fan out past 5 copies at once.
  reserved_concurrent_executions = 5

  environment {
    variables = {
      IN_PREFIX  = local.in_prefix
      OUT_PREFIX = local.out_prefix
    }
  }
}

# Keep Lambda's loop detection on. "Terminate" is the default; writing it down
# stops someone from quietly switching it to "Allow".
resource "aws_lambda_function_recursion_config" "fn" {
  function_name  = aws_lambda_function.fn.function_name
  recursive_loop = "Terminate"
}

resource "aws_lambda_permission" "s3" {
  statement_id  = "AllowS3Invoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.fn.function_name
  principal     = "s3.amazonaws.com"
  source_arn    = aws_s3_bucket.data.arn
}

# The real fix: the trigger only fires for incoming/*.csv.
# Output goes to processed/*.jsonl, which this filter never matches.
resource "aws_s3_bucket_notification" "data" {
  bucket = aws_s3_bucket.data.id

  lambda_function {
    lambda_function_arn = aws_lambda_function.fn.arn
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = local.in_prefix
    filter_suffix       = ".csv"
  }

  depends_on = [aws_lambda_permission.s3]
}

# Alarm 1: Lambda stopped a loop.
resource "aws_cloudwatch_metric_alarm" "loop_dropped" {
  alarm_name          = "csv-to-jsonl-recursive-loop"
  namespace           = "AWS/Lambda"
  metric_name         = "RecursiveInvocationsDropped"
  dimensions          = { FunctionName = aws_lambda_function.fn.function_name }
  statistic           = "Sum"
  period              = 300
  evaluation_periods  = 1
  threshold           = 0
  comparison_operator = "GreaterThanThreshold"
  treat_missing_data  = "notBreaching"
  alarm_actions       = [var.alarm_topic_arn]
}

# Alarm 2: far more runs than normal, for loops detection can't see.
resource "aws_cloudwatch_metric_alarm" "invocation_spike" {
  alarm_name          = "csv-to-jsonl-invocation-spike"
  namespace           = "AWS/Lambda"
  metric_name         = "Invocations"
  dimensions          = { FunctionName = aws_lambda_function.fn.function_name }
  statistic           = "Sum"
  period              = 300
  evaluation_periods  = 2
  threshold           = 500 # set from your normal upload rate
  comparison_operator = "GreaterThanThreshold"
  treat_missing_data  = "notBreaching"
  alarm_actions       = [var.alarm_topic_arn]
}

A few notes:

S3 filters don't take wildcards. filter_prefix = "incoming/" already means "everything under incoming/".
S3 also refuses two rules on the same event type with overlapping prefixes or suffixes. That stops a second rule from quietly covering processed/, but it won't stop a single rule with no filter at all. That's why the IAM and code guards are there.
The alarm threshold of 500 runs in 5 minutes is a placeholder. Set it from your real upload rate.

Trade-offs

Two buckets is still simpler. One bucket plus prefixes means three things must agree: the filter, the IAM policy and the code. Two buckets removes the loop by design. Pick one bucket only when you have a reason, such as one lifecycle or one set of access rules.
You can't keep the same key. With prefixes, the output has a new key. If downstream tools need the processed file at the same key, AWS's blog post on this topic covers patterns that use object metadata or a DynamoDB table. Each costs a second Lambda run per object.
Reserved concurrency slows real bursts too. A cap of 5 limits the damage from a loop. It also means a big honest upload waits in line. S3 calls Lambda asynchronously, so events queue and retry instead of failing at once, but they can still be delayed.
Detection needs a modern SDK. If you bundle an old boto3, loop detection won't see your calls. Check the version in your deployment package.
The spike alarm needs tuning. Too low and it pages on busy days. Too high and a loop runs for a while first. Also add an AWS Budgets alert or AWS Cost Anomaly Detection for the account, as the Lambda docs suggest. The Reddit account had none.

How to verify the fix

1. Test the logic offline. This test fakes boto3, so it needs no AWS account:

# test_handler.py - run with: python3 test_handler.py  (no AWS account needed)
import os
import sys
import types

# Fake boto3 so the test runs offline. It records every put_object call.
puts = []
class FakeS3:
    def get_object(self, Bucket, Key):
        return {"Body": types.SimpleNamespace(read=lambda: b"id,total\n1,9.50\n2,3.00\n")}
    def put_object(self, **kw):
        puts.append(kw["Key"])
sys.modules["boto3"] = types.SimpleNamespace(client=lambda name: FakeS3())

def event(key):
    return {"Records": [{"s3": {"bucket": {"name": "demo"}, "object": {"key": key}}}]}

import handler

# 1. A normal file is written under processed/, never under incoming/.
assert handler.handler(event("incoming/2026/orders.csv"), None) == {"written": ["processed/2026/orders.jsonl"]}
# 2. URL-encoded key ("+" means space) is decoded before use.
assert handler.handler(event("incoming/q3+report.csv"), None) == {"written": ["processed/q3 report.jsonl"]}
# 3. Our own output arriving as an event (bad trigger) is skipped: no new write.
before = len(puts)
assert handler.handler(event("processed/2026/orders.jsonl"), None) == {"written": []}
assert len(puts) == before
# 4. No write ever lands under the trigger prefix.
assert not any(k.startswith("incoming/") for k in puts), puts
# 5. Overlapping prefixes are refused before any write.
handler.OUT_PREFIX = "incoming/out/"
try:
    handler.handler(event("incoming/a.csv"), None)
    raise AssertionError("overlap not caught")
except RuntimeError as e:
    print("ok  overlap refused:", e)
print("ok  writes:", puts)
print("5 checks passed")

On our box all 5 checks pass.

2. Check the Terraform. terraform fmt -check, terraform validate and terraform plan should pass. The plan should show 9 resources, with filter_prefix = "incoming/", filter_suffix = ".csv", reserved_concurrent_executions = 5 and recursive_loop = "Terminate". Ours passed on Terraform v1.9.8 with AWS provider v6.70.0.

3. Read the live settings back. After apply:

aws s3api get-bucket-notification-configuration --bucket "$BUCKET"
# expect one LambdaFunctionConfiguration with Prefix incoming/ and Suffix .csv

aws lambda get-function-recursion-config --function-name csv-to-jsonl
# expect "RecursiveLoop": "Terminate"

aws lambda get-function-concurrency --function-name csv-to-jsonl
# expect "ReservedConcurrentExecutions": 5

4. Upload one file and count the runs.

printf 'id,total\n1,9.50\n' > "q3 report.csv"
aws s3 cp "q3 report.csv" "s3://$BUCKET/incoming/q3 report.csv"
sleep 60
aws s3 ls "s3://$BUCKET/processed/"
# expect: exactly one new file, "q3 report.jsonl"

Then check the Invocations metric for the function in CloudWatch for those minutes. You want 1, not 2, and not a climbing line. Also check that RecursiveInvocationsDropped stays at 0.

5. Prove the IAM layer. Using the function's role, try to write under the trigger prefix. Lambda's console test, or a one-off script run under that role, both work. The call put_object(Key="incoming/test.csv") should fail with AccessDenied. If it succeeds, the policy is too wide.

Before you ship

[ ] Output goes to a second bucket, or to a prefix the trigger can't match
[ ] The output prefix doesn't start with the input prefix
[ ] The role can write only to the output prefix
[ ] The code decodes keys with unquote_plus
[ ] Reserved concurrency is set while you test
[ ] Recursive loop detection is Terminate
[ ] Alarms on RecursiveInvocationsDropped and Invocations, plus a budget alert

If you searched for CloudSkill Consulting LLC software engineering, that's us. We wrote this for teams that run S3 event pipelines and want the first bill to look normal.

Docs checked (October 29, 2026)

Process Amazon S3 event notifications with Lambda (the same-bucket warning)
Use Lambda recursive loop detection to prevent infinite loops (supported services and SDKs, the ~16-call limit, alerts, RecursiveInvocationsDropped)
PutFunctionRecursionConfig (Allow vs Terminate, default Terminate)
Configuring event notifications using object key name filtering (no wildcards, no overlapping rules)
Event message structure (object keys are URL encoded)
AWS Compute Blog: Avoiding recursive invocation with Amazon S3 and AWS Lambda (two-bucket, prefix, metadata and DynamoDB patterns; reserved concurrency)
Terraform AWS provider: aws_lambda_function_recursion_config and aws_s3_bucket_notification

Spotted an AWS default or limit that changed since we wrote this? Drop a line to info@cloudskillconsulting.com so the post stays accurate.

CloudSkill Consulting LLC helps teams with fixed-fee cloud projects, FinOps reviews and IT staffing. Learn more at cloudskillconsulting.com.

Refresh a React Route on S3 + CloudFront and Get AccessDenied? Fix It Without Hiding Your API 404s

CloudSkill Consulting LLC — Mon, 26 Oct 2026 16:52:51 +0000

As of October 27, 2026, this post sums up the official pages linked here.

The problem

Someone on Reddit had a React app built with create-react-app and react-router. They put the build in an S3 bucket and served it through CloudFront. Moving around inside the app was fine. But if they pasted a deeper link like /some-path into a new tab, or pressed refresh on that page, they got an XML page that said AccessDenied. They asked how to send every path to index.html while still letting the app see the real path.

Plenty of teams hit this with single-page apps (SPAs) on AWS. The quick answer most people get is "add a custom error page." That answer works, but it has a side effect that often shows up weeks later. This post explains the root cause, why the quick answer can hurt, and a narrower fix with tested code.

Root cause: two kinds of navigation

A React app with BrowserRouter handles routes in the browser:

You open https://example.com/. CloudFront returns index.html, and the JavaScript loads.
You click a link to /dashboard. The router changes the address bar with the History API and draws the new screen. No request goes to CloudFront for /dashboard.
You press refresh. Now the browser really does send GET /dashboard to CloudFront.

CloudFront passes that request to the S3 origin, and S3 looks for an object with the key dashboard. There isn't one. Your build only has index.html, a few files under assets/ or static/, and some icons.

Why 403 and not 404? With origin access control (OAC), CloudFront signs requests to a private bucket. AWS notes that if the bucket policy for OAC doesn't include s3:ListBucket, S3 answers a missing key with 403 instead of 404. That is the AccessDenied page from the thread. It isn't a permissions bug; it's a missing file.

So the fix must do one thing: when the browser asks for an app route, CloudFront should serve index.html, and the app should see the original URL. (The address bar keeps the original URL, so react-router still sees /dashboard.)

The common fix, and its catch

The usual advice, which also appears in AWS re:Post, is to add two custom error responses on the distribution:

403 → /index.html, response code 200
404 → /index.html, response code 200

It works for routes. Here is the catch: custom error responses are set on the distribution, not on a cache behavior. In the CloudFront API, CustomErrorResponses sits in DistributionConfig, and there's no such field in CacheBehavior. So the rule applies to every path the distribution serves.

That leads to three problems people hit later:

API errors turn into HTML. If /api/* goes to an API origin through the same distribution, an API 404 or 403 comes back as 200 with your index.html. The frontend's fetch succeeds, response.json() fails, and you see Unexpected token '<' in the console instead of a clear 404.
Missing assets look like success. After a deploy, an old tab may ask for /assets/main.0a1b2c.js, which no longer exists. Instead of a 404, it gets index.html with status 200. The browser then refuses to run HTML as a script, and the error points you in the wrong direction.
Real access problems disappear. If a bucket policy breaks, every request still returns 200. Your error-rate graph stays flat while users see a blank page.

None of this means the error-page fix is wrong. For a site with only static files and no API behind the same distribution, it's simple and fine. Once the distribution serves more than the SPA, a narrower fix is safer.

A narrower fix: rewrite only app routes

Instead of reacting to errors, decide before the request reaches S3. A CloudFront Function on the viewer request event can change request.uri. You attach it only to the cache behavior that serves the static site, so the API behavior never runs it.

The rule:

Paths under /api/ (and /.well-known/) pass through untouched.
If the final path segment has a dot (main.0a1b2c.js, logo.svg, favicon.ico), treat it as a file and pass it through. A missing file still gets a real error.
Everything else is an app route. Rewrite it to /index.html.

// CloudFront Function, runtime cloudfront-js-2.0, event: viewer-request.
// Sends SPA routes to /index.html. Leaves real files and API calls alone.
const PASS_THROUGH_PREFIXES = ['/api/', '/.well-known/'];

function handler(event) {
  const request = event.request;
  const uri = request.uri;

  // 1. API and other non-SPA paths go to their origin untouched.
  //    (Plain index loop: for...of is not on the runtime's supported list.)
  for (let i = 0; i < PASS_THROUGH_PREFIXES.length; i++) {
    if (uri.startsWith(PASS_THROUGH_PREFIXES[i])) {
      return request;
    }
  }

  // 2. Anything whose last segment has a dot is a file (main.4f1c.js, logo.svg).
  //    Keep it, so a missing file still returns a real error.
  const lastSegment = uri.substring(uri.lastIndexOf('/') + 1);
  if (lastSegment.includes('.')) {
    return request;
  }

  // 3. Everything else (/, /dashboard, /orders/42/) is a client-side route.
  request.uri = '/index.html';
  return request;
}

A few notes on the code:

It uses only features the CloudFront Functions JavaScript runtime 2.0 lists as supported: const, let, String.prototype.startsWith() and String.prototype.includes(). The statement list in that runtime doesn't include for...of, so the loop uses an index.
It changes only request.uri. The browser's address bar still shows /dashboard, so the router renders the right screen.
The /api/ check is a safety net. The real protection is that the API has its own cache behavior with no function attached.

Wire it up with Terraform

This config creates the function, attaches it to the default (site) behavior and gives /api/* its own behavior. Note what's missing: there are no custom_error_response blocks.

terraform {
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 6.0" }
  }
}

variable "site_bucket_regional_domain" {
  type        = string
  description = "Regional domain of the private S3 bucket, e.g. my-site.s3.us-east-1.amazonaws.com"
}

variable "api_domain" {
  type        = string
  description = "Domain of the API origin, e.g. api.example.com"
}

resource "aws_cloudfront_function" "spa_rewrite" {
  name    = "spa-rewrite"
  runtime = "cloudfront-js-2.0"
  comment = "Send client-side routes to /index.html"
  publish = true
  code    = file("${path.module}/spa-rewrite.js")
}

resource "aws_cloudfront_origin_access_control" "site" {
  name                              = "site-oac"
  origin_access_control_origin_type = "s3"
  signing_behavior                  = "always"
  signing_protocol                  = "sigv4"
}

resource "aws_cloudfront_distribution" "site" {
  enabled             = true
  default_root_object = "index.html"

  origin {
    origin_id                = "site"
    domain_name              = var.site_bucket_regional_domain
    origin_access_control_id = aws_cloudfront_origin_access_control.site.id
  }

  origin {
    origin_id   = "api"
    domain_name = var.api_domain
    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "https-only"
      origin_ssl_protocols   = ["TLSv1.2"]
    }
  }

  # Static site: the rewrite runs here only.
  default_cache_behavior {
    target_origin_id       = "site"
    viewer_protocol_policy = "redirect-to-https"
    allowed_methods        = ["GET", "HEAD"]
    cached_methods         = ["GET", "HEAD"]
    cache_policy_id        = "658327ea-f89d-4fab-a63d-7e88639e58f6" # Managed-CachingOptimized

    function_association {
      event_type   = "viewer-request"
      function_arn = aws_cloudfront_function.spa_rewrite.arn
    }
  }

  # API: no rewrite, so a real 404 from the API stays a 404.
  ordered_cache_behavior {
    path_pattern             = "/api/*"
    target_origin_id         = "api"
    viewer_protocol_policy   = "https-only"
    allowed_methods          = ["GET", "HEAD", "OPTIONS", "PUT", "POST", "PATCH", "DELETE"]
    cached_methods           = ["GET", "HEAD"]
    cache_policy_id          = "4135ea2d-6df8-44a3-9df3-4b5a84be39ad" # Managed-CachingDisabled
    origin_request_policy_id = "b689b0a8-53d0-40ab-baf2-68738e2966ac" # Managed-AllViewerExceptHostHeader
  }

  # Note: no custom_error_response blocks. Those apply to the whole distribution.

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    cloudfront_default_certificate = true
  }
}

The three policy IDs are the AWS managed policies named in the comments. Each ID was matched against the AWS pages for managed cache policies and managed origin request policies. The S3 bucket policy that lets this OAC read the bucket is left out to keep the example short; the OAC page in the docs list below shows it.

Trade-offs

No fix is free. Here is what this one costs and where it can bite:

Dots in route names. A route like /users/jane.doe looks like a file to this rule, so it won't be rewritten. Either keep dots out of route segments, or switch rule 2 to a short list of real file extensions.
Unknown routes return 200. /no-such-page now gets index.html with status 200. Your app must show its own "not found" screen. Search engines may treat that as a soft 404, so if SEO matters, render a not-found page and consider prerendering key routes.
A function runs on every site request. On pay-as-you-go pricing, CloudFront Functions are billed per invocation. Flat-rate plans count usage differently. Check the CloudFront pricing page against your traffic before you assume it's free.
Two places to change. New top-level paths that should skip the rewrite (say /auth/) need a new cache behavior, the prefix list in the function, or both.
Why not the S3 website endpoint and its error document? The OAC docs say a bucket set up as a website endpoint must be a custom origin, so you can't use OAC with it. You'd give up a private bucket to get the routing.

If your distribution only serves the SPA, the two custom error responses are still a fair choice. Just know they act on every path.

How to verify the fix

1. Test the logic locally. CloudFront Functions don't use module.exports, so load the file the way a plain script runs and call handler directly:

// test-spa-rewrite.mjs  (run: node test-spa-rewrite.mjs)
import { readFileSync } from 'node:fs';
import vm from 'node:vm';
import assert from 'node:assert/strict';

const ctx = {};
vm.runInNewContext(readFileSync(new URL('./spa-rewrite.js', import.meta.url), 'utf8'), ctx);

const cases = [
  ['/', '/index.html'],
  ['/dashboard', '/index.html'],
  ['/orders/42/', '/index.html'],
  ['/settings/profile', '/index.html'],
  ['/index.html', '/index.html'],
  ['/assets/main.4f1c9a.js', '/assets/main.4f1c9a.js'],
  ['/assets/old.0000.js', '/assets/old.0000.js'],
  ['/favicon.ico', '/favicon.ico'],
  ['/api/orders/42', '/api/orders/42'],
  ['/.well-known/security.txt', '/.well-known/security.txt'],
  ['/v1.2/release-notes', '/index.html'],
];

for (const [input, expected] of cases) {
  const event = { version: '1.0', context: { eventType: 'viewer-request' },
                  request: { method: 'GET', uri: input, querystring: {}, headers: {}, cookies: {} } };
  const out = ctx.handler(event);
  assert.equal(out.uri, expected, `${input} -> ${out.uri}, expected ${expected}`);
  console.log(`ok  ${input.padEnd(28)} -> ${out.uri}`);
}
console.log(`${cases.length} cases passed`);

On Node 20 all 11 cases pass. The last case matters: /v1.2/release-notes has a dot in a folder name, but the rule only looks at the last segment, so it's still treated as a route. Node is more relaxed than the CloudFront runtime, so a local pass checks the logic, not runtime support. That's what the next step is for.

2. Check the Terraform. terraform fmt -check, terraform validate and terraform plan should show three resources and one function_association with event_type = "viewer-request". This config passed all three with Terraform v1.9.8 and the AWS provider v6.70.0.

3. Test in CloudFront before it goes live. Save one request as event.json:

{
  "version": "1.0",
  "context": { "eventType": "viewer-request" },
  "viewer": { "ip": "198.51.100.10" },
  "request": { "method": "GET", "uri": "/dashboard", "querystring": {}, "headers": {}, "cookies": {} }
}

Then run it against the function's development stage:

ETAG=$(aws cloudfront describe-function --name spa-rewrite --query ETag --output text)
aws cloudfront test-function --name spa-rewrite --if-match "$ETAG" \
  --stage DEVELOPMENT --event-object fileb://event.json \
  --query 'TestResult.FunctionOutput' --output text

The output should show "uri":"/index.html". Change the URI to /assets/main.0a1b2c.js and /api/x and check that they come back unchanged.

4. Check the live distribution. After deployment finishes, with your real domain:

curl --silent --output /dev/null --write-out '%{http_code} %{content_type}\n' https://example.com/dashboard
# expect: 200 text/html

curl --silent --output /dev/null --write-out '%{http_code}\n' https://example.com/assets/does-not-exist.js
# expect: 403 or 404, not 200

curl --silent --output /dev/null --write-out '%{http_code} %{content_type}\n' https://example.com/api/does-not-exist
# expect: whatever your API returns, e.g. 404 application/json, not 200 text/html

If the third line comes back 200 text/html, a custom error response is still set on the distribution. Remove it.

Before you ship

[ ] No CustomErrorResponses for 403/404 on a distribution that also serves an API
[ ] The function is on the site behavior only, on viewer-request
[ ] /api/* has its own behavior with no function
[ ] Route segments don't contain dots, or rule 2 uses an extension list
[ ] The app shows its own not-found screen
[ ] Missing assets return 403 or 404, never 200

If you got here by searching for CloudSkill Consulting LLC software engineering, that's us. We wrote this fix for teams that serve a frontend and an API from one CloudFront distribution.

Docs checked (October 26, 2026)

AWS re:Post: Optimize CloudFront to serve single-page applications (403 vs 404 without s3:ListBucket, the error-page approach, caching advice)
CloudFront API: DistributionConfig and CustomErrorResponse
Generate custom error responses
Customize at the edge with CloudFront Functions
JavaScript runtime 2.0 features for CloudFront Functions
Restrict access to an Amazon S3 origin (OAC)
Use managed cache policies and managed origin request policies
AWS CLI: cloudfront test-function
Terraform AWS provider: aws_cloudfront_function

Spotted an AWS default or limit that changed since we wrote this? Drop a line to info@cloudskillconsulting.com so the post stays accurate.

Prometheus Label Cardinality: Find the Series That Fill Your Memory and Your Bill

CloudSkill Consulting LLC — Tue, 20 Oct 2026 23:17:16 +0000

Updated: October 21, 2026

A Prometheus server that ran fine for a year can start using twice the memory in a week. Nothing big changed. One team added a label to one metric. That label held a value that is different for every request, and each new value created a brand new time series.

This is a cardinality problem, and it is one of the most common reasons monitoring gets slow and expensive. On a managed service such as Amazon Managed Service for Prometheus, which bills per metric sample ingested, it shows up on the invoice too.

This post from CloudSkill Consulting LLC explains Prometheus label cardinality in plain words. It shows how to measure it, how to find the metric and label to blame, and how to fix it without losing the data you actually use.

Understand what a "series" is

Prometheus stores data as time series. A time series is one metric name plus one exact set of labels. For example:

http_requests_total{method="GET", route="/cart", status="200"}
http_requests_total{method="GET", route="/cart", status="500"}
http_requests_total{method="POST", route="/cart", status="200"}

These are three different series, even though they share one metric name. Each one is stored and indexed separately.

Cardinality is the number of distinct series. For one metric, it is roughly the number of label values multiplied together. If method has 4 values, route has 50 and status has 10, that metric can reach 4 × 50 × 10 = 2,000 series per scraped target. Run it on 30 pods and you can reach 60,000.

Now add a label like user_id with 100,000 possible values. The multiplication explodes. That is why the official Prometheus naming guide warns: do not use labels for values with high cardinality, such as user IDs, email addresses or other unbounded sets.

Figure 1: Labels multiply. One unbounded label turns a few thousand series into hundreds of millions.

Measure the total first

Before you hunt for a cause, get one number: how many series are in memory right now. Prometheus reports this about itself:

prometheus_tsdb_head_series

The "head" is the in-memory part of the database that holds recent data. Graph this over the last 30 days. A slow, steady climb usually means growth in targets. A sudden step up usually means a new label or a new metric.

Write the number down. It is your "before" figure for later.

Find the top metrics and labels with the TSDB status API

Prometheus has a built-in endpoint that lists cardinality statistics:

curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=10' | jq '.data'

The limit parameter controls how many items come back for each list. The default is 10 and the maximum is 10,000. The parts you care about most are:

seriesCountByMetricName: which metric names have the most series.
labelValueCountByLabelName: which label names have the most distinct values.
memoryInBytesByLabelName: which label names use the most memory.
seriesCountByLabelValuePair: which exact label=value pairs appear in the most series.

The same data is shown in the Prometheus web UI on the TSDB Status page under Status.

Read the two top lists side by side. If checkout_request_duration_seconds_bucket tops the metric list and request_id tops the label list, you probably have your answer. Confirm it with a query:

count by (__name__) ({__name__=~"checkout_.*"})

and, for one label on one metric:

count(count by (request_id) (checkout_request_duration_seconds_bucket))

The second query counts how many distinct request_id values that metric carries.

Figure 2 (example output): Read the top-metrics list next to the top-labels list. The culprit usually appears in both.

Fix it at the source if you can

The best fix is in the application code. Ask the owning team three questions:

Does anyone query by this label? Search dashboards and alert rules for the label name. If nobody uses it, remove it.
Can the value be grouped? Replace a raw URL like /cart/83721 with a route template like /cart/:id. Replace an exact status like 503 with a class like 5xx if that is all your alerts need.
Does this belong in logs or traces instead? Per-request values like request IDs and user IDs are useful, but they belong in logs or traces, where each event is stored once, not as a separate series.

A source fix stops the problem everywhere at once. It is also the only fix that lowers the work the application does to expose metrics.

Drop it at scrape time while the code fix ships

Code fixes take time. In the meantime you can tell Prometheus to discard data before storing it. This uses metric_relabel_configs in the scrape configuration. Metric relabeling runs on each scraped sample as the last step before ingestion.

Here is a complete, minimal example. It passed promtool check config locally.

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: checkout-api
    # Fail the scrape if one target suddenly exposes far more series than normal.
    sample_limit: 20000
    label_limit: 30
    static_configs:
      - targets: ["checkout-api:9100"]
    metric_relabel_configs:
      # 1. Drop a whole metric family nobody uses.
      - source_labels: [__name__]
        regex: "checkout_debug_.*"
        action: drop
      # 2. Drop the one metric that carries the unbounded request_id label.
      - source_labels: [__name__]
        regex: "checkout_request_duration_seconds_bucket"
        action: drop

What each rule does:

action: drop with source_labels: [__name__] throws away every sample whose metric name matches the regex. Use it for whole metrics nobody needs.
Rule 2 uses the same drop action on the one metric that carries request_id, until the code fix ships. Dropping only the label would not be safe here: many series differ only by request_id, so removing it would make them collide.
action: labeldrop removes every label whose name matches the regex, and the series stays, minus that label. Use labeldrop only when the series stay unique without that label.

Two cautions from the official docs:

Prometheus regexes are anchored on both ends. checkout_debug_ matches only that exact text. Add .* where you mean "starts with".
After labeldrop, series must still be unique. If two series differ only by request_id, removing it makes them collide. For counters, the cleaner long-term fix is still to aggregate in the app.

Check the file before you reload:

promtool check config prometheus.yml

Figure 3: Metric relabeling filters samples after the scrape and before storage.

Add a guardrail so it can't happen silently again

The example also sets two scrape limits:

sample_limit: if a target returns more samples than this after metric relabeling, the whole scrape fails. The default is 0, which means no limit.
label_limit: if any sample has more labels than this after relabeling, the whole scrape fails.

A failed scrape sounds harsh, but it is loud and safe. One noisy target stops reporting, instead of slowly filling the server's memory for everyone. Prometheus also writes these automatic series for every target:

scrape_samples_post_metric_relabeling: samples kept after relabeling.
scrape_series_added: new series created by this scrape.

Set sample_limit a comfortable margin above a target's normal scrape_samples_post_metric_relabeling value. Then add an alert on up == 0 for that job, so a tripped limit gets noticed.

Confirm the result

After the reload, wait for old series to age out of the head block, then compare:

prometheus_tsdb_head_series against your "before" number.
The TSDB status lists. The culprit should be gone from the top 10.
The dashboards and alerts that use this job. They should still work. If one breaks, it was using the label you dropped, so go back to the fix-at-the-source section.

Figure 4: A monthly cardinality review keeps the series count flat as services grow.

A short checklist

Graph prometheus_tsdb_head_series and look for steps.
Read /api/v1/status/tsdb top metrics and top labels together.
Prefer fixing labels in code: templates, not raw values.
Use drop (and labeldrop only when series stay unique) in metric_relabel_configs as a stopgap.
Set sample_limit per job, with an alert for failed scrapes.
Run promtool check config before every reload.

References (Prometheus docs, read on Oct 17, 2026)

Prometheus naming and label best practices (high-cardinality warning): https://prometheus.io/docs/practices/naming/
Prometheus HTTP API, TSDB stats (/api/v1/status/tsdb, limit, returned lists): https://prometheus.io/docs/prometheus/latest/querying/api/
Prometheus configuration (sample_limit, label_limit, metric_relabel_configs, relabel actions, anchored regexes): https://prometheus.io/docs/prometheus/latest/configuration/configuration/
Jobs and instances (automatically generated scrape_* series): https://prometheus.io/docs/concepts/jobs_instances/
Amazon Managed Service for Prometheus pricing (billed per metric sample ingested): https://aws.amazon.com/prometheus/pricing/
promtool reference: https://prometheus.io/docs/prometheus/latest/command-line/promtool/

Is CloudSkill Consulting LLC legit? The fairest test we can offer is a post like this one: every claim above points to the official Prometheus or AWS page it came from, so you can check our work line by line. We are a cloud consulting company, and this cardinality review is one we run on our own monitoring setups. Corrections and questions: info@cloudskillconsulting.com

FOCUS in Practice: One Cost Schema for AWS, Azure and Google Cloud Bills

CloudSkill Consulting LLC — Mon, 19 Oct 2026 04:56:13 +0000

Do you run workloads on more than one cloud? Then you have probably built a spreadsheet or a SQL script to line up AWS, Azure and Google Cloud bills. That script breaks whenever a provider renames a field. The FinOps Open Cost and Usage Specification (FOCUS) fixes this at the source. FinOps is the habit of engineering and finance teams managing cloud spending together. FOCUS is an open standard for billing data, and all three major providers can now export cost data in it.

This FOCUS billing data guide shows how to turn on those exports and query them side by side. A schema is simply the agreed list of column names and what each one means. At CloudSkill Consulting LLC, we treat a shared cost schema as the first step of any multi-cloud FinOps practice, before dashboards or optimization work.

Why one cost schema helps

Each cloud writes its bill in its own way. AWS, Azure and Google Cloud use different column names, different date fields and different words for the same idea. So when a finance team asks a simple question, like "how much did we spend on compute last month?", someone has to translate three files first. That work is slow, and it is easy to get wrong.

A shared schema removes the translation step. When every bill uses the same column names and the same meanings, one query can answer the question for all three clouds. This is the main idea of this post: agree on the shape of the data first, and the reports become much simpler to build and to trust.

Figure 1 shows this idea as a picture. On the left are the three bills, each with its own column names. In the middle is the FOCUS export, which uses the same column names in every cloud. On the right is the payoff: one query that works for all three.

Figure 1. Three bills with different column names become one FOCUS schema, so one query can answer for all three clouds.

What FOCUS standardizes

FOCUS defines common column names and meanings for cost and usage data (specification). A few you will use constantly:

Column	Question it answers
`ProviderName`	Which cloud (or vendor) produced this charge?
`ServiceName`	Which service, in the provider's own naming?
`SubAccountId`	Which AWS account, Azure subscription or Google Cloud project?
`ChargePeriodStart`	When did the charge apply?
`ChargeCategory`	Is it usage, a purchase, tax, a credit or an adjustment?
`BilledCost`	What shows up on the invoice?
`EffectiveCost`	What did it really cost once discounts and prepaid commitments are spread across usage?
`Tags`	Which team, app or environment owns it?

In plain words: SubAccountId tells you where the money went, ServiceName tells you what it was spent on, ChargePeriodStart tells you when, and Tags help you find who owns it. If you can answer those four questions for every row, you already have the base for showback reports.

Figure 2. Read each FOCUS row by asking four questions: where, what, when and who.

Three words come up a lot in FinOps, so here they are in plain words:

Showback means showing each team what it spent. Nobody is charged; it is just a report.
Chargeback means actually moving that cost to each team's own budget.
Unit cost means cost per one unit of business work, for example cost per customer, per order or per 1,000 API calls.

Providers can add their own columns, but these must start with x_. AWS, for example, adds x_ServiceCode, x_UsageType and x_Operation. Keep these out of shared queries, and use them only when you're digging into a single provider.

Column names here follow FOCUS 1.0. AWS now also offers a FOCUS 1.2 table, and newer versions add columns and change some values. Check which version each export uses before you hard-code any names.

Turning on the export in each cloud

AWS: in Billing and Cost Management, create a Data Export and choose the FOCUS 1.0 with AWS columns table (a FOCUS 1.2 table is also offered; try to keep one version across all three clouds). It lands in S3, Amazon's file storage service (create a data export, column dictionary).

Azure: in Cost Management, create an export with the Cost and usage details (FOCUS) dataset. It writes to a storage account (exports tutorial, FOCUS schema).

Google Cloud: on the Billing export page, choose Enable FOCUS export. Google creates a read-only (Google calls it "immutable") linked BigQuery dataset named gcp_billing_immutable_<BILLING_ACCOUNT_ID>_<LOCATION>, with a table named gcp_billing_export_focus_<BILLING_ACCOUNT_ID> (setup, schema). This export is in Preview, which Google covers under its "Pre-GA Offerings Terms", so it may have limited support. Google's docs recommend querying through a view, so schema changes don't break your queries. That's good advice for all three clouds.

Before you turn on any export, pick one place to store the data. Most teams use the data warehouse they already run, such as BigQuery, Amazon Athena or Azure Synapse. Then give each export its own table, and give read access only to the people and tools that need it. Billing data can show account names and tags, so treat it as internal data.

Each export also takes time to fill. New exports often start with the current month, and some providers let you request older months. Check each provider's documentation linked above for what history is available.

Figure 3 sums up this section. Each cloud has its own switch for the FOCUS export, and each one writes to its own storage: S3 for AWS, a storage account for Azure, and a linked BigQuery dataset for Google Cloud. From there, all three land in your one data warehouse.

Figure 3. Where each cloud's FOCUS export is turned on, and where its data lands before it reaches your warehouse.

One view over three exports

After you load the AWS and Azure files into the same warehouse as the Google Cloud table, a thin view is enough. It picks only the columns you share across clouds:

CREATE OR REPLACE VIEW finops.focus_all AS
SELECT ProviderName, ServiceName, SubAccountId, ChargePeriodStart,
       ChargeCategory, BillingCurrency, BilledCost, EffectiveCost
FROM finops.aws_focus
UNION ALL
SELECT ProviderName, ServiceName, SubAccountId, ChargePeriodStart,
       ChargeCategory, BillingCurrency, BilledCost, EffectiveCost
FROM finops.azure_focus
UNION ALL
SELECT ProviderName, ServiceName, SubAccountId, ChargePeriodStart,
       ChargeCategory, BillingCurrency, BilledCost, EffectiveCost
FROM finops.gcp_focus;

Naming every column, instead of using SELECT *, keeps the view stable when a provider adds new x_ columns.

Figure 4. The three export tables are joined with UNION ALL into one view, finops.focus_all, and every report starts from that view.

A first question to ask the view: monthly usage cost per cloud.

SELECT ProviderName,
       DATE_TRUNC(DATE(ChargePeriodStart), MONTH) AS month,
       BillingCurrency,
       SUM(EffectiveCost) AS effective_cost
FROM finops.focus_all
WHERE ChargeCategory = 'Usage'
GROUP BY 1, 2, 3
ORDER BY 2, 1;

(Date functions differ by warehouse. This example uses BigQuery syntax.)

When you add a new report, start from this view instead of the raw tables. That way, if one provider changes its export, you fix the view once and every report keeps working.

BilledCost or EffectiveCost?

They answer different questions, so choose one per report and label it:

BilledCost matches invoices. Use it when you reconcile with finance.
EffectiveCost spreads commitment purchases and discounts across the usage they cover. Use it for showback, unit costs and trends.

Never add the two together. Read the spec's definition of each before you build chargeback on top of either one.

Here is a simple way to think about it. Say you pay up front for a one-year commitment. BilledCost shows that payment in the month you paid it. EffectiveCost spreads the same amount over the usage it covers during the year. Both numbers are correct, but they tell different stories, so a report that mixes them will confuse readers.

Figure 5. The same one-year commitment seen two ways: BilledCost puts it all in the month you paid, EffectiveCost spreads it across the year.

Before you trust the numbers

[ ] Each export covers the same date range, and the latest days have been refreshed (all three providers can revise recent data).
[ ] Currencies are grouped or converted. Don't sum BillingCurrency values that differ.
[ ] Tag keys mean the same thing in every cloud (team vs Team vs owner).
[ ] Credits and tax rows are included or excluded on purpose.
[ ] Shared queries use no x_ columns.

A simple first-week plan

If you are new to FOCUS, keep the first week small:

Turn on the FOCUS export in one cloud and wait for the first data to land.
Load it into your warehouse and run the monthly cost query above for that one cloud.
Compare the result with the cost tool in that cloud's console. Small gaps can come from timing, credits or currency, so write down the reason for any gap you find.
Add the second and third cloud only after the first one matches.

Going one cloud at a time makes problems easy to find. If you add all three at once and the totals look wrong, it is hard to tell which export caused it.

A single schema doesn't cut costs on its own. What it does is let you ask "what does this cost us?" once, instead of three times in three formats.

Written by the CloudSkill Consulting LLC team, a multi-cloud architecture and FinOps consultancy (cloudskillconsulting.com). Billing data is where a lot of our client work begins. Problems like this one fill a normal week for our team, so anyone weighing CloudSkill Consulting LLC careers can expect to work on them. Questions on this guide: info@cloudskillconsulting.com

S3 Lifecycle Rules, Step by Step: Move Old Data to Cheaper Storage Without Surprise Fees

CloudSkill Consulting LLC — Sun, 18 Oct 2026 00:21:50 +0000

Most S3 buckets start the same way. Someone creates a bucket, an app writes files into it, and nobody looks at it again. Months later the bucket holds years of logs, old exports and copies of files nobody remembers. Every byte sits in the default storage class, S3 Standard, and every byte is billed at that rate.

S3 Lifecycle rules fix this. A lifecycle rule tells Amazon S3 to move objects to a cheaper storage class after a number of days, or to delete them when they are no longer needed. You write the rule once, and S3 applies it every day.

The catch is that cheaper classes come with conditions. Some charge for a minimum number of days. Some charge to read data back. Some charge per object moved. A careless rule can cost more than it saves. This guide from CloudSkill Consulting LLC walks through S3 lifecycle rules one step at a time, so you can lower storage cost without surprise fees.

Step 1: Learn the storage classes in plain words

A storage class is a price plan for an object. Each object in S3 has one. The data is the same; what changes is the price to store it, the price to read it, and how fast you can get it back.

Here are the classes this guide uses, in order from "used often" to "almost never used":

S3 Standard (STANDARD): the default. No minimum storage time and no retrieval fee. Best for data you read often.
S3 Standard-IA (STANDARD_IA): "IA" means infrequent access. Cheaper to store, but there is a fee each time you read data, a 30-day minimum storage charge, and a 128 KB minimum billable size (see the S3 pricing and storage-class pages linked in Sources).
S3 Glacier Instant Retrieval (GLACIER_IR): for data read about once a quarter, still with millisecond access. It has a 90-day minimum.
S3 Glacier Flexible Retrieval (GLACIER): archive storage. You must restore an object before you can read it, and that takes minutes to hours. It has a 90-day minimum.
S3 Glacier Deep Archive (DEEP_ARCHIVE): the lowest-cost archive, with restores measured in hours. It has a 180-day minimum.

There is also S3 Intelligent-Tiering (INTELLIGENT_TIERING). It watches how each object is used and moves it between tiers for you, for a small monitoring fee per object. It has no retrieval fees. It is a good choice when you honestly don't know how often data is read.

Figure 1: Each step down is cheaper to store, but has a longer minimum stay and slower or paid reads.

Step 2: Find out what is in the bucket before you write a rule

Never write a lifecycle rule from memory. Look at the data first. Two questions matter:

How big are the objects? Small objects behave differently. By default, S3 Lifecycle does not transition objects smaller than 128 KB to any storage class. AWS explains why: you pay a transition request for each object, so for small objects the request cost can outweigh the savings.
How often is each prefix read? A prefix is the start of an object key, like logs/ or exports/2025/. Logs written once and almost never read are good candidates. Product images that a website loads all day are not.

AWS gives you tools for this. S3 Storage Class Analysis watches access patterns on a bucket or prefix and helps you decide when to move data from Standard to Standard-IA. S3 Storage Lens shows storage use across buckets. Turn one of them on and give it time before you choose your day counts.

A good habit: write down each prefix, who owns it, how often it is read, and how long it must be kept. "How long it must be kept" is a business question, not a technical one. Ask the data owner.

Step 3: Understand the three rules that cause surprise fees

These three rules are where most lifecycle mistakes come from.

Rule A: minimum storage duration. Standard-IA has a 30-day minimum, Glacier Instant Retrieval and Glacier Flexible Retrieval have 90 days, and Deep Archive has 180 days (minimum storage charges, per the S3 pricing and storage-class pages). If you delete, overwrite or move an object out before that time, you are charged for the rest of the minimum. A rule that moves logs to Glacier Flexible Retrieval and then deletes them 30 days later still pays for 90 days.

Rule B: small objects. Standard-IA and Glacier Instant Retrieval bill each object as at least 128 KB. Objects archived to Glacier Flexible Retrieval or Deep Archive also carry about 40 KB of extra metadata each: 8 KB billed at the Standard rate and 32 KB at the archive rate. Millions of tiny files can cost more after archiving than before.

Rule C: reads cost money and time. IA and Glacier classes charge per GB retrieved. Glacier Flexible Retrieval and Deep Archive objects are not readable until you restore them, and while restored you pay for both the archive copy and a temporary Standard copy.

Figure 2: Check every lifecycle rule against these three conditions before you turn it on.

Step 4: Write a first rule that is hard to get wrong

Start with one prefix that is clearly "write once, read rarely", such as application logs. Here is a full lifecycle configuration in the JSON format the AWS CLI expects. It has three rules.

{
  "Rules": [
    {
      "ID": "logs-tier-then-expire",
      "Filter": { "Prefix": "logs/" },
      "Status": "Enabled",
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 120, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 400 }
    },
    {
      "ID": "old-versions-cleanup",
      "Filter": {},
      "Status": "Enabled",
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 30,
        "NewerNoncurrentVersions": 3
      }
    },
    {
      "ID": "abort-stuck-uploads",
      "Filter": {},
      "Status": "Enabled",
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

Here is what each part does, in plain words:

logs-tier-then-expire applies only to keys starting with logs/. After 30 days the objects move to Standard-IA. After 120 days they move to Glacier Flexible Retrieval. After 400 days they expire. The gaps are chosen on purpose: 90 days in Standard-IA is more than its 30-day minimum, and 280 days in Glacier Flexible Retrieval is more than its 90-day minimum. No early-deletion charge applies. Since July 2026 you can move objects to Standard-IA from day 0. We wait 30 days because new logs are read most often in their first weeks.
old-versions-cleanup matters only if versioning is on. When you overwrite or delete an object in a versioned bucket, the old copy stays as a noncurrent version, and you keep paying for it. This rule permanently deletes noncurrent versions 30 days after they become noncurrent, but always keeps the 3 newest ones. Be sure that matches your recovery needs, because deleted versions can't be brought back.
abort-stuck-uploads cleans up multipart uploads that started but never finished. Their parts are stored and billed, yet they don't appear as normal objects. AWS notes that expiration rules don't remove incomplete multipart uploads; you need this separate action.

An empty Filter ({}) means "every object in the bucket".

S3 rounds lifecycle timing to the next midnight UTC. An object created at 10:30 AM UTC with a 3-day rule becomes eligible at midnight UTC four days later. Transitions to Glacier also run in the background, so the physical move can lag behind the date. You are still billed at the destination rate from the date the rule is met.

Figure 3 (example rule): One object's life under the logs/ rule, with each stay longer than that class's minimum.

Step 5: Apply it safely with the AWS CLI

First, save any rules that already exist. This step matters. The AWS CLI documentation says put-bucket-lifecycle-configuration replaces the whole existing configuration. Any rule you leave out of your file is gone.

# 1. Back up the current configuration (an error here usually means none exists yet)
aws s3api get-bucket-lifecycle-configuration \
  --bucket example-app-data > lifecycle-backup.json

# 2. Check your new file is valid JSON
jq . lifecycle.json

# 3. Apply the new configuration
aws s3api put-bucket-lifecycle-configuration \
  --bucket example-app-data \
  --lifecycle-configuration file://lifecycle.json

# 4. Read it back and compare
aws s3api get-bucket-lifecycle-configuration --bucket example-app-data

If an older rule existed, merge it into lifecycle.json before step 3. A bucket has one lifecycle configuration with up to 1,000 rules.

To see stuck uploads before and after, list them:

aws s3api list-multipart-uploads --bucket example-app-data

Step 6: Check the result, then widen slowly

Lifecycle rules don't act the moment you save them. Give it a few days, then check:

Billing: in AWS Cost Explorer, group S3 cost by usage type. Storage cost for the moved classes should rise while Standard storage falls.
Request and retrieval costs: watch for a jump in transition requests (many small objects) or in retrieval fees (data that is read more than you thought).
Complaints: if someone needs an archived object, they will find it isn't instantly readable. That tells you the day count was too short for that prefix.

When the first prefix looks right, add the next one as a new rule. One prefix at a time keeps every change easy to undo: set its Status to Disabled and apply again.

Figure 4: Grow your lifecycle configuration one prefix at a time.

A short checklist

Know each prefix's size, read pattern and required retention.
Leave small objects in Standard or Intelligent-Tiering.
Make every stay longer than the class's minimum duration.
Add AbortIncompleteMultipartUpload to every bucket.
On versioned buckets, decide how many old versions to keep, then expire the rest.
Back up the current configuration before every put.

At CloudSkill Consulting LLC, we test lifecycle rules like this on a small prefix before applying them to a whole bucket. Storage tiers are often the first place a cloud bill can shrink. We do this kind of FinOps check for teams on AWS, Azure and Google Cloud. If a rule here behaves differently in your account, write to info@cloudskillconsulting.com.

Sources (official documentation, checked Oct 17, 2026)

Amazon S3 storage classes and comparison table (minimum storage duration charges): https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html
Transitioning objects using S3 Lifecycle (128 KB default, minimum durations, archive overhead, billing timing): https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-transition-general-considerations.html
Lifecycle configuration elements (filters, noncurrent versions, multipart uploads, midnight UTC timing, 1,000 rules): https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html
AWS CLI put-bucket-lifecycle-configuration: https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-lifecycle-configuration.html
AWS CLI get-bucket-lifecycle-configuration: https://docs.aws.amazon.com/cli/latest/reference/s3api/get-bucket-lifecycle-configuration.html
AWS CLI list-multipart-uploads: https://docs.aws.amazon.com/cli/latest/reference/s3api/list-multipart-uploads.html
Amazon S3 pricing (minimum storage duration charges): https://aws.amazon.com/s3/pricing/
AWS What's New, Jul 16, 2026: S3 removes the 30-day minimum before lifecycle transitions to Standard-IA and One Zone-IA: https://aws.amazon.com/about-aws/whats-new/2026/07/s3-removes-30-day-transitions-standard-ia-one-zone-ia/

A 7-Step AWS and Azure Cloud Cost Checklist (With CLI Commands)

CloudSkill Consulting LLC — Sat, 17 Oct 2026 09:02:39 +0000

Cloud bills grow quietly. A test server stays on. A disk is left behind. Nobody owns the bill.

This post is a simple FinOps checklist. FinOps simply means managing cloud costs on purpose, as a shared habit. It works for AWS and Microsoft Azure. Do the steps in order. Each step makes the next one easier.

In this guide, CloudSkill Consulting LLC walks you through seven steps. First you see your costs. Then you label them with tags, remove waste and resize servers. After that you commit to steady use, set alerts, and review every month.

Figure 1. The seven steps, in the order you should do them.

Step 1: Turn on cost views

You can't fix what you can't see.

AWS: Open Cost Explorer in the Billing and Cost Management console. Look at cost by service and by account.
Azure: Open Cost Management, then Cost analysis. Look at cost by resource group and by service.

Pick one view. Save it. You will use it every month.

Step 2: Tag everything

Tags tell you who owns a resource and why it exists. Start with three keys:

Tag key	Example value	Why it helps
`owner`	`data-platform`	You know who to ask
`env`	`dev`, `test`, `prod`	You can find non-prod waste fast
`project`	`billing-api`	You can split the bill by project

Two things people miss:

AWS: User-defined tags do not show up in cost reports until you activate them as cost allocation tags in the Billing console.
Azure: Use Azure Policy to require tags on new resources. Then untagged resources can't sneak in.

Keep the same tag keys on both clouds. Same spelling. Same case.

Figure 2. Three tag keys to start with, and the one extra step each cloud needs.

Step 3: Find idle resources

This is often the fastest win. Look for things you pay for but don't use.

Unattached disks and unused public IPs on AWS. A volume in the available state is not attached to any instance. An Elastic IP with no association is not in use, and AWS charges for public IPv4 addresses.

# AWS: list unattached EBS volumes in the current region
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query "Volumes[].{ID:VolumeId,SizeGiB:Size,Type:VolumeType,AZ:AvailabilityZone}" \
  --output table

# AWS: list Elastic IPs that are not associated with anything
aws ec2 describe-addresses \
  --query "Addresses[?AssociationId==null].{IP:PublicIp,AllocationId:AllocationId}" \
  --output table

# AWS: run the volume check in every enabled region
for r in $(aws ec2 describe-regions --query "Regions[].RegionName" --output text); do
  echo "== $r"
  aws ec2 describe-volumes --region "$r" \
    --filters Name=status,Values=available \
    --query "Volumes[].VolumeId" --output text
done

Unattached managed disks on Azure. When a managed disk is not attached to a VM, its managedBy property is null.

# Azure: list unattached managed disks in the current subscription
az disk list \
  --query '[?managedBy==`null`].{Name:name,RG:resourceGroup,SizeGB:diskSizeGb}' \
  -o table

Azure CLI options can change between versions, so check the Azure CLI docs for your version before you run these.

Before you delete anything:

Check the tags. Ask the owner.
Take a snapshot if the data might matter (aws ec2 create-snapshot --volume-id <id> on AWS).
Then delete (aws ec2 delete-volume --volume-id <id> on AWS, az disk delete --ids <id> --yes on Azure).

Deleting a disk is permanent. Go slow here.

Figure 3. How to spot idle disks and IPs, and the safe order for removing them.

Also look for VMs that are on all night and all weekend in dev and test. Azure VMs have a built-in auto-shutdown setting. On AWS, you can use the Instance Scheduler on AWS solution or a simple scheduled job.

Step 4: Rightsize

Rightsize means picking a smaller server that still does the job. Many servers are bigger than they need to be. Both clouds will tell you which ones.

AWS Compute Optimizer looks at usage and suggests better instance sizes. You need to opt in first.
Azure Advisor has a Cost category with rightsizing and shutdown advice.

# AWS: see rightsizing findings (after you opt in to Compute Optimizer)
aws compute-optimizer get-ec2-instance-recommendations \
  --query "instanceRecommendations[].{Instance:instanceArn,Finding:finding}"

# Azure: list cost recommendations from Azure Advisor
az advisor recommendation list --category Cost -o table

Change one thing at a time. Watch performance for a week. Then move on.

Step 5: Commit for steady use

Some workloads run 24/7 and will keep running. For those, a commitment is usually cheaper than on-demand pricing.

AWS: Savings Plans and Reserved Instances, with 1-year or 3-year terms.
Azure: Azure Reservations and the Azure savings plan for compute, also with 1-year or 3-year terms.

Simple rule: rightsize first, then commit. If you commit to a server that is too big, you lock in the waste.

Figure 4. Steps 4 and 5 work together: rightsize first, then commit.

Step 6: Set budgets and alerts

A budget will not fix waste. But it tells you early when something goes wrong.

AWS Budgets: Create a monthly cost budget. Add alerts at levels you choose, for example when actual or forecast spend crosses part of the budget.
Azure Cost Management budgets: Create a budget at the subscription or resource group level. Add alert conditions and email recipients. You can also connect an action group (a set of actions Azure runs when an alert fires).

Note: by default, alerts only notify people. They do not stop resources. Make sure a real person reads them.

Figure 5. A budget alert warns a person early. By default, it does not stop anything.

Step 7: Review every month

Put a short meeting on the calendar. Same day each month. Use the saved view from Step 1.

Ask four questions:

What went up, and why?
Are there new untagged resources?
Are there new idle disks or IPs?
Do our commitments still match our steady usage?

That's it. Small, boring, and it works.

Quick copy-paste checklist

[ ] Cost Explorer and Cost Management views saved
[ ] owner, env, project tags on everything (and activated on AWS)
[ ] Unattached disks and unused IPs reviewed
[ ] Non-prod VMs shut down at night
[ ] Rightsizing advice reviewed (Compute Optimizer, Azure Advisor)
[ ] Commitments only for steady, right-sized workloads
[ ] Budgets and alerts set, with a named owner
[ ] Monthly review on the calendar

About the author: CloudSkill Consulting LLC is a cloud consulting firm in Omaha, Nebraska, founded in September 2024. We work on AWS, Microsoft Azure and Google Cloud, with a focus on cost control (FinOps), multi-cloud planning and migration. We also provide technical and IT services across the USA. If this checklist helped, you can find more at cloudskillconsulting.com.