DEV Community: Harun R. Rayhan

Why You Should Use S3 Presigned URLs to Upload Big Static Files

Harun R. Rayhan — Thu, 29 Oct 2026 04:00:07 +0000

A 2 GB video. Someone needed it on S3 from a phone. The fast fix was a bucket policy with Principal: "*". The upload worked. So did every other key in the bucket, for as long as that policy stayed.

A pre-signed URL is the narrower tool. For a file this size you do not sign one giant PUT. The API starts a multipart upload, signs each part, and the browser PUTs those parts straight to S3. The object stays private. The figure below is that cycle as a small game: watch the parts land, or lock Browser or API and see what each side does.

Your API only signs and completes. The 2 GB of bytes go from the browser to S3 and never pass through the API.The figure starts when it is on screen. First the API creates the multipart upload and signs the part URLs. Then the browser uploads Part 1 through Part 4 of the 2 GB video. Then the API completes the upload so S3 assembles the object. Choose Browser or API and it stays on that side. Click the selected one again to return to the full cycle. Speed and reset sit at the bottom.

▶ Try the interactive demo on harun.dev

Why the API should not carry the file

The tempting shape is: browser posts the video to your API, the API writes it to S3 with the AWS SDK, and you call it done. That shape breaks once the file is measured in gigabytes.

API Gateway has a hard 10 MB payload limit. A Lambda behind it times out at 15 minutes and still has to hold the body while it streams to S3. You pay for the ingress into the API and the egress out to S3. A small EC2 or container box has the same problem: two hops, double bandwidth, and a socket that dies when the phone drops wifi mid-upload.

The other failure mode is credentials. If the browser talks to S3 with a long-lived access key, that key leaves your VPC. A pre-signed part URL is temporary permission for one part. The IAM secret never ships to the client.

The requests that make a multipart cycle

A 2 GB video is not one PUT. S3 multipart splits it into parts (minimum 5 MB except the last part). In the figure I use four parts of about 512 MB each so you can see them stack.

Step	Who	What moves	What S3 sees
1	Browser → your API	JSON: filename, content type, size	Nothing yet
2	Your API → S3 + signing library	`CreateMultipartUpload`, then SigV4 over each `UploadPart`	An open multipart upload id
3	Browser → S3	PUT part 1…N to each signed URL (~512 MB each)	Parts land with ETags
4	Browser → your API → S3	Part numbers + ETags	`CompleteMultipartUpload` assembles the 2 GB object

Steps 1, 2, and 4 are small on purpose. The API checks auth, picks a key like uploads/hrr_export/2026-10-08/video.mp4, opens the multipart upload, and returns signed part URLs. Step 3 is the only place the video bytes go. In the figure, the Lambda column is the signing and the complete. The Browser column plus the part bars on S3 are the uploads. The final 2 GB bar is the assembled object.

If you lock Browser before any multipart session has started, the figure shows a 403. That matches a real bucket with Block Public Access on and no signature in the query string.

How each part signature is built

The API holds the IAM credentials. The browser does not. Each part URL is Signature Version 4 over a specific UploadPart request: method, bucket, key, upload id, part number, region, time window, and the headers you chose to sign.

You will see the usual query params on every part URL:

X-Amz-Algorithm: always AWS4-HMAC-SHA256 for SigV4.
X-Amz-Credential: access key id, date, region, service (s3), and the literal aws4_request.
X-Amz-Date: when the signature was created, in UTC.
X-Amz-Expires: lifetime in seconds from that date. Parts of a 2 GB video need enough time for a slow mobile link. I still keep this in minutes or a couple of hours, not days.
X-Amz-SignedHeaders: which headers were part of the string to sign. Often just host for part uploads.
X-Amz-Signature: the hex HMAC. Change the part number, the upload id, or the key and S3 rejects that part.

The upload id from CreateMultipartUpload ties the parts together. A signature for part 2 will not write part 3. A complete call with the wrong ETag list fails. That is the point: each URL is one part of one object, not a key to the bucket.

The API that starts multipart and signs the parts

This is the shape I use with the AWS SDK for JavaScript v3. The handler never sees the video body.

import {
  S3Client,
  CreateMultipartUploadCommand,
  UploadPartCommand,
  CompleteMultipartUploadCommand,
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { randomUUID } from "crypto";

const hrr_s3 = new S3Client({ region: "us-east-1" });
const HRR_BUCKET = process.env.HRR_UPLOAD_BUCKET!;
const HRR_PART_COUNT = 4;

export async function hrr_createMultipartUpload(input: {
  filename: string;
  contentType: string;
}) {
  const uploadId = randomUUID();
  const key = `uploads/${uploadId}/${input.filename}`;

  const created = await hrr_s3.send(
    new CreateMultipartUploadCommand({
      Bucket: HRR_BUCKET,
      Key: key,
      ContentType: input.contentType,
    }),
  );

  const multipartUploadId = created.UploadId!;
  const partUrls: string[] = [];

  for (let partNumber = 1; partNumber <= HRR_PART_COUNT; partNumber += 1) {
    const command = new UploadPartCommand({
      Bucket: HRR_BUCKET,
      Key: key,
      UploadId: multipartUploadId,
      PartNumber: partNumber,
    });

    partUrls.push(await getSignedUrl(hrr_s3, command, { expiresIn: 3600 }));
  }

  return {
    key,
    uploadId: multipartUploadId,
    partUrls,
    partCount: HRR_PART_COUNT,
    expiresIn: 3600,
  };
}

export async function hrr_completeMultipartUpload(input: {
  key: string;
  uploadId: string;
  parts: { ETag: string; PartNumber: number }[];
}) {
  await hrr_s3.send(
    new CompleteMultipartUploadCommand({
      Bucket: HRR_BUCKET,
      Key: input.key,
      UploadId: input.uploadId,
      MultipartUpload: { Parts: input.parts },
    }),
  );
}

getSignedUrl builds the canonical request for each part, derives the signing key from the IAM secret, and appends the X-Amz-* query params. The role on this Lambda needs s3:CreateMultipartUpload, s3:UploadPart (for signing), and s3:CompleteMultipartUpload on that prefix. It does not need to read the video bytes.

The browser that uploads the parts

The browser slices the 2 GB file, PUTs each slice to its signed URL, and keeps the ETag S3 returns. No AWS SDK in the client. The query string is the auth.

async function hrr_uploadVideoParts(
  file: File,
  partUrls: string[],
) {
  const partSize = Math.ceil(file.size / partUrls.length);
  const parts: { ETag: string; PartNumber: number }[] = [];

  for (let index = 0; index < partUrls.length; index += 1) {
    const start = index * partSize;
    const blob = file.slice(start, start + partSize);
    const response = await fetch(partUrls[index], {
      method: "PUT",
      body: blob,
    });

    if (!response.ok) {
      throw new Error(`Part ${index + 1} failed: ${response.status}`);
    }

    const etag = response.headers.get("ETag");
    if (!etag) {
      throw new Error(`Part ${index + 1} missing ETag`);
    }

    parts.push({ ETag: etag, PartNumber: index + 1 });
  }

  return parts;
}

After every part succeeds, the browser posts the ETag list back to your API so it can call CompleteMultipartUpload. Until that call, S3 still holds loose parts, not a finished object. If the phone dies mid-upload, you can resume from the next missing part instead of restarting the whole 2 GB.

CORS still matters. The bucket needs a rule that allows PUT from your origin, and it must expose ETag so the browser can read it. The pre-signed URL does not replace CORS. It replaces long-lived credentials.

Is this secure?

Yes, when you keep the bucket private and you treat each part URL as a short-lived ticket for one slice of one object. It is more secure than shipping IAM keys to Chrome, and more secure than opening the bucket with Principal: "*".

It is not magic. A leaked part URL still lets whoever has it upload that one part until expiry. Security here is narrow permission plus a short clock, not "the URL is secret forever."

How?

The IAM access key and secret stay on the API. Chrome only receives signed URLs. Those URLs encode SigV4 over a fixed request: this bucket, this key, this multipart upload id, this part number, this expiry. Change any of those and the signature stops matching, so S3 returns 403.

That is what the figure is showing. Lock Browser before the API has signed anything and S3 denies the upload. After the API signs, Chrome can PUT parts, but it still cannot list the bucket, read other keys, or complete the multipart upload without another call that only the API is allowed to make.

A few habits keep that promise honest:

Leave Block Public Access on. Do not open the bucket to make the upload "just work."
Sign only the part numbers you need. Prefer one set of part URLs per upload attempt.
Keep X-Amz-Expires short enough for a slow mobile upload, not for a week of Slack forwards.
On cancel or failure, call AbortMultipartUpload so incomplete parts do not sit around.
Treat a pasted part URL like a temporary password for one slice. Log when you mint them.

A part URL will not list objects. It will not write a different key or a different part number. It will not switch from PUT to GET or DELETE. Completing or aborting the multipart upload stays on the API.

Keep the bucket private

I do not open the bucket to get a 2 GB video uploaded. A public bucket would accept those PUTs tomorrow, and every other key too. Block Public Access on the account is there because this fix keeps coming back.

I still make an object public when it is meant to be public: a cover image, a package on a CDN, something I would put on a website without a login. A customer's video export is not that. Upload with pre-signed multipart parts. Read it later with a short-lived pre-signed GET, or serve it through CloudFront with an origin access control. Those are separate decisions from the upload.

The 2 GB video in the figure goes from Chrome to S3 in parts. The API holds the key, starts the multipart upload, signs each part, completes the object, and stays out of the byte path. That is the whole reason to use pre-signed URLs for big static files: narrow permission, resumable parts, no proxy in the middle.

Hope you enjoyed this one. If a multipart part has bitten you on a missing ETag header in CORS, find me on X at https://x.com/harundotdev, and grab every other link from my bio at https://harun.dev/bio.

Originally published at harun.dev. The full interactive version lives there. Follow me on X or read my bio.

Terraform or Pulumi, Pick From the Team You Have

Harun R. Rayhan — Thu, 22 Oct 2026 05:00:06 +0000

The argument started as a logo fight. Terraform on one side, Pulumi on the other, as if the cloud would behave once we picked the right binary.

Both tools do the same loop. You describe the resources. You preview. You apply. State records what actually exists. The two terminals below run that deploy side by side for the same S3 bucket. Speed and reset sit at the bottom.

Same loop either way: describe it, preview it, apply it, record it in state. Only the language you describe it in changes.

▶ Try the interactive demo on harun.dev

The loop is the product

I stop people when the debate is only about HCL versus TypeScript. The product is the loop. Pulumi can now run existing Terraform HCL too (runtime: hcl), so even the language is no longer a hard line:

Step	Terraform	Pulumi	What you are checking
Describe	`main.tf` / modules	`index.ts` (or Python, Go, HCL, …)	Desired state is readable by the people who must review it
Preview	`terraform plan`	`pulumi preview`	Creates, updates, and deletes before anything changes
Apply	`terraform apply`	`pulumi up`	Cloud APIs run with credentials you intended
State	`terraform.tfstate` (remote)	stack state (remote)	Real IDs, and often secrets, stay locked down

Miss the preview and you get surprise destroys. Miss the state lock and two applies race. Miss who can read state and you leak database passwords. The CLI brand does not fix those.

Same bucket, two descriptions

Here is an S3 bucket both ways. Prefixes stay hrr_.

resource "aws_s3_bucket" "hrr_uploads" {
  bucket = "hrr-uploads-example"

  tags = {
    Project = "hrr-demo"
  }
}

import * as aws from "@pulumi/aws";

const hrrUploads = new aws.s3.Bucket("hrrUploads", {
  bucket: "hrr-uploads-example",
  tags: {
    Project: "hrr-demo",
  },
});

Same object in AWS after apply. The review question is who can read the diff in the pull request. If half the reviewers do not write TypeScript, the Pulumi preview is noise. Pulumi can also run your existing .tf files through its HCL runtime, which softens that gap. It is new, so try it on a non-critical stack first. If the app team already owns the account and the repo, HCL is the foreign language.

State does not care which CLI wrote it

Both tools need a memory of what they created. That memory is more sensitive than a .env file, and switching CLIs does not delete the risk. It renames the file.

The Terraform state caveat

terraform.tfstate is JSON that maps your configuration to real cloud objects. After an apply it commonly holds:

resource IDs, ARNs, and relationships across the stack
database usernames and passwords passed as resource arguments
IAM access key secrets Terraform created
private keys from the TLS provider
API tokens, webhook URLs, and Lambda environment variables

Marking a variable sensitive = true hides it from normal CLI output. It does not remove the value from state. Terraform needs those attributes to plan the next diff. Treat every production state snapshot as if it contains credentials, even when last month's file looked clean. Providers change. Modules grow. A data source you used while debugging can leave a secret behind.

Two operators applying at once without a lock can corrupt that file. A state file in git, in a CI artifact, or in a bucket every developer can read is a credential dump with a map of your account attached. I wrote the fuller Terraform treatment in Why Your Terraform State Is More Sensitive Than Your .env File.

My minimum remote backend looks like this:

terraform {
  backend "s3" {
    bucket       = "hrr-terraform-state"
    key          = "demo/production/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

Dedicated bucket. Encryption on. Lock on. Read access only for the people and CI roles that must run plan and apply. Not the uploads bucket. Not a laptop copy "just for today."

The Pulumi equivalent

Pulumi's equivalent is stack state. Same job as Terraform state: remember which cloud objects belong to this stack so the next preview and up are honest.

Where it lives depends on the backend you chose: Pulumi Cloud, an S3/DIY backend, or another supported store. The product name changes. The contents do not get safer. Stack state still carries resource identities and can carry secrets from configuration and provider outputs. Pulumi can encrypt secret values in state and config, which is useful. Encryption at rest is not the same as "anyone on the team can download the stack export."

So I apply the same rules:

Do not keep stack state on a developer laptop as the source of truth.
Use a remote backend you can lock and audit.
Limit who can read the stack, export it, or decrypt secrets.
Prefer a passphrase or KMS-backed secrets provider you control, not a shared Slack password.

What is not the same: the file format, the CLI commands, and the migration path. You do not "rename" terraform.tfstate into a Pulumi stack and call it done. Importing or rebuilding state is a project of its own. The caveat that transfers is the security model, not the bytes on disk.

In the terminals above, both sides end on a locked remote state line for a reason. The green "1 added" / "1 created" is incomplete until that record is somewhere you would trust with a production password.

Preview is the review

terraform plan and pulumi preview are the diff. I do not apply from a laptop against production without that diff in the pull request. CI should run the preview on every change that touches infra. Humans read the create/update/delete list. The language of the description changes who can read it. It does not remove the need for one.

When the plan says destroy on a database, stop. When the preview is empty and you expected a change, stop. The green "1 added" / "1 created" lines in the terminals only show up after that gate. Apply is only honest after it.

Which one should you pick?

Pick from the team you have, not the conference talk you liked.

Stay on Terraform when infrastructure reviewers already write and review HCL, when you have a module library that works, and when operators live in the Terraform workflow.
Choose Pulumi when the same people already own the app and the account, when loops and types would cut real duplication, and when the preview will be read by TypeScript (or Python) developers.

I have used both. The outages I remember came from state that anyone could read, and from an apply that nobody previewed. They did not come from the extension on the file.

Hope you enjoyed this one. If your team is mid-argument about this, find me on X at https://x.com/harundotdev, and grab every other link from my bio at https://harun.dev/bio.

Originally published at harun.dev. The full interactive version lives there. Follow me on X or read my bio.

Serverless Laravel on AWS Lambda with Bref and Terraform

Harun R. Rayhan — Thu, 22 Oct 2026 04:42:57 +0000

Why Bref Instead of Vapor

Laravel Vapor is the official serverless deployment tool for Laravel. It works. I've used it on two client projects. But it costs $399 per year just for the subscription, and on top of that you're still paying for all the AWS resources underneath. What bothered me most wasn't the price though. It was the abstraction. Vapor manages your infrastructure behind the scenes, and when something goes sideways at 1am, you're digging through Vapor's CloudFormation stacks trying to figure out what it actually provisioned. I wanted more control.

So I started looking for alternatives. Bref has been around for a while now. It's an open-source project that gives you PHP runtimes for AWS Lambda. You get the same serverless model as Vapor, but you own every single piece of infrastructure. And because it's just Lambda layers, you can wire everything up with whatever IaC tool you prefer.

This post walks through deploying a full Laravel application to Lambda using Bref and Terraform. I'm talking the whole setup. Aurora Serverless v2 for Postgres, ElastiCache Serverless for Redis, SQS for queues, EventBridge for cron, S3 for uploads. If you've read my previous posts, some of this architecture will look familiar. In Serverless SaaS on AWS Lambda with Fastify I covered many of the same Lambda patterns but with Node.js. And in Deploying a Laravel Application to Amazon EC2 Using Terraform I showed the traditional EC2 approach. Think of this post as the upgrade from that EC2 setup. Same Terraform mindset, but no servers.

We're deploying Laravel 13 by the way. Everything here works with Laravel 11 and 12 too, but the config examples match 13.

Let's get into it.

Photo by ThisIsEngineering on Pexels

Architecture Overview

Here's the full picture of what we're building:

Three Lambda functions total. The web Lambda handles HTTP via the Bref FPM runtime (php-83-fpm), which boots PHP-FPM and proxies API Gateway events to it. Your Laravel app thinks it's running behind Nginx. The worker Lambda processes SQS jobs, and the artisan Lambda fires schedule:run every minute via EventBridge. Both use the Bref CLI runtime (php-83), no FPM overhead.

Aurora Serverless v2 scales Postgres capacity in ACUs based on actual load. No instance sizing, no patching, Multi-AZ by default. If you'd rather skip VPC networking entirely, Supabase and Neon both work as drop-in Postgres replacements accessible over the public internet. For this post we're staying full AWS.

ElastiCache Serverless gives you pay-per-use Redis with zero cluster management. Laravel needs it for sessions and cache because Lambda's filesystem is ephemeral between cold starts. Both Aurora and ElastiCache live inside a VPC, so your Lambda functions need to be there too.

Photo by Brett Sayles on Pexels

Setting Up Laravel + Bref

First, pull in the Bref packages:

composer require bref/bref bref/laravel-bridge
php artisan vendor:publish --tag=serverless-config

The bref/laravel-bridge package does the heavy lifting: it bootstraps Laravel for Lambda, handles FPM event conversion, and provides Artisan handlers for queue and CLI invocations. After publishing you'll get a serverless.php entrypoint in your project root.

Lambda doesn't have a .env file on disk. You pass environment variables through the Lambda function config. Here's what needs to change from a typical server setup:

SESSION_DRIVER=cookie
CACHE_STORE=redis
FILESYSTEM_DISK=s3
LOG_CHANNEL=stderr
QUEUE_CONNECTION=sqs
VIEW_COMPILED_PATH=/tmp/views

Cookie sessions are stateless and the simplest starting point. Upgrade to Redis sessions if your session data outgrows the 4KB cookie limit. Redis cache is mandatory because Lambda's filesystem doesn't persist across cold starts. Stderr logging goes straight to CloudWatch. And /tmp/views is the only writable path for compiled Blade templates. Any temp files your app writes (uploads, exports) should also target /tmp before being pushed to S3.

Aurora Serverless v2 + ElastiCache Serverless

Let's talk about the database and cache layers before we get into Terraform, because understanding these services makes the infrastructure code much easier to follow.

Aurora Serverless v2 scales Postgres capacity in ACUs instead of fixing an instance size. You set a minimum (I use 0.5) and a maximum (4 to 8 for most Laravel apps), and Aurora adjusts within that range based on load. Billing is per ACU-hour, roughly $0.12 in us-east-1. It's real Aurora under the hood: same replication, same query optimizer, Multi-AZ by default.

From Laravel's side, the connection config is standard Postgres:

DB_CONNECTION=pgsql
DB_HOST=hrr-laravel-db.cluster-xxxxxxxxxxxx.us-east-1.rds.amazonaws.com
DB_PORT=5432
DB_DATABASE=hrr_laravel
DB_USERNAME=hrr_admin
DB_PASSWORD=<from-secrets-manager>

If you'd rather avoid VPC networking for the database, Supabase and Neon are both solid alternatives. They're publicly accessible Postgres, so you just swap the host and skip the NAT Gateway entirely. For this guide we're going full AWS.

ElastiCache Serverless is pay-per-use Redis with no node type or replication group to manage. Bref's PHP layer includes phpredis, so no extra setup needed.

REDIS_HOST=hrr-laravel-cache-xxxxxx.serverless.use1.cache.amazonaws.com
REDIS_PORT=6379

Both services require VPC placement, so your Lambda functions need to be in the same VPC. If your app calls external APIs (Stripe, Mailgun, etc.), you'll need a NAT Gateway in a public subnet. That runs about $32 per month and is often the biggest fixed cost in what's supposed to be a "serverless" setup.

Terraform Infrastructure

Time for the fun part. The full Terraform config is around 300 lines, so I'll walk through the key blocks and explain what connects to what.

VPC and Subnets

We need a VPC with both public and private subnets. Lambda, Aurora, and ElastiCache go in private subnets. The NAT Gateway sits in a public subnet so Lambda can reach the internet.

resource "aws_vpc" "hrr_laravel_vpc" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = { Name = "hrr-laravel-vpc" }
}

resource "aws_subnet" "hrr_private_a" {
  vpc_id            = aws_vpc.hrr_laravel_vpc.id
  cidr_block        = "10.0.1.0/24"
  availability_zone = "${var.aws_region}a"
  tags = { Name = "hrr-laravel-private-a" }
}

resource "aws_subnet" "hrr_private_b" {
  vpc_id            = aws_vpc.hrr_laravel_vpc.id
  cidr_block        = "10.0.2.0/24"
  availability_zone = "${var.aws_region}b"
  tags = { Name = "hrr-laravel-private-b" }
}

resource "aws_subnet" "hrr_public" {
  vpc_id                  = aws_vpc.hrr_laravel_vpc.id
  cidr_block              = "10.0.3.0/24"
  availability_zone       = "${var.aws_region}a"
  map_public_ip_on_launch = true
  tags = { Name = "hrr-laravel-public" }
}

Two private subnets in different AZs because Aurora requires at least two. One public subnet for the NAT Gateway and Internet Gateway.

Aurora Serverless v2

This part tripped me up the first time. Aurora Serverless v2 doesn't use the old engine_mode = "serverless". That was v1. For v2, you set engine_mode = "provisioned" and add a serverlessv2_scaling_configuration block. Confusing naming, I know.

resource "aws_rds_cluster" "hrr_laravel_db" {
  cluster_identifier     = "hrr-laravel-db"
  engine                 = "aurora-postgresql"
  engine_mode            = "provisioned"
  engine_version         = "16.4"
  database_name          = "hrr_laravel"
  master_username        = "hrr_admin"
  master_password        = var.db_password
  db_subnet_group_name   = aws_db_subnet_group.hrr_laravel.name
  vpc_security_group_ids = [aws_security_group.hrr_db_sg.id]
  skip_final_snapshot    = true

  serverlessv2_scaling_configuration {
    min_capacity = 0.5
    max_capacity = 4.0
  }

  tags = { Name = "hrr-laravel-db" }
}

resource "aws_rds_cluster_instance" "hrr_laravel_db_instance" {
  cluster_identifier = aws_rds_cluster.hrr_laravel_db.id
  instance_class     = "db.serverless"
  engine             = aws_rds_cluster.hrr_laravel_db.engine
  engine_version     = aws_rds_cluster.hrr_laravel_db.engine_version

  tags = { Name = "hrr-laravel-db-instance" }
}

resource "aws_db_subnet_group" "hrr_laravel" {
  name       = "hrr-laravel-db-subnet"
  subnet_ids = [
    aws_subnet.hrr_private_a.id,
    aws_subnet.hrr_private_b.id,
  ]
}

The key is instance_class = "db.serverless". That tells Aurora to use serverless v2 scaling on this instance. The cluster-level serverlessv2_scaling_configuration sets the ACU range.

ElastiCache Serverless

resource "aws_elasticache_serverless_cache" "hrr_laravel_cache" {
  engine = "redis"
  name   = "hrr-laravel-cache"

  cache_usage_limits {
    data_storage {
      maximum = 1
      unit    = "GB"
    }
    ecpu_per_second {
      maximum = 1000
    }
  }

  subnet_ids         = [
    aws_subnet.hrr_private_a.id,
    aws_subnet.hrr_private_b.id,
  ]
  security_group_ids = [aws_security_group.hrr_cache_sg.id]
}

ElastiCache Serverless is beautifully simple compared to the old way. No replication groups, no node types, no parameter groups. You set usage limits and AWS handles everything else.

Lambda Function (Bref FPM)

resource "aws_lambda_function" "hrr_laravel_web" {
  function_name = "hrr_laravel_web"
  runtime       = "provided.al2023"
  architectures = ["arm64"]
  handler       = "Bref\\LaravelBridge\\Http\\OctaneHandler"
  memory_size   = 1024
  timeout       = 28

  filename         = "${path.module}/dist/app.zip"
  source_code_hash = filebase64sha256("${path.module}/dist/app.zip")

  layers = [
    "arn:aws:lambda:${var.aws_region}:534081306603:layer:php-83-fpm:68",
  ]

  vpc_config {
    subnet_ids         = [
      aws_subnet.hrr_private_a.id,
      aws_subnet.hrr_private_b.id,
    ]
    security_group_ids = [aws_security_group.hrr_lambda_sg.id]
  }

  environment {
    variables = {
      APP_ENV            = "production"
      APP_KEY            = var.app_key
      DB_CONNECTION      = "pgsql"
      DB_HOST            = aws_rds_cluster.hrr_laravel_db.endpoint
      DB_PORT            = "5432"
      DB_DATABASE        = "hrr_laravel"
      DB_USERNAME        = "hrr_admin"
      DB_PASSWORD        = var.db_password
      REDIS_HOST         = aws_elasticache_serverless_cache.hrr_laravel_cache.endpoint[0].address
      REDIS_PORT         = "6379"
      CACHE_STORE        = "redis"
      SESSION_DRIVER     = "cookie"
      LOG_CHANNEL        = "stderr"
      QUEUE_CONNECTION   = "sqs"
      SQS_QUEUE          = aws_sqs_queue.hrr_laravel_queue.url
      FILESYSTEM_DISK    = "s3"
      AWS_BUCKET         = aws_s3_bucket.hrr_laravel_uploads.id
    }
  }

  tags = { Name = "hrr-laravel-web" }
}

A few things to note. The runtime is provided.al2023, not php83 or anything like that. Bref provides the PHP runtime as a Lambda layer. The handler points to Bref's Laravel bridge, which bootstraps your application and converts API Gateway events into HTTP requests that FPM can process.

I set memory_size to 1024MB. Lambda allocates CPU proportionally to memory, so 1024MB gives you a decent amount of compute. The timeout is 28 seconds because API Gateway HTTP API has a 29-second hard limit. You want Lambda to finish before the gateway times out, otherwise you get a 502 with no useful error message. I learned this one by debugging a 502 for about 45 minutes before realizing the Lambda was timing out at 30 seconds, one second after API Gateway gave up.

Using arm64 (Graviton) gives you about 20% better price-performance than x86. Bref supports both architectures.

API Gateway HTTP API

resource "aws_apigatewayv2_api" "hrr_laravel_api" {
  name          = "hrr-laravel-api"
  protocol_type = "HTTP"
}

resource "aws_apigatewayv2_integration" "hrr_laravel_integration" {
  api_id                 = aws_apigatewayv2_api.hrr_laravel_api.id
  integration_type       = "AWS_PROXY"
  integration_uri        = aws_lambda_function.hrr_laravel_web.invoke_arn
  payload_format_version = "2.0"
}

resource "aws_apigatewayv2_route" "hrr_laravel_default" {
  api_id    = aws_apigatewayv2_api.hrr_laravel_api.id
  route_key = "\$default"
  target    = "integrations/${aws_apigatewayv2_integration.hrr_laravel_integration.id}"
}

resource "aws_apigatewayv2_stage" "hrr_laravel_stage" {
  api_id      = aws_apigatewayv2_api.hrr_laravel_api.id
  name        = "\$default"
  auto_deploy = true
}

resource "aws_lambda_permission" "hrr_laravel_apigw" {
  statement_id  = "AllowAPIGateway"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.hrr_laravel_web.function_name
  principal     = "apigateway.amazonaws.com"
  source_arn    = "${aws_apigatewayv2_api.hrr_laravel_api.execution_arn}/*/*"
}

The $default route catches everything and sends it to Lambda. Laravel's router handles the actual URL matching, so we don't need to define API Gateway routes for every endpoint. The payload_format_version = "2.0" is important because Bref expects the v2 event format.

S3 Bucket and IAM

resource "aws_s3_bucket" "hrr_laravel_uploads" {
  bucket = "hrr-laravel-uploads-${var.aws_account_id}"
  tags   = { Name = "hrr-laravel-uploads" }
}

resource "aws_iam_role" "hrr_laravel_lambda_role" {
  name = "hrr-laravel-lambda-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "hrr_lambda_vpc" {
  role       = aws_iam_role.hrr_laravel_lambda_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}

resource "aws_iam_policy" "hrr_laravel_app_policy" {
  name = "hrr-laravel-app-policy"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"]
        Resource = "${aws_s3_bucket.hrr_laravel_uploads.arn}/*"
      },
      {
        Effect   = "Allow"
        Action   = ["sqs:SendMessage", "sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
        Resource = aws_sqs_queue.hrr_laravel_queue.arn
      },
      {
        Effect   = "Allow"
        Action   = ["secretsmanager:GetSecretValue"]
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "hrr_laravel_app" {
  role       = aws_iam_role.hrr_laravel_lambda_role.name
  policy_arn = aws_iam_policy.hrr_laravel_app_policy.arn
}

The IAM role gives Lambda permission to run inside the VPC, access S3 for uploads, send and receive SQS messages, and pull secrets from Secrets Manager. In production, you'd scope the Secrets Manager resource down to specific secret ARNs instead of using a wildcard.

Photo by Godfrey Atima on Pexels

SQS Queues and EventBridge Cron

Laravel's queue system on Lambda works differently than on a traditional server. Normally you'd run php artisan queue:work as a long-running process. On Lambda, that's not possible. Instead, SQS triggers a Lambda function for each message, and Bref's queue handler processes it as a single job invocation.

SQS Queue and Dead Letter Queue

resource "aws_sqs_queue" "hrr_laravel_dlq" {
  name                      = "hrr_laravel_dlq"
  message_retention_seconds = 1209600

  tags = { Name = "hrr-laravel-dlq" }
}

resource "aws_sqs_queue" "hrr_laravel_queue" {
  name                       = "hrr_laravel_queue"
  visibility_timeout_seconds = 120
  message_retention_seconds  = 1209600

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.hrr_laravel_dlq.arn
    maxReceiveCount     = 3
  })

  tags = { Name = "hrr-laravel-queue" }
}

The visibility timeout is set to 120 seconds, which is about twice the Lambda timeout. If a job fails and Lambda doesn't delete the message, SQS waits 120 seconds before making it visible again for retry. After 3 failed attempts, the message moves to the DLQ where it sits for 14 days.

Queue Worker Lambda

resource "aws_lambda_function" "hrr_laravel_worker" {
  function_name = "hrr_laravel_worker"
  runtime       = "provided.al2023"
  architectures = ["arm64"]
  handler       = "Bref\\LaravelBridge\\Queue\\QueueHandler"
  memory_size   = 1024
  timeout       = 60
  role          = aws_iam_role.hrr_laravel_lambda_role.arn

  filename         = "${path.module}/dist/app.zip"
  source_code_hash = filebase64sha256("${path.module}/dist/app.zip")

  layers = [
    "arn:aws:lambda:${var.aws_region}:534081306603:layer:php-83:68",
  ]

  vpc_config {
    subnet_ids         = [
      aws_subnet.hrr_private_a.id,
      aws_subnet.hrr_private_b.id,
    ]
    security_group_ids = [aws_security_group.hrr_lambda_sg.id]
  }

  environment {
    variables = {
      APP_ENV          = "production"
      APP_KEY          = var.app_key
      DB_CONNECTION    = "pgsql"
      DB_HOST          = aws_rds_cluster.hrr_laravel_db.endpoint
      DB_PORT          = "5432"
      DB_DATABASE      = "hrr_laravel"
      DB_USERNAME      = "hrr_admin"
      DB_PASSWORD      = var.db_password
      REDIS_HOST       = aws_elasticache_serverless_cache.hrr_laravel_cache.endpoint[0].address
      REDIS_PORT       = "6379"
      CACHE_STORE      = "redis"
      LOG_CHANNEL      = "stderr"
      QUEUE_CONNECTION = "sqs"
      SQS_QUEUE        = aws_sqs_queue.hrr_laravel_queue.url
    }
  }

  tags = { Name = "hrr-laravel-worker" }
}

resource "aws_lambda_event_source_mapping" "hrr_laravel_sqs_trigger" {
  event_source_arn = aws_sqs_queue.hrr_laravel_queue.arn
  function_name    = aws_lambda_function.hrr_laravel_worker.arn
  batch_size       = 1

  function_response_types = ["ReportBatchItemFailures"]
}

Notice this Lambda uses the php-83 layer, not php-83-fpm. Queue workers don't need FPM because there's no HTTP request to process. The handler Bref\LaravelBridge\Queue\QueueHandler knows how to deserialize SQS messages back into Laravel job objects. Your job classes work exactly like they would on a normal server. You dispatch jobs the same way too. dispatch(new ProcessPodcast($podcast)) sends a message to SQS, and the worker Lambda picks it up.

EventBridge Cron for Artisan Schedule

resource "aws_lambda_function" "hrr_laravel_artisan" {
  function_name = "hrr_laravel_artisan"
  runtime       = "provided.al2023"
  architectures = ["arm64"]
  handler       = "artisan"
  memory_size   = 512
  timeout       = 120
  role          = aws_iam_role.hrr_laravel_lambda_role.arn

  filename         = "${path.module}/dist/app.zip"
  source_code_hash = filebase64sha256("${path.module}/dist/app.zip")

  layers = [
    "arn:aws:lambda:${var.aws_region}:534081306603:layer:php-83:68",
    "arn:aws:lambda:${var.aws_region}:534081306603:layer:console:68",
  ]

  vpc_config {
    subnet_ids         = [
      aws_subnet.hrr_private_a.id,
      aws_subnet.hrr_private_b.id,
    ]
    security_group_ids = [aws_security_group.hrr_lambda_sg.id]
  }

  environment {
    variables = {
      APP_ENV          = "production"
      APP_KEY          = var.app_key
      DB_CONNECTION    = "pgsql"
      DB_HOST          = aws_rds_cluster.hrr_laravel_db.endpoint
      DB_PORT          = "5432"
      DB_DATABASE      = "hrr_laravel"
      DB_USERNAME      = "hrr_admin"
      DB_PASSWORD      = var.db_password
      REDIS_HOST       = aws_elasticache_serverless_cache.hrr_laravel_cache.endpoint[0].address
      REDIS_PORT       = "6379"
      CACHE_STORE      = "redis"
      LOG_CHANNEL      = "stderr"
    }
  }

  tags = { Name = "hrr-laravel-artisan" }
}

resource "aws_cloudwatch_event_rule" "hrr_laravel_cron" {
  name                = "hrr-laravel-cron"
  schedule_expression = "rate(1 minute)"

  tags = { Name = "hrr-laravel-cron" }
}

resource "aws_cloudwatch_event_target" "hrr_laravel_cron_target" {
  rule = aws_cloudwatch_event_rule.hrr_laravel_cron.name
  arn  = aws_lambda_function.hrr_laravel_artisan.arn

  input = jsonencode({
    cli = "schedule:run"
  })
}

resource "aws_lambda_permission" "hrr_laravel_eventbridge" {
  statement_id  = "AllowEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.hrr_laravel_artisan.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.hrr_laravel_cron.arn
}

EventBridge fires every minute, just like a normal cron. The input payload tells Bref's console layer to run php artisan schedule:run. Laravel's scheduler then checks which commands are due and executes them. This is the exact same behavior as having * * * * * cd /path && php artisan schedule:run >> /dev/null 2>&1 in your crontab on a server.

I also use this artisan Lambda for ad-hoc commands. Need to run migrations? Invoke it with {"cli": "migrate --force"}. Need to clear cache? {"cli": "cache:clear"}. Really handy during deployments.

Deploy, Test, and Iterate

With all the Terraform written, let's actually ship this thing.

Build the Deployment Package

Before running Terraform, you need to package your Laravel app into a zip. This zip gets uploaded as the Lambda function code. Here's how I do it:

# Install production dependencies
composer install --no-dev --optimize-autoloader --classmap-authoritative

# Optimize Laravel
php artisan config:cache
php artisan route:cache
php artisan view:cache
php artisan event:cache

# Package everything into a zip
zip -r dist/app.zip . \
  -x ".git/*" \
  -x "node_modules/*" \
  -x "tests/*" \
  -x "storage/logs/*" \
  -x ".env"

The --classmap-authoritative flag skips PSR-4 autoloading for faster cold starts, and caching config, routes, and views cuts down the work Laravel does on each request. Both matter more on Lambda than on a server.

Run Terraform

terraform init
terraform plan

The plan output is going to be long. You're creating a VPC, subnets, security groups, Aurora cluster, ElastiCache cache, three Lambda functions, API Gateway, SQS queues, EventBridge rules, IAM roles, and a bunch of supporting resources. Around 30 to 40 resources total. Scan through the plan to make sure nothing looks wrong.

terraform apply

Type yes and go grab a coffee. The Aurora cluster alone takes 5 to 10 minutes to provision. ElastiCache Serverless is usually faster, about 2 to 3 minutes. The rest finishes in under a minute each. Total time from apply to everything being live is roughly 10 to 15 minutes.

Run Migrations

Once apply completes, run your database migrations through the artisan Lambda:

aws lambda invoke \
  --function-name hrr_laravel_artisan \
  --payload '{"cli": "migrate --force"}' \
  --cli-binary-format raw-in-base64-out \
  output.json

cat output.json

You should see the familiar migration output. If the Lambda can't reach Aurora, you'll get a connection timeout error, which means your VPC security groups or subnet configuration needs debugging. Double check that the Lambda security group allows outbound traffic to the database security group on port 5432.

Hit the Endpoint

Terraform should output your API Gateway URL. Something like https://abc123def.execute-api.us-east-1.amazonaws.com.

curl https://abc123def.execute-api.us-east-1.amazonaws.com

I still remember the first time I saw the Laravel welcome page come back from a curl to a Lambda URL. Took me a second to process that there was no server running anywhere. No EC2, no container, no process manager. Just PHP running inside Lambda's microVM for the duration of a single request and then disappearing. Pretty wild.

CI/CD

For ongoing deployments, I keep it simple with GitHub Actions. The workflow looks roughly like this: checkout code, run composer install --no-dev, cache routes and config, zip the app, run terraform apply with -auto-approve, then invoke the artisan Lambda to run migrations. You store your AWS credentials and Terraform state backend config as GitHub secrets.

The full CI/CD setup is its own topic. Running the commands manually is fine to start.

Cost Breakdown and Wrapping Up

Let's put real numbers on three ways to run this app. Roughly 100,000 requests per month, a handful of background jobs, standard session and cache usage.

If you read my earlier Terraform post about deploying Laravel on EC2, that setup uses a t3.medium at about (30 per month just for the instance. Then you tack on RDS for Postgres at )15 to (25 depending on the size, plus ElastiCache for Redis at )12 to (15. So you land somewhere in the )57 to $70 range. The thing that always bugged me about that approach is you're paying the same amount whether your app is handling traffic or sitting idle at 3am on a Tuesday. And if you suddenly get a spike, you're either overprovisioned already (wasting money the rest of the time) or scrambling to bolt on auto-scaling after things start falling over.

Now, Laravel Vapor is interesting because it genuinely is convenient. The subscription runs (399 per year, so about )33 per month before AWS charges even enter the picture. On top of that you're still paying for Lambda, RDS, ElastiCache, and all the usual AWS stuff at normal rates. For our 100K request scenario the AWS portion might run (15 to )25, which puts your total around (48 to )58 per month. I actually considered Vapor for a while. It's well-integrated with Laravel and the deployment experience is smooth. But I kept coming back to the fact that I was paying a monthly premium for something I could wire up myself with Terraform and a weekend of focused work.

So what does this Bref and Terraform setup actually cost? There's no subscription, which is already a win. Lambda's free tier covers 1 million requests and 400,000 GB-seconds per month. Our 100K requests barely registers. Even past the free tier, Lambda pricing for this kind of workload comes out to roughly (2 to )5 per month. The biggest line item is Aurora Serverless v2. At 0.5 ACUs minimum, you're looking at about (43 per month at the floor. I'll be honest, that one stung a little when I first saw the bill. If you swapped Aurora for Neon or Supabase's free tier, you could cut that to nearly zero for low-traffic apps. ElastiCache Serverless for light cache and session usage runs about )3 to (5 per month. Then there's the NAT Gateway, the one that catches people off guard at roughly )32 per month. API Gateway HTTP API at 100K requests is under (1, and S3 plus SQS are negligible. All together with Aurora and the NAT Gateway, you're at )80 to (85 per month. But if your app doesn't need to call external APIs from Lambda, you can ditch the NAT Gateway. Pair that with a third-party Postgres provider and you drop to )10 to $15 per month. The real advantage shows up as traffic grows, because you're only paying for what you use instead of keeping idle capacity warm.

So which one should you pick? Honestly, it depends. I went with Bref and Terraform because I wanted full control and didn't mind spending time on infrastructure. If that sounds like too much hassle and you'd rather just vapor deploy and move on, Vapor is perfectly fine for that. And funny enough, if your traffic is steady and predictable, a plain EC2 instance might actually work out cheaper than any of the serverless options. There's no universal answer here.

This is Part 1 of my Serverless Laravel series. In Part 2, I'll cover running Laravel on Fargate and App Runner for workloads that don't fit Lambda's constraints. Think long-running processes, WebSocket connections, or apps that need more than 10GB of ephemeral storage. Different tools for different problems.

Hope you enjoyed this one. If you're running Laravel on Lambda or thinking about making the switch, I'd love to hear how it's going. Drop me a message on Twitter/X and let's chat about it.

Originally published at harun.dev.

Building a Multi-Tenant Billing System on AWS Lambda with Stripe

Harun R. Rayhan — Mon, 19 Oct 2026 04:47:37 +0000

In the last post I walked through the multi-tenant auth layer running in front of my Fastify API on AWS Lambda. JWT tokens, hashed API keys, tenant context decorated onto every request. That solved the "who is this and which tenant do they belong to" problem.

Then my second customer asked about pricing tiers. The third one wanted to upgrade themselves without emailing me. That's when I hit the wall every SaaS hits eventually. You need billing. Not someday, right now. And it has to know which plan a tenant is on before the request even reaches your handler, because some routes are pro-only and some are free-forever.

So I bolted Stripe onto the existing auth layer. One Stripe Customer per tenant. Plans modeled as Products and Prices in Stripe. Subscription state mirrored locally so the auth-then-billing pipeline can decide in microseconds whether this request is allowed. Webhooks for the source of truth. Checkout sessions for upgrades. The customer portal for everything else.

This post walks through the whole thing. The Fastify billing plugin that sits next to the auth plugin, the webhook handler with the raw-body gotcha that ate two hours of my life, the checkout flow, and the bits I got wrong before I got them right. The code is sanitized but it's the same code running in my production stack.

Photo by Towfiqu barbhuiya on Pexels

Architecture overview

I kept the model boring on purpose. One Stripe Customer per tenant, one active subscription per customer, plans stored as Stripe Products with one or more Prices. The local database mirrors the subscription state so every request can read the plan without calling Stripe.

                  ┌──────────────────────┐
                  │   Tenant frontend    │
                  └──────────┬───────────┘
                             │
                             ▼
        ┌────────────────────────────────────────┐
        │  API Gateway / Lambda URL              │
        │  ┌──────────────────────────────────┐  │
        │  │ Fastify app                      │  │
        │  │  1. auth-plugin   (tenantId)     │  │
        │  │  2. billing-plugin (plan)        │  │
        │  │  3. routes  (request.plan check) │  │
        │  └──────────────────────────────────┘  │
        └────────────┬───────────────┬───────────┘
                     │               │
                     ▼               ▼
              ┌────────────┐   ┌────────────┐
              │ Postgres   │   │  Stripe    │
              │ hrr_tenants│◀──│  webhooks  │
              └────────────┘   └────────────┘

Stripe is the source of truth for subscription status. The local hrr_tenants row is a cache that gets reconciled on every webhook event. The auth plugin runs first and resolves request.tenantId. The billing plugin runs right after and decorates request.plan and request.billingPeriodEnd. By the time a route handler runs, both pieces of context are already sitting on the request.

The reason billing comes after auth is obvious once you say it out loud. You can't load a tenant's plan if you don't know which tenant. But I'll spell it out because Fastify plugin order is load-bearing here. Get it backwards and the billing plugin reads undefined for the tenant ID, then silently treats every request as anonymous. Ask me how I know.

Photo by Ivan S on Pexels

Database schema

The hrr_tenants table from the auth post grows a few new columns. Stripe gives you stable identifiers (cus_*, sub_*) that I store directly. I reuse the plan column as the cached subscription tier. billing_period_end lets the API show "renews on" without a Stripe round trip.

ALTER TABLE hrr_tenants
  ADD COLUMN stripe_customer_id     VARCHAR(64),
  ADD COLUMN stripe_subscription_id VARCHAR(64),
  ADD COLUMN billing_period_end     TIMESTAMPTZ,
  ADD COLUMN billing_status         VARCHAR(32) NOT NULL DEFAULT 'inactive'
    CHECK (billing_status IN ('inactive', 'active', 'past_due', 'canceled', 'trialing'));

-- The plan column already exists from the auth post,
-- but widen the check constraint to match Stripe products.
ALTER TABLE hrr_tenants
  DROP CONSTRAINT hrr_tenants_plan_check;

ALTER TABLE hrr_tenants
  ADD CONSTRAINT hrr_tenants_plan_check
  CHECK (plan IN ('free', 'starter', 'pro'));

-- Webhook idempotency table. One row per Stripe event ID we've processed.
CREATE TABLE hrr_billing_events (
  event_id     VARCHAR(64) PRIMARY KEY,
  type         VARCHAR(64) NOT NULL,
  tenant_id    UUID REFERENCES hrr_tenants(id),
  processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_hrr_tenants_stripe_customer
  ON hrr_tenants(stripe_customer_id);
CREATE INDEX idx_hrr_tenants_stripe_subscription
  ON hrr_tenants(stripe_subscription_id);

The two indexes matter. Webhook handlers look up tenants by stripe_subscription_id (when a subscription updates) or by stripe_customer_id (when a customer event fires). Without indexes you're scanning the whole tenants table on every webhook. Sounds fine until you have ten thousand tenants and Stripe replays a backlog.

The hrr_billing_events table is my idempotency log. Stripe will absolutely deliver the same event twice. Sometimes more. The PRIMARY KEY on event_id turns "did I already process this" into a single insert that either succeeds (first time) or fails with a duplicate key error (replay). I'll show how that's wired up in the webhook section.

Free is the odd one out. Free tenants have no stripe_subscription_id and no stripe_customer_id until they upgrade for the first time. I still create the customer at signup though. More on why in the next section.

Photo by Tima Miroshnichenko on Pexels

Creating a Stripe customer on signup

I went back and forth on this one. Lazy customer creation (only call Stripe when the tenant first tries to upgrade) feels cleaner. But it means the upgrade path has an extra failure mode and an extra latency hit at the worst possible time. Eager creation (call Stripe at signup) costs one API call per signup and removes a whole class of problems later.

I went eager. The signup handler creates the Stripe customer right after creating the tenant row, in the same request.

import Stripe from 'stripe';

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!, {
  apiVersion: '2024-06-20',
  // Stripe SDK retries on its own. Lambda timeouts are short,
  // so cap the timeout per attempt.
  timeout: 8_000,
  maxNetworkRetries: 2,
});

interface SignupBody {
  tenantName: string;
  ownerEmail: string;
}

app.post<{ Body: SignupBody }>('/api/signup', async (request, reply) => {
  const { tenantName, ownerEmail } = request.body;

  // 1. Create the tenant row first so we have a tenant ID.
  const tenant = await fastify.pg.one<{ id: string }>(
    `INSERT INTO hrr_tenants (name, plan, billing_status)
     VALUES ($1, 'free', 'inactive')
     RETURNING id`,
    [tenantName],
  );

  // 2. Create the Stripe customer. The idempotency key is keyed on
  //    the tenant ID so retries from the client (or Lambda)
  //    don't create duplicate customers in Stripe.
  const customer = await stripe.customers.create(
    {
      email: ownerEmail,
      name: tenantName,
      metadata: { tenant_id: tenant.id },
    },
    { idempotencyKey: `hrr_signup_${tenant.id}` },
  );

  // 3. Persist the customer ID back on the tenant.
  await fastify.pg.query(
    `UPDATE hrr_tenants
     SET stripe_customer_id = $1
     WHERE id = $2`,
    [customer.id, tenant.id],
  );

  return reply.status(201).send({ tenantId: tenant.id });
});

Two things worth pointing out. The idempotencyKey on stripe.customers.create is the unsexy hero here. If the Lambda times out after Stripe creates the customer but before I save the ID, the client retries, and I'd end up with a second customer in Stripe and no way to clean it up. With the idempotency key, Stripe returns the same customer object on retry. Free duplicate protection.

The metadata.tenant_id is also load-bearing. Every Stripe object I create from this point on (customers, subscriptions, checkout sessions, invoices) gets stamped with tenant_id in metadata. When a webhook arrives, that metadata gives me a backup path to resolve the tenant if the customer ID lookup ever misses for whatever reason.

The order matters. Tenant row first, then Stripe customer, then update the row. If step 2 fails, I have a tenant with no Stripe customer, which is a recoverable state (the next signup retry or a backfill job can fix it). If I'd created the Stripe customer first and then the tenant row failed, I'd have an orphaned Stripe customer with no local tenant to bind it to. That's harder to clean up.

Photo by Pixabay on Pexels

The billing Fastify plugin

This piece does the real work on every request. It runs after the auth plugin, reads request.tenantId, and decorates request.plan plus request.billingPeriodEnd so route handlers can do plan checks without writing any boilerplate.

import fp from 'fastify-plugin';
import type { FastifyInstance, FastifyRequest } from 'fastify';

type Plan = 'free' | 'starter' | 'pro';

declare module 'fastify' {
  interface FastifyRequest {
    plan: Plan;
    billingPeriodEnd: Date | null;
    billingStatus: 'inactive' | 'active' | 'past_due' | 'canceled' | 'trialing';
  }
}

interface CachedBilling {
  plan: Plan;
  billingPeriodEnd: Date | null;
  billingStatus: FastifyRequest['billingStatus'];
  cachedAt: number;
}

// Per-container cache. Lambda reuses warm containers, so this gives us
// near-zero overhead for repeat requests from the same tenant.
const billingCache = new Map<string, CachedBilling>();
const CACHE_TTL_MS = 30_000;

async function billingPlugin(fastify: FastifyInstance) {
  fastify.decorateRequest('plan', 'free');
  fastify.decorateRequest('billingPeriodEnd', null);
  fastify.decorateRequest('billingStatus', 'inactive');

  fastify.addHook('preHandler', async (request: FastifyRequest) => {
    // Skip routes that didn't run the auth plugin (e.g. /health, webhooks).
    if (!request.tenantId) return;

    const cached = billingCache.get(request.tenantId);
    if (cached && Date.now() - cached.cachedAt < CACHE_TTL_MS) {
      request.plan = cached.plan;
      request.billingPeriodEnd = cached.billingPeriodEnd;
      request.billingStatus = cached.billingStatus;
      return;
    }

    const row = await fastify.pg.oneOrNone<{
      plan: Plan;
      billing_period_end: Date | null;
      billing_status: FastifyRequest['billingStatus'];
    }>(
      `SELECT plan, billing_period_end, billing_status
       FROM hrr_tenants
       WHERE id = $1`,
      [request.tenantId],
    );

    const plan = row?.plan ?? 'free';
    const billingPeriodEnd = row?.billing_period_end ?? null;
    const billingStatus = row?.billing_status ?? 'inactive';

    request.plan = plan;
    request.billingPeriodEnd = billingPeriodEnd;
    request.billingStatus = billingStatus;

    billingCache.set(request.tenantId, {
      plan,
      billingPeriodEnd,
      billingStatus,
      cachedAt: Date.now(),
    });
  });
}

export default fp(billingPlugin, {
  name: 'billing-plugin',
  dependencies: ['auth-plugin', 'database-plugin'],
});

The dependencies array is what enforces plugin ordering. Fastify will refuse to start if auth-plugin isn't registered first. That's exactly the kind of bug I'd rather catch at boot than at 3am via a "why is every request unauthenticated" alert.

The cache is a per-Lambda-container Map with a 30 second TTL. Not Redis. Not DynamoDB. Just a Map. Lambda reuses warm containers for several minutes, so a busy tenant hits the cache for almost every request after the first one. When the webhook fires and a tenant's plan changes, the worst case is 30 seconds of stale data before the next read pulls fresh state. For billing that's totally fine. If you need stricter consistency you can bust the cache from the webhook handler, but I haven't needed to.

I deliberately do NOT call Stripe from this plugin. Every request would hit Stripe's API otherwise, which blows your rate limit and adds 100ms of latency to every authenticated request. The webhook handler updates the local row, and the plugin reads from the local row. Stripe is the source of truth, the database is the read path.

Photo by Onur Yumlu on Pexels

Enforcing plan limits in route handlers

With request.plan on every authenticated request, plan gating becomes a one-liner. I wrap it in a tiny helper so the intent is obvious in route code.

import type { FastifyRequest, FastifyReply } from 'fastify';

type Plan = 'free' | 'starter' | 'pro';

const PLAN_RANK: Record<Plan, number> = {
  free: 0,
  starter: 1,
  pro: 2,
};

export function requirePlan(minimum: Plan) {
  return async (request: FastifyRequest, reply: FastifyReply) => {
    if (PLAN_RANK[request.plan] < PLAN_RANK[minimum]) {
      return reply.status(402).send({
        error: 'payment_required',
        message: `This endpoint requires the ${minimum} plan or higher.`,
        currentPlan: request.plan,
        upgradeUrl: '/api/billing/checkout',
      });
    }
  };
}

Then in a pro-only route.

app.post('/api/exports/csv', {
  preHandler: [app.authenticate, requirePlan('pro')],
  handler: async (request, reply) => {
    const csv = await generateExport(request.tenantId);
    return reply
      .header('content-type', 'text/csv')
      .send(csv);
  },
});

I return 402 Payment Required deliberately. It's an underused HTTP status that means exactly this: the request is well-formed and authenticated, but your account doesn't have access. Frontend code can branch on 402 specifically and route the user straight to the upgrade page. Way better UX than a generic 403.

The upgradeUrl in the body is a tiny detail that paid for itself. Frontend devs working on this API don't have to remember which endpoint creates a checkout session. The error response tells them. One less thing to look up.

Photo by Quentin Guiot on Pexels

Stripe webhooks on Lambda

Webhooks are the source of truth for subscription state. When a tenant upgrades, downgrades, fails a payment, or cancels, Stripe sends an event to my webhook endpoint. The handler updates hrr_tenants and that's it. Everything downstream reads from the local row.

The first thing that bit me, and I mean really bit me, is that Stripe's signature verification needs the raw request body. Not the parsed JSON. The exact bytes Stripe sent. Fastify's default content-type parser eats the body and hands you a parsed object, which means stripe.webhooks.constructEvent() throws "No signatures found matching the expected signature for payload" no matter what you do.

The fix is registering a custom content-type parser that captures the raw buffer for the webhook route only.

import Stripe from 'stripe';
import type { FastifyInstance, FastifyRequest } from 'fastify';

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!, {
  apiVersion: '2024-06-20',
});

const WEBHOOK_SECRET = process.env.STRIPE_WEBHOOK_SECRET!;

declare module 'fastify' {
  interface FastifyRequest {
    rawBody?: Buffer;
  }
}

export async function registerWebhookRoute(app: FastifyInstance) {
  // Custom parser that keeps the raw buffer alongside the parsed JSON.
  app.addContentTypeParser(
    'application/json',
    { parseAs: 'buffer' },
    (request, body, done) => {
      (request as FastifyRequest).rawBody = body as Buffer;
      try {
        const json = body.length ? JSON.parse(body.toString('utf8')) : {};
        done(null, json);
      } catch (err) {
        done(err as Error, undefined);
      }
    },
  );

  app.post('/webhooks/stripe', async (request, reply) => {
    const signature = request.headers['stripe-signature'];
    if (typeof signature !== 'string' || !request.rawBody) {
      return reply.status(400).send({ error: 'missing_signature' });
    }

    let event: Stripe.Event;
    try {
      event = stripe.webhooks.constructEvent(
        request.rawBody,
        signature,
        WEBHOOK_SECRET,
      );
    } catch (err) {
      request.log.warn({ err }, 'webhook signature verification failed');
      return reply.status(400).send({ error: 'invalid_signature' });
    }

    // Idempotency. Insert before processing. If the event was already
    // handled, the unique violation tells us to skip.
    try {
      await app.pg.query(
        `INSERT INTO hrr_billing_events (event_id, type, tenant_id)
         VALUES (\(1, \)2, $3)`,
        [event.id, event.type, extractTenantId(event)],
      );
    } catch (err: any) {
      if (err.code === '23505') {
        // duplicate event, already processed
        return reply.status(200).send({ received: true, deduped: true });
      }
      throw err;
    }

    switch (event.type) {
      case 'customer.subscription.created':
      case 'customer.subscription.updated':
        await handleSubscriptionChange(app, event.data.object as Stripe.Subscription);
        break;
      case 'customer.subscription.deleted':
        await handleSubscriptionDeleted(app, event.data.object as Stripe.Subscription);
        break;
      default:
        request.log.info({ type: event.type }, 'unhandled webhook event');
    }

    return reply.status(200).send({ received: true });
  });
}

function extractTenantId(event: Stripe.Event): string | null {
  const obj = event.data.object as { metadata?: Record<string, string> };
  return obj.metadata?.tenant_id ?? null;
}

The two handlers do the actual database work.

const PRICE_TO_PLAN: Record<string, 'starter' | 'pro'> = {
  [process.env.STRIPE_PRICE_STARTER!]: 'starter',
  [process.env.STRIPE_PRICE_PRO!]: 'pro',
};

async function handleSubscriptionChange(
  app: FastifyInstance,
  sub: Stripe.Subscription,
) {
  const priceId = sub.items.data[0]?.price.id;
  const plan = priceId ? PRICE_TO_PLAN[priceId] ?? 'free' : 'free';
  const periodEnd = new Date(sub.current_period_end * 1000);

  await app.pg.query(
    `UPDATE hrr_tenants
     SET plan                   = $1,
         stripe_subscription_id = $2,
         billing_period_end     = $3,
         billing_status         = $4
     WHERE stripe_customer_id   = $5`,
    [plan, sub.id, periodEnd, sub.status, sub.customer],
  );
}

async function handleSubscriptionDeleted(
  app: FastifyInstance,
  sub: Stripe.Subscription,
) {
  await app.pg.query(
    `UPDATE hrr_tenants
     SET plan                   = 'free',
         stripe_subscription_id = NULL,
         billing_period_end     = NULL,
         billing_status         = 'canceled'
     WHERE stripe_customer_id   = $1`,
    [sub.customer],
  );
}

The webhook route does NOT use the auth plugin. Stripe doesn't send your JWT, it sends a signature header. The signature is the auth. That's why the webhook lives on its own path that the auth plugin's preHandler doesn't apply to. If you accidentally guard the webhook route with app.authenticate, every Stripe event returns 401 and your subscription state goes stale fast.

API Gateway and Lambda Function URLs both work for this. I use a Lambda Function URL because it's simpler and cheaper for a single-purpose endpoint, but make sure your Function URL is configured to NOT buffer or transform the body. AWS_PROXY integration on API Gateway also works. Just verify that binary body passthrough is on.

Photo by RDNE Stock project on Pexels

Checkout session flow

When a tenant clicks "Upgrade to Pro" in the dashboard, the frontend calls my API, my API creates a Stripe Checkout Session, and the frontend redirects to the URL Stripe returns. The session knows which tenant it belongs to because the customer ID is already stored on hrr_tenants.

interface CheckoutBody {
  priceId: string; // e.g. 'price_1Abc...' for the pro tier
}

app.post<{ Body: CheckoutBody }>('/api/billing/checkout', {
  preHandler: [app.authenticate],
  handler: async (request, reply) => {
    const tenant = await fastify.pg.one<{
      id: string;
      stripe_customer_id: string | null;
    }>(
      `SELECT id, stripe_customer_id
       FROM hrr_tenants
       WHERE id = $1`,
      [request.tenantId],
    );

    if (!tenant.stripe_customer_id) {
      return reply.status(400).send({ error: 'no_stripe_customer' });
    }

    const session = await stripe.checkout.sessions.create(
      {
        mode: 'subscription',
        customer: tenant.stripe_customer_id,
        line_items: [
          { price: request.body.priceId, quantity: 1 },
        ],
        success_url: `${process.env.APP_URL}/billing/success?session_id={CHECKOUT_SESSION_ID}`,
        cancel_url: `${process.env.APP_URL}/billing`,
        metadata: { tenant_id: tenant.id },
        subscription_data: {
          metadata: { tenant_id: tenant.id },
        },
      },
      { idempotencyKey: `hrr_checkout_\({tenant.id}_\){Date.now()}` },
    );

    return reply.send({ url: session.url });
  },
});

A few details worth pointing out. Passing customer: tenant.stripe_customer_id means Stripe doesn't ask the tenant for their email again, doesn't create a new customer, and ties the resulting subscription to the customer I already have. Exactly what I want.

I set metadata.tenant_id at both levels: the checkout session itself and the subscription_data. The session metadata is useful for checkout.session.completed events. The subscription metadata is what shows up on every customer.subscription.updated event, which is what I actually rely on in the webhook handler.

The success URL has {CHECKOUT_SESSION_ID} in it, which Stripe replaces with the real session ID at redirect time. My success page calls a small API endpoint that retrieves the session and shows a "you're now on the Pro plan" confirmation. The actual plan change happens via the webhook, not via the success page. The user could close the tab before hitting success, and the upgrade still needs to complete.

Photo by Nataliya Vaitkevich on Pexels

The billing portal

Stripe gives you a hosted customer portal for free. Tenants can update their card, see their invoices, cancel their subscription, and switch plans without me writing a single line of UI for any of it. Wiring it up is one API call.

app.post('/api/billing/portal', {
  preHandler: [app.authenticate],
  handler: async (request, reply) => {
    const tenant = await fastify.pg.one<{
      stripe_customer_id: string | null;
    }>(
      `SELECT stripe_customer_id
       FROM hrr_tenants
       WHERE id = $1`,
      [request.tenantId],
    );

    if (!tenant.stripe_customer_id) {
      return reply.status(400).send({ error: 'no_stripe_customer' });
    }

    const session = await stripe.billingPortal.sessions.create({
      customer: tenant.stripe_customer_id,
      return_url: `${process.env.APP_URL}/billing`,
    });

    return reply.send({ url: session.url });
  },
});

The frontend hits this endpoint, gets a URL back, and redirects. That's it. Configure the portal once in the Stripe dashboard (which products customers can switch to, whether to allow cancellation immediately or at period end, etc.) and it just works.

Every action a tenant takes in the portal fires a webhook. Plan switch fires customer.subscription.updated. Cancel fires customer.subscription.deleted or customer.subscription.updated with cancel_at_period_end: true depending on your portal settings. As long as the webhook handler is solid, the portal is basically a free feature you didn't have to build.

Photo by iMin Technology on Pexels

Testing webhooks locally

You can't test webhooks against a localhost endpoint without help. Stripe can't reach your laptop. The Stripe CLI bridges that gap and is by far the fastest local feedback loop I've found.

# Install once
brew install stripe/stripe-cli/stripe

# Log in (opens a browser, pairs your CLI to your Stripe account)
stripe login

# Forward live webhook events to your local Fastify server
stripe listen --forward-to localhost:3000/webhooks/stripe

When the listener starts, it prints a webhook signing secret. Set that as STRIPE_WEBHOOK_SECRET for your local process. The CLI replays every event your Stripe account sees in real time, against your laptop, with valid signatures. Trigger a checkout session, complete it with a test card, and watch customer.subscription.created land in your handler within seconds.

For deterministic flows there's also stripe trigger.

stripe trigger customer.subscription.updated
stripe trigger customer.subscription.deleted

These send synthetic events that exercise your handler without you having to click through a checkout flow. Great for testing failure paths.

Test clocks are the other underrated tool. They let you simulate the passage of time on a subscription so you can watch what happens at renewal, when an invoice fails, when a trial ends. The clock is attached to a customer and you can advance it forward by days or months with one API call. If you have any logic that fires on invoice.payment_failed or trial expiration, test clocks are how you exercise it without waiting weeks of real time.

# Create a test clock starting now
stripe test_helpers test_clocks create --frozen-time=$(date +%s)

# Advance one month
stripe test_helpers test_clocks advance \
  --id=clock_xxx \
  --frozen-time=$(date -v +1m +%s)

Photo by Godfrey Atima on Pexels

What I got wrong

A few things tripped me up before I figured them out. Worth listing so you don't lose the same hours.

The raw body issue with Fastify's JSON parser. This one kept me up for two hours. The Stripe SDK kept throwing "No signatures found matching the expected signature for payload" and I was 100% sure I had the right webhook secret. The signing secret was right. The header was right. The problem was that Fastify had already parsed the body into an object, and when the SDK serialized that object to verify the signature, the byte-for-byte comparison failed because object key order isn't guaranteed and JSON whitespace was different. Use addContentTypeParser with parseAs: 'buffer' and stash the raw bytes on the request. Then pass those raw bytes to stripe.webhooks.constructEvent. Don't pass request.body. Don't pass JSON.stringify(request.body). Pass the buffer.

Idempotency on webhook handlers. Stripe will deliver the same event twice. Sometimes more. Network blips, your handler timing out, Stripe replaying after a deploy, doesn't matter, it happens. Without idempotency you'll process the same upgrade twice, which usually doesn't break anything because the database update is idempotent on its own. But it definitely breaks anything where you charge a credit on plan change, send a "welcome to Pro" email, or write to an analytics pipeline. The hrr_billing_events table with a PRIMARY KEY on event_id is the simplest solution I could come up with. Insert before processing. If the insert fails with a duplicate key, return 200 and skip. Costs nothing, prevents a whole class of weird bugs.

Eager vs lazy customer creation. I tried lazy creation first because it felt cleaner. Don't call Stripe until the tenant actually needs to upgrade. The annoying part is that "needs to upgrade" is the absolute worst time for a Stripe API call to fail. The user has clicked "Upgrade", they've entered their card, and now my backend has to do two Stripe calls (create customer, create checkout session) before they see Stripe's checkout page. Either of those failing is a terrible upgrade experience. Eager creation moves the only Stripe call to signup, which is way less time-sensitive, and means the upgrade flow is one Stripe call instead of two. Worth the unused customers in Stripe.

Caching plan info per Lambda container. I started with no cache and hit the database on every authenticated request just to read the plan. Fine until I realized a busy tenant might do 50 requests per second and every single one was doing a redundant primary-key lookup. The 30-second per-container Map cache cut my database load by something like 80% for read traffic. The trade-off is up to 30 seconds of stale plan data after an upgrade, which is invisible to anyone who isn't deliberately watching for it.

Plan changes from the customer portal don't always fire what you expect. When a tenant downgrades through the portal, you might get customer.subscription.updated with cancel_at_period_end: true instead of customer.subscription.deleted. The subscription doesn't actually end until the period rolls over, at which point you get the deleted event. Handle both. I had a bug for two days where downgrades didn't take effect because I was only listening for deleted. Not my finest debugging hour.

Photo by Daniil Komov on Pexels

Wrapping up

Stripe billing on Lambda isn't hard, but there are about five things that need to be right or nothing works. Raw bodies for webhook signatures. Idempotency on event handling. Customer creation early so the upgrade path is fast. A local cache for plan reads so you don't hammer your database. Plugin order so billing context is on the request before any handler runs.

Get those right and the rest is glue. The Fastify plugin pattern from the auth post extends naturally to billing. request.tenantId from auth, request.plan from billing, route handlers stay clean.

Hope you enjoyed this walkthrough. If you're building something similar or have questions about any of these patterns, drop a comment below or reach out on Twitter.

Follow me on Twitter/X for more posts about AWS, serverless architecture, and building SaaS products.

Photo by ThisIsEngineering on Pexels

Originally published at harun.dev.

Top 8 Load Balancer Algorithms You Should Know About

Harun R. Rayhan — Thu, 15 Oct 2026 07:00:28 +0000

A health check was green on all three boxes. CPU on the first one was pinned. The office NAT had hashed onto that one server, and every laptop in the building followed it there.

The algorithm was doing what it was told. I had just never watched the requests land.

There are 8 below. Each one gets its own figure. It starts sending when the figure is on screen, and it stops when you scroll away. Speed is under the picture. Reset clears the lap and starts it again.

All eight at a glance. Green is the server that gets the next request under each rule.1. Round robin

Request 1 goes to A, 2 to B, 3 to C, 4 back to A. nginx uses this unless you say otherwise. HAProxy calls it balance roundrobin.

▶ Try the interactive demo on harun.dev

Round robin does not look at the servers. It keeps a list, and it walks that list. Request 1 is A, request 2 is B, request 3 is C. Request 4 is A again, because the list ran out.

How one lap lands

Request	Server	Why that server
1	A	First name in the list
2	B	Next name
3	C	Next name
4	A	The list ended, so it starts over
5	B	Same seat as request 2
6	C	Same seat as request 3

Six requests is two full laps, so A, B, and C each hold two. Request 7 starts a third lap on A. Nothing in that walk asks whether A is already busy.

It is fair when every request costs about the same. It is a bad fit when one request uploads a video and the next one hits /health. The video stays on A while B and C keep taking cheap checks.

What round robin decides

Question	Answer
What it counts	The next name in a fixed list
What it ignores	CPU, open connections, response time, and request size
nginx	This is the default. You write a directive only when you want a different algorithm
HAProxy	`balance roundrobin`
Same client, next request	Can land on a different server. This is not sticky
Equal machines, short requests	This is enough
One huge upload, then health checks	The upload stays where it landed. The cheap checks keep walking the list
A server that is slow	It still gets its turn
Against least connections	Least connections skips a busy box. Round robin does not

2. Weighted round robin

Use this when the machines are different sizes. A c7g.xlarge and a t4g.small should not take the same count. nginx writes it as weight=5 on the upstream. HAProxy writes it as weight on the server line.

▶ Try the interactive demo on harun.dev

The lap is fixed. A takes five, then B takes one, then C takes one, then A starts again. A server that finishes early does not get to jump the queue.

3. Least connections

The next request goes to the server with the fewest open requests. Long requests pile up, so the balancer stops sending them more work.

▶ Try the interactive demo on harun.dev

nginx calls this least_conn. HAProxy calls it leastconn. An Application Load Balancer does not give you a round-robin dropdown. It uses least outstanding requests, which is this idea: in-flight requests, not a completed count.

4. Weighted least connections

Least connections on its own still treats a large box and a small box as equals. Weights fix that. One open request on a weight-5 server is less busy than one open request on a weight-1 server.

▶ Try the interactive demo on harun.dev

HAProxy writes this as balance leastconn plus a weight on each server. The picture still leans on A, and a server that drains its queue can take the next request before the lap says so.

5. Least response time

Connections are not the whole story. A server with two open requests that answers in 12 ms can be a better pick than an idle server that answers in 40 ms.

▶ Try the interactive demo on harun.dev

HAProxy calls this least response time. It multiplies open connections by a recent response time and picks the smaller product. I use it when the boxes are the same size on paper and one of them is actually slow.

6. IP hash

The same client IP keeps landing on the same server. That is sticky routing without a cookie.

▶ Try the interactive demo on harun.dev

It is also how one office NAT melts a single box. Every laptop shares the public address, so they all hash together. nginx calls this ip_hash. A Network Load Balancer does not hash the IP for the life of a laptop. It picks a target for a flow, and that choice sticks for the life of the connection.

7. Consistent hashing

IP hash sticks a client to a server. Consistent hashing sticks a key to a server, and it tries not to move every key when you add or remove a node.

▶ Try the interactive demo on harun.dev

Envoy calls a form of this ring hash. You meet it in caches and in meshes, where the point is that key A stays on the same box for the next request. Maglev, which Google uses in front of a lot of traffic, is the same family.

8. Power of two choices

Pick two servers at random. Send the request to the one with less load. You skip a scan of the whole pool, and you still dodge the server that is already buried.

▶ Try the interactive demo on harun.dev

Plain random is the version that does not take the second look. Power of two is the one worth remembering. Envoy and a lot of service meshes use a form of it. You will not find a radio button with this name on an Application Load Balancer.

What I check before I pick one

If the requests are short and the machines match, round robin is enough. If the machines differ and the requests are short, add weights. If some requests stay open, use least connections, and add weights if the machines differ too. If one box is slow even when it looks idle, look at response time.

If you need a client to stay on one box, know that IP hash will also glue a whole NAT to that box. If you need a cache key to stay put, use consistent hashing. Power of two is what I want inside a mesh, where no human is going to retune weights at 2am.

Hope you enjoyed this one. If you want to argue about sticky sessions versus a shared store, find me on X at https://x.com/harundotdev.

Originally published at harun.dev. The full interactive version lives there. Follow me on X or read my bio.

How I Review Terraform and Lambda PRs with AI Before They Merge

Harun R. Rayhan — Wed, 14 Oct 2026 14:57:46 +0000

If my earlier posts were about how I write infrastructure and how I build AI-assisted workflows, this one is about the gate right before merge.

That distinction matters. Writing code with AI is useful. Reviewing production changes with AI is where the real value shows up for me. The problem is not getting a first draft fast. The problem is making sure the change is still safe when it touches IAM, Terraform state, Lambda runtime settings, or anything that can quietly cost money or break production.

So I started using AI as a first-pass reviewer for Terraform and Lambda pull requests. Not as the decision maker. Not as the person who approves the merge. Just as the reviewer that helps me catch the obvious and semi-obvious issues before I spend my own time on the full manual pass.

That keeps the review fast, but it also keeps the final responsibility where it belongs, with me.

This post is the workflow I trust today.

Short version: AI reviews the diff, I review the risk.

Why This Is a Different Post From My Other AI and AWS Articles

I already write about AI in AWS codebases, Terraform workflows, and agentic DevOps routines. This post is not another version of that same story.

The earlier posts focus on one of two things:

how I use AI to help build infrastructure
how I structure the workflow around the codebase

This one is narrower. It is about reviewing a production change before it merges.

That narrower scope makes it more useful in practice, because the questions are more specific:

Did this IAM policy get broader?
Did someone accidentally make a Lambda slower, hotter, or more expensive?
Did Terraform replace something that should not be replaced?
Did we open a networking path we did not mean to open?
Did the PR forget tests, docs, or rollback notes?

That is the kind of review AI is actually good at helping with.

What I Feed the Reviewer

I do not ask the model to guess from thin air. I give it the actual change surface.

Typically that means some mix of:

the pull request diff
the Terraform plan output
the files touched by the change
the environment or service context
any recent deployment or incident notes that matter

The quality of the review depends on the input. If I give it a sloppy prompt, I get a sloppy review. If I give it the real plan and the real diff, it can usually spot the things I want it to spot.

One caution: redact before you paste. Plan output can include values that are not marked sensitive, so I strip anything I would not put in a public issue.

I want the reviewer to answer a simple question: what would make this change risky to merge?

The Checks I Care About Most

For Terraform and Lambda work, I care about a small set of risk categories more than anything else.

1. IAM and permissions

This is the first thing I look at because permission creep is easy to miss in a diff. A policy can go from narrow to broad with one extra wildcard or one extra resource pattern.

I want AI to flag:

new wildcards in actions or resources
roles that gained access to more services than before
policies that look temporary but are actually permanent
anything that could grant access across environments

2. Lambda runtime and cost impact

Lambda changes can look tiny and still affect cost or reliability.

I want the reviewer to notice things like:

memory jumps that are probably unnecessary
timeouts that are too short or too generous
concurrency changes that might cause throttle or burst issues
extra cold-start risk from dependencies or packaging changes

3. Terraform replacement risk

This is where production changes can become unpleasant. A harmless-looking refactor can trigger a destroy-and-recreate path if someone changes the wrong attribute.

I want the model to call out:

resources marked for replacement
stateful resources with a destructive path
name changes that can force recreation
security groups, targets, or buckets that could disappear briefly

4. Networking and exposure

Anything that changes public access, private access, VPC wiring, or routing deserves extra attention.

I want the reviewer to be loud about:

public exposure that was not there before
new ingress or egress paths
load balancer or API Gateway routing changes
subnet or security group edits with hidden blast radius

5. Missing operational detail

Even if the code is correct, the review is not complete if the PR forgot the operational side.

So I want AI to ask about:

rollback plan
testing evidence
migration order
monitoring and alert impact
what happens if this fails halfway through

What I Still Review Manually

AI helps me move faster, but some things should stay human-led.

My manual pass focuses on:

the actual blast radius
whether the change matches product intent
whether rollback is realistic
what stateful resources are involved
whether the diff makes sense in the larger release plan

I also sanity-check anything that looks too confident. If the model says a change is safe, that is not enough. I want the evidence behind the claim.

The best outcome is not “AI approved it.” The best outcome is “AI surfaced the right questions before I reviewed it.”

My Review Flow

The flow is pretty simple.

Collect the PR diff and Terraform plan.
Ask the AI reviewer to summarize the change in plain English.
Ask for the top risks, sorted by severity.
Ask for anything that looks inconsistent or missing.
Do my own manual pass on the flagged areas.
Approve, request changes, or split the PR if the risk is too broad.

That sounds basic, but it works because it keeps the review structured.

Here is the kind of prompt I use conceptually:

You are reviewing a production AWS change.

Focus on:
- IAM and permissions
- Terraform replacement risk
- Lambda runtime and cost impact
- networking and exposure
- missing rollback or operational detail

Return:
1. plain-English summary of the change
2. top risks ranked by severity
3. specific lines or resources that look suspicious
4. questions I should answer before merge
5. a merge recommendation: safe, caution, or block

I do not need poetry from the reviewer. I need a compact risk report I can act on.

A Good Review Finds the Questions, Not Just the Bugs

Sometimes the most useful thing the AI does is not catch a mistake outright. It points out that something deserves a question.

Examples:

Why did this Lambda need double the memory?
Why is this IAM role suddenly allowed to touch more than one environment?
Why does this Terraform change replace the resource instead of updating it in place?
Why is this public endpoint now reachable from a wider range of traffic?

That kind of review is valuable because it slows me down in the right place. It makes the risky bit visible before I merge.

Why This Complements My Other AI Workflow Posts

If I compare this to my other posts, the shape is different.

Earlier posts are about:

using AI to build faster
breaking work into specialists
keeping agents safe in production codebases

This one is about the final gate before the change goes live.

That is why I think it is a good companion piece instead of a duplicate. It belongs in the same family, but it answers a different question.

One post is about the tools that help me work. This post is about the review layer that keeps me honest.

What I Want the Reviewer To Avoid

There are a few failure modes I do not want from an AI reviewer.

Do not rubber-stamp the diff.
Do not hide uncertainty behind confident language.
Do not recommend merge without explaining why.
Do not ignore infrastructure changes just because the application code looks small.
Do not treat production Terraform like toy code.

If the model cannot explain the risk, it is not helping much.

Final Thought

AI review works best when it makes the important things easier to see, not when it replaces judgment.

That is the real pattern I keep coming back to in AWS work. I want speed, but I want it inside a review process that still respects the cost of being wrong.

So yes, I use AI to review Terraform and Lambda PRs. But the point is not to automate approval. The point is to catch the risky stuff earlier, ask better questions, and make the human review sharper.

That is the version I trust in production.

Originally published at harun.dev. The full interactive version lives there. Follow me on X or read my bio.

Building a Production RAG Pipeline on AWS Lambda and pgvector

Harun R. Rayhan — Tue, 13 Oct 2026 13:36:41 +0000

I was knee-deep in building my SaaS when users started asking questions I couldn't answer from the UI alone. My app had a solid backend, Lambda functions, Fastify, PostgreSQL on RDS. I wrote about that whole architecture in my previous post. But users wanted to search through documentation, find answers buried in help articles, and ask natural-language questions about the product. The UI just wasn't built for that.

So I started looking into RAG. Every tutorial I found showed the same setup: LangChain running locally, Pinecone for vectors, and a Python script tying it all together. Great for demos. Not great for a production SaaS already running on AWS.

Here's the thing. I didn't want to add another service to my stack. I was already paying for PostgreSQL on RDS. I was already managing Lambda functions. Adding Pinecone or Weaviate meant another bill, another SDK, another thing to monitor at 2 AM. Then I found pgvector, and everything clicked.

pgvector is a PostgreSQL extension. It adds vector storage and similarity search right inside the database I was already running. No new infrastructure. No new vendor. Just an extension. I paired it with Lambda for compute, Bedrock for the LLM, and RDS Proxy for connection pooling. The whole pipeline costs me about $54/month at 10K queries per day.

Let me walk you through every piece of how I built it.

Photo by Tara Winstead on Pexels

What Is RAG and Why Does It Work

RAG stands for Retrieval-Augmented Generation. The name sounds academic, but the idea is simple. And once I understood it, I realized it was exactly what my users were asking for without knowing the term.

You have your own data. Documents, help articles, product specs, whatever. You want an LLM to answer questions about that data. But the LLM was trained on the internet, not your private stuff. So you give it context at query time.

The flow looks like this:

User asks a question. "How do I reset my API key?"
Retrieval step. You search your own documents for the most relevant chunks. This is semantic search, not keyword matching. You're comparing the meaning of the question against the meaning of each chunk.
Generation step. You stuff those relevant chunks into a prompt and send it to the LLM. "Based on the following context, answer this question."
LLM responds with an answer grounded in your actual data.

That's it. No fine-tuning. No retraining. No waiting weeks for a model to learn your docs.

And that's exactly why RAG beats fine-tuning for most use cases. Fine-tuning is slow, expensive, and your model becomes stale the moment your data changes. With RAG, you just update the documents in your database. The next query picks up the new content automatically. Your data is always live.

The magic is in the retrieval step. If you retrieve the right chunks, the LLM gives great answers. If you retrieve garbage, the LLM confidently presents garbage. So the vector search layer matters a lot. That's where pgvector comes in. And for my SaaS, it meant I could keep everything inside the PostgreSQL instance I was already running.

Photo by cottonbro studio on Pexels

Setting Up pgvector in PostgreSQL (RDS)

When I first looked at Pinecone, I nearly added it. The docs are slick and the onboarding is smooth. Then I checked the pricing page and realized I was already paying for PostgreSQL. Why add another database when I could just enable an extension?

If you're already running PostgreSQL on RDS, adding vector search is surprisingly easy. You don't need Pinecone. You don't need Weaviate. You just enable an extension.

pgvector adds a vector column type to PostgreSQL. You store embeddings as vectors, then query them with distance operators. Cosine similarity, L2 distance, inner product. All the standard stuff.

Enable the Extension

First, make sure your RDS parameter group allows pgvector. Here's the CloudFormation snippet:

RDSParameterGroup:
  Type: AWS::RDS::DBParameterGroup
  Properties:
    Family: postgres16
    Description: "PostgreSQL 16 with pgvector"
    Parameters:
      shared_preload_libraries: "pg_stat_statements,pgvector"

Then run the SQL to create the extension:

CREATE EXTENSION IF NOT EXISTS vector;

That's it. PostgreSQL now understands vectors.

Create the Embeddings Table

Here's the schema I use. The vector(1536) column size matches OpenAI's text-embedding-3-small model. If you're using Bedrock Titan, switch to vector(1024).

CREATE TABLE hrr_document_chunks (
  id            BIGSERIAL PRIMARY KEY,
  document_id   TEXT NOT NULL,
  chunk_index   INTEGER NOT NULL,
  content       TEXT NOT NULL,
  metadata      JSONB DEFAULT '{}',
  embedding     vector(1536) NOT NULL,
  created_at    TIMESTAMPTZ DEFAULT NOW(),
  updated_at    TIMESTAMPTZ DEFAULT NOW(),

  UNIQUE (document_id, chunk_index)
);

The metadata column is a JSONB field where I store things like the source URL, document title, section heading, and any other context that helps with filtering later. You'll want this when your dataset grows and you need to scope searches to specific document types.

Create an HNSW Index

Without an index, pgvector does a sequential scan. That's fine for a few thousand rows. But once you hit tens of thousands of embeddings, you need an index.

I recommend HNSW over IVFFlat. HNSW is slightly slower to build but gives better recall and works much better for Lambda workloads. More on that in the lessons learned section.

CREATE INDEX idx_chunks_embedding_hnsw
  ON hrr_document_chunks
  USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 200);

The m and ef_construction parameters control the index quality. Higher values give better recall but slower inserts. These defaults work well for datasets up to about 500K embeddings. I landed on these numbers after a week of testing different configurations against my own dataset.

Photo by Rahul Pandit on Pexels

Generating and Storing Embeddings

Now you need to get your data into that table. This is the ingestion pipeline: take your documents, break them into chunks, generate embeddings for each chunk, and store everything in pgvector.

For my SaaS, the documents were help articles, product guides, and FAQ pages that users kept asking about. I run ingestion as a separate Lambda function. It gets triggered whenever new documents are added, either through an S3 event or a direct API call.

Chunking

Chunking strategy matters more than most people think. I split documents into chunks of about 500 tokens with a 50-token overlap between consecutive chunks. The overlap prevents information from getting lost at chunk boundaries.

function chunkDocument(text, maxTokens = 500, overlap = 50) {
  const sentences = text.split(/(?<=[.!?])\s+/);
  const chunks = [];
  let currentChunk = [];
  let currentLength = 0;

  for (const sentence of sentences) {
    const sentenceTokens = estimateTokens(sentence);

    if (currentLength + sentenceTokens > maxTokens && currentChunk.length > 0) {
      chunks.push(currentChunk.join(" "));

      // Keep last few sentences for overlap
      const overlapSentences = [];
      let overlapLength = 0;
      for (let i = currentChunk.length - 1; i >= 0; i--) {
        overlapLength += estimateTokens(currentChunk[i]);
        if (overlapLength > overlap) break;
        overlapSentences.unshift(currentChunk[i]);
      }
      currentChunk = overlapSentences;
      currentLength = overlapLength;
    }

    currentChunk.push(sentence);
    currentLength += sentenceTokens;
  }

  if (currentChunk.length > 0) {
    chunks.push(currentChunk.join(" "));
  }

  return chunks;
}

function estimateTokens(text) {
  return Math.ceil(text.split(/\s+/).length * 1.3);
}

Generating Embeddings with Bedrock Titan

Here's how to call Bedrock's Titan Embeddings model. This uses the AWS SDK v3:

import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";

const bedrockClient = new BedrockRuntimeClient({ region: "us-east-1" });

async function generateEmbedding(text) {
  const response = await bedrockClient.send(
    new InvokeModelCommand({
      modelId: "amazon.titan-embed-text-v2:0",
      contentType: "application/json",
      accept: "application/json",
      body: JSON.stringify({
        inputText: text,
        dimensions: 1024,
        normalize: true,
      }),
    })
  );

  const result = JSON.parse(new TextDecoder().decode(response.body));
  return result.embedding;
}

If you prefer OpenAI embeddings instead, the call is straightforward:

async function generateEmbeddingOpenAI(text) {
  const response = await fetch("https://api.openai.com/v1/embeddings", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.OPENAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "text-embedding-3-small",
      input: text,
    }),
  });

  const data = await response.json();
  return data.data[0].embedding;
}

Upserting Chunks

Now bring it all together. Chunk the document, embed each chunk, and upsert into pgvector:

async function ingestDocument(pool, documentId, text, metadata = {}) {
  const chunks = chunkDocument(text);

  for (let i = 0; i < chunks.length; i++) {
    const embedding = await generateEmbedding(chunks[i]);

    await pool.query(
      `INSERT INTO hrr_document_chunks (document_id, chunk_index, content, metadata, embedding)
       VALUES ($1, $2, $3, $4, $5::vector)
       ON CONFLICT (document_id, chunk_index)
       DO UPDATE SET content = $3, metadata = $4, embedding = $5::vector, updated_at = NOW()`,
      [documentId, i, chunks[i], JSON.stringify(metadata), JSON.stringify(embedding)]
    );
  }

  // Clean up old chunks if document got shorter
  await pool.query(
    `DELETE FROM hrr_document_chunks WHERE document_id = $1 AND chunk_index >= $2`,
    [documentId, chunks.length]
  );

  return chunks.length;
}

The ON CONFLICT clause handles re-ingestion cleanly. If a document gets updated, you just run ingest again and it overwrites the old chunks. The cleanup query at the end removes stale chunks if the document got shorter after an edit.

One thing to watch out for: Bedrock Titan has a rate limit. I hit this on day one when I tried to ingest my entire knowledge base at once. If you're ingesting thousands of documents, add a small delay between embedding calls or use batch processing. I throttle to about 10 requests per second and that's been reliable.

Photo by Media Dung on Pexels

Retrieval: Semantic Search on Lambda

This is the part that made it all feel real. A user sends a question, and you need to find the most relevant chunks from your database. Fast.

The first time I ran a semantic search against my own docs and got back exactly the right paragraph, I knew this was going to work. The retrieval Lambda takes a query string, embeds it using the same model you used for ingestion, then runs a cosine similarity search against pgvector. The whole thing takes about 100-200ms on a warm Lambda invocation.

async function retrieveChunks(pool, query, topK = 5) {
  // Embed the user's query with the same model used for ingestion
  const queryEmbedding = await generateEmbedding(query);

  // Cosine similarity search using pgvector's <=> operator
  const result = await pool.query(
    `SELECT
       id,
       document_id,
       content,
       metadata,
       1 - (embedding <=> $1::vector) AS similarity
     FROM hrr_document_chunks
     WHERE 1 - (embedding <=> $1::vector) > 0.3
     ORDER BY embedding <=> $1::vector
     LIMIT $2`,
    [JSON.stringify(queryEmbedding), topK]
  );

  return result.rows;
}

A few things to notice here.

The <=> operator is pgvector's cosine distance. Lower distance means higher similarity. I compute 1 - distance to get a similarity score between 0 and 1. The WHERE clause filters out anything below 0.3 similarity, which cuts out the noise. You don't want the LLM seeing irrelevant chunks.

The topK parameter controls how many chunks you return. I default to 5. Going higher gives the LLM more context but costs more tokens. Going lower risks missing relevant information. Five is a good starting point for most use cases.

Connection Pooling with RDS Proxy

If you read my previous post, you know this pattern. Lambda functions can spin up hundreds of concurrent instances. Each one opens a database connection. Without connection pooling, you'll exhaust PostgreSQL's connection limit in seconds.

RDS Proxy handles this. Your Lambda points to the proxy endpoint, and the proxy multiplexes connections to the actual database. The code is identical to a normal pg.Pool setup:

import pg from "pg";

const pool = new pg.Pool({
  host: process.env.DB_HOST,     // RDS Proxy endpoint
  port: 5432,
  database: process.env.DB_NAME,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
  max: 5,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 5000,
  ssl: { rejectUnauthorized: true },
});

Set max: 5 on the Lambda side. The proxy handles the actual pooling across all Lambda instances. Keeping the per-function pool small prevents any single Lambda from hogging connections.

Photo by Nataliya Vaitkevich on Pexels

Generation: Calling the LLM with Context

You've retrieved the relevant chunks. Now you need to turn them into an answer. This is where the LLM comes in.

I went back and forth on this part. The idea is simple: build a prompt that includes the retrieved chunks as context, along with the user's question. Then call the LLM and return whatever it generates. But getting the prompt right took more iterations than I expected.

The Prompt Template

Here's the prompt template I use. It's deliberately strict about staying grounded in the context:

function buildPrompt(question, chunks) {
  const context = chunks
    .map((chunk, i) => `[Source ${i + 1}] ${chunk.content}`)
    .join("\n\n");

  return {
    system: `You are a helpful assistant that answers questions based on the provided context.
Rules:
- Only use information from the provided context to answer.
- If the context does not contain enough information, say so honestly.
- Keep answers concise and direct.
- Reference which source(s) you used when possible.`,
    user: `Context:
${context}

Question: ${question}

Answer based only on the context above.`,
  };
}

That system prompt is important. I learned this after my first demo went sideways. Without explicit instructions to stay grounded, the LLM will happily make things up using its training data. One of my test users asked about a feature we didn't have, and the LLM invented a whole tutorial for it. The "say so honestly" part is key. I'd rather get an "I don't have enough information" response than a confident hallucination.

Calling Bedrock Claude

Here's the Bedrock call using Claude Haiku. I use Haiku instead of Sonnet for RAG because it's 10x cheaper and the quality difference is minimal when you're giving it good context:

import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";

const bedrockClient = new BedrockRuntimeClient({ region: "us-east-1" });

async function generateAnswer(question, chunks) {
  const prompt = buildPrompt(question, chunks);

  const response = await bedrockClient.send(
    new InvokeModelCommand({
      modelId: "anthropic.claude-3-haiku-20240307-v1:0",
      contentType: "application/json",
      accept: "application/json",
      body: JSON.stringify({
        anthropic_version: "bedrock-2023-05-31",
        max_tokens: 1024,
        system: prompt.system,
        messages: [{ role: "user", content: prompt.user }],
      }),
    })
  );

  const result = JSON.parse(new TextDecoder().decode(response.body));
  return result.content[0].text;
}

Handling Token Limits

Here's a practical concern. Each chunk might be 500 tokens. If you retrieve 5 chunks, that's 2,500 tokens of context. Add the system prompt, the question, and the response, and you're looking at maybe 4,000 tokens total. That fits easily within Haiku's context window.

But what if your chunks are longer, or you need more of them? I cap the total context at 3,000 tokens and truncate the lowest-similarity chunks first:

function truncateContext(chunks, maxTokens = 3000) {
  let totalTokens = 0;
  const selected = [];

  for (const chunk of chunks) {
    const tokens = estimateTokens(chunk.content);
    if (totalTokens + tokens > maxTokens) break;
    selected.push(chunk);
    totalTokens += tokens;
  }

  return selected;
}

Since the chunks are already sorted by similarity (highest first), this naturally keeps the most relevant ones and drops the least relevant. Simple and effective.

Photo by Matheus Bertelli on Pexels

Wiring It Together: The API Endpoint

Now let's connect everything. This is where my existing SaaS architecture paid off. The API endpoint receives a question, runs retrieval, generates an answer, and returns it. I use Fastify on Lambda (same setup as my previous post), and adding a new route was straightforward. The pattern works with any framework.

async function ragHandler(request, reply) {
  const { question } = request.body;

  if (!question || question.trim().length === 0) {
    return reply.status(400).send({ error: "Question is required" });
  }

  // Step 1: Retrieve relevant chunks
  const chunks = await retrieveChunks(request.server.pg, question, 5);

  if (chunks.length === 0) {
    return reply.send({
      answer: "I couldn't find any relevant information to answer your question.",
      sources: [],
    });
  }

  // Step 2: Truncate context to fit token limits
  const trimmedChunks = truncateContext(chunks);

  // Step 3: Generate answer
  const answer = await generateAnswer(question, trimmedChunks);

  // Step 4: Return answer with source metadata
  return reply.send({
    answer,
    sources: trimmedChunks.map((c) => ({
      documentId: c.document_id,
      similarity: parseFloat(c.similarity.toFixed(3)),
      preview: c.content.substring(0, 150) + "...",
    })),
  });
}

// Register the route
app.post("/rag/ask", {
  schema: {
    body: {
      type: "object",
      required: ["question"],
      properties: {
        question: { type: "string", minLength: 1, maxLength: 1000 },
      },
    },
  },
  preHandler: [app.authenticate],
  handler: ragHandler,
});

The handler is intentionally thin. Retrieval and generation are separate modules. This makes testing easy. You can unit test the chunking, the retrieval query, and the prompt template independently. The handler just orchestrates the flow.

I also return the source metadata alongside the answer. This is useful for two reasons. First, your frontend can show users where the answer came from, which builds trust. Second, it helps you debug retrieval quality. If the answer is wrong, check the sources. Nine times out of ten, the problem is that retrieval returned the wrong chunks, not that the LLM misunderstood them.

The preHandler uses the same authentication middleware from my existing API. This was one of the perks of building RAG into my existing stack instead of bolting on a separate service. Same auth, same rate limiting, same logging. RAG endpoints should always be authenticated. You don't want someone querying your private documents without authorization.

Photo by Brett Sayles on Pexels

Cost Breakdown

Let's talk real numbers. This was a big deal for me because I didn't want RAG to double my AWS bill. I'm going to break down costs for a small AI SaaS handling about 10,000 RAG queries per day.

The Infrastructure

Component	Monthly Cost	Notes
RDS db.t3.micro (PostgreSQL + pgvector)	~$15	2 vCPU, 1GB RAM, 20GB storage
RDS Proxy	~$20	Connection pooling for Lambda
Lambda (retrieval + generation)	~$5	512MB, avg 300ms per invocation
API Gateway (HTTP API)	~$3	$1 per million requests
Bedrock Titan Embeddings	~$3	~$0.10 per 1M tokens, ~30M tokens/month
Bedrock Claude Haiku (generation)	~$8	~$0.25/1M input, ~$1.25/1M output tokens
Total	~$54/month

That's $54/month for a production RAG pipeline handling 10K queries per day. Not bad.

Compared to Hosted Vector Databases

Now let's compare to what you'd pay with a dedicated vector database:

Pinecone Starter: ~$70/month for 1M vectors. And you still need the LLM costs on top of that.
Weaviate Cloud: ~$25/month for their starter tier. Plus compute for the actual search.
Qdrant Cloud: ~$30/month for a small instance.

With pgvector, the vector storage cost is zero because it's part of your existing PostgreSQL instance. You're already paying for RDS. The extension is free. This is exactly why I went this route. I just added vector search to a database I was already running and paying for.

The math is clear: if you already have PostgreSQL on RDS, pgvector is the cheapest vector database alternative out there. You avoid a whole separate service, a separate bill, and separate operational overhead.

Where the Money Actually Goes

The biggest cost in a RAG pipeline is not storage or compute. It's the LLM generation calls. At 10K queries per day with Claude Haiku, you're spending about $8/month on generation alone. Switch to Claude Sonnet and that jumps to ~$80/month. Switch to GPT-4o and you're looking at ~$150/month.

So choose your generation model carefully. For most RAG use cases, Haiku or GPT-4o-mini gives you 90% of the quality at 10% of the cost. Save the big models for cases where answer quality justifies the spend.

Photo by Leeloo The First on Pexels

Lessons Learned

After running this RAG pipeline in production for a few months, here are the things I learned the hard way. Every one of these cost me hours of debugging or unnecessary spending.

1. Chunking Strategy Matters More Than the Embedding Model

I wasted about two weeks on this one. I was obsessing over Titan vs. OpenAI embeddings, running benchmarks, comparing similarity scores. The difference in retrieval quality between them was maybe 5%. Then I changed my chunking from fixed 1000-token chunks to 500-token chunks with overlap, and retrieval quality jumped by 30%. Two weeks of model comparison, undone by a single config change.

The reason is obvious in hindsight. Smaller chunks are more precise. When a user asks a specific question, a 500-token chunk about that exact topic scores much higher than a 1000-token chunk where the relevant part is buried in the middle. The overlap makes sure you don't lose context at chunk boundaries.

If your RAG answers feel "off," look at your chunks before you look at your embedding model. I wish someone had told me that from day one.

2. HNSW Index vs. IVFFlat for Lambda

This one bit me in production. pgvector supports two index types: IVFFlat and HNSW. I started with IVFFlat because the internet says it's faster to build. That was a mistake I didn't catch until users complained about slow first responses.

IVFFlat requires a "warm-up" query to load the index into memory. On Lambda, that warm-up happens on every cold start. It added about 400ms to the first query after a cold start. My users noticed. I got support tickets about it.

HNSW doesn't need warm-up. The first query is about the same speed as subsequent ones. The moment I switched, the cold start complaints stopped. For Lambda workloads where cold starts are a reality, HNSW is the better choice. The trade-off is that HNSW uses more memory and takes longer to build, but you're building the index once and querying it thousands of times. The build cost is irrelevant.

3. Cache Embeddings for Repeated Queries

I noticed this pattern about a month in. Users ask the same questions over and over. "How do I reset my password?" "What are the pricing tiers?" "How do I cancel?" I was paying Bedrock to embed identical strings dozens of times a day.

Embedding the same question every time wastes money and adds latency. I added a simple cache using a hash of the query text:

const embeddingCache = new Map();

async function getEmbedding(text) {
  const cacheKey = createHash("sha256").update(text.toLowerCase().trim()).digest("hex");

  if (embeddingCache.has(cacheKey)) {
    return embeddingCache.get(cacheKey);
  }

  const embedding = await generateEmbedding(text);
  embeddingCache.set(cacheKey, embedding);

  // Keep cache from growing forever
  if (embeddingCache.size > 1000) {
    const firstKey = embeddingCache.keys().next().value;
    embeddingCache.delete(firstKey);
  }

  return embedding;
}

This is an in-memory cache, so it lives as long as the Lambda container. For a warmer setup, you could use DynamoDB or ElastiCache, but the in-memory approach works surprisingly well. In my case, about 40% of queries are cache hits within the same Lambda container.

4. Monitor Retrieval Quality Separately from Generation Quality

I spent an entire weekend debugging a "wrong answer" that a user reported. I was blaming the LLM, tweaking the prompt, trying different temperature settings. Then I finally looked at the retrieved chunks and realized the retrieval step had pulled in completely irrelevant paragraphs. The LLM was doing its best with garbage context. That weekend taught me to always check retrieval first.

Now I log both steps separately:

Retrieval quality: log the query, the returned chunks, and their similarity scores. Review the lowest-scoring returned chunks regularly. If they're irrelevant, your similarity threshold is too low.
Generation quality: log the prompt (with context) and the LLM's response. When users report bad answers, check whether the context contained the right information.

Most of the time, bad answers come from bad retrieval. The LLM is usually fine if you give it the right context. So investing in better chunking, better metadata filtering, and tuning your similarity threshold pays off way more than switching to a fancier LLM.

Photo by Mikhail Nilov on Pexels

Conclusion

My SaaS now has a /rag/ask endpoint that my users actually love. They type a question, get an answer grounded in our own documentation, and see exactly which sources the answer came from. The whole thing runs on the same Lambda and PostgreSQL stack I was already using. No new services. No new vendors. Just pgvector and a few hundred lines of code.

Looking back, the decision to keep everything inside my existing stack was the right call. I avoided a separate vector database bill, a separate SDK to maintain, and a separate service to monitor. pgvector gave me semantic search, HNSW indexing, and cosine similarity queries inside the database I already knew how to operate. The total cost sits around $54/month at 10K queries per day.

If I were starting over, I'd change one thing. I would spend more time on chunking from the very beginning instead of comparing embedding models. Good chunks make everything downstream better. Bad chunks make everything worse, no matter how fancy your LLM is. I learned that lesson the expensive way.

For anyone building an AI SaaS on AWS, this setup hits a sweet spot. Low cost, minimal operational overhead, and your data stays in PostgreSQL where you can query it with regular SQL alongside your vector searches. If you already have the Lambda and RDS stack running, you're closer to production RAG than you think.

Now go build something cool with it.

Hope you enjoyed this walkthrough. If you have questions about any part of the pipeline, or if you've built something similar and want to compare notes, reach out.

Follow me on Twitter/X for more posts about AWS, serverless, and building AI products. I share what actually works in production, not just what looks good in tutorials.

Photo by Nicolas Foster on Pexels

Originally published at harun.dev.

First 3 things I always do after creating a new AWS account

Harun R. Rayhan — Wed, 06 Jul 2022 14:24:50 +0000

As part of my job, I create AWS accounts frequently. I do lots of things depending on account needs before starting to use it. I found 3 common things I do with all of the new AWS accounts. I made a checklist. Nowadays I do these 3 things first and then everything else.

Let's make your account more secure and more protected than ever. 🔐

1. Budget with Alerts

One of the common issues with AWS is bills. People gets surprising bills because of misconfiguration or keep expensive services running unknowingly. Although, budget and alerts can NOT fix these issues but help keep bills tidy. And gives alerts if crosses the budget almost instantly.

Let's set a budget and a couple of alerts.

Go to Budget Console
Click on the "Create Budget" button.
Choose the first option, "Cost budget - Recommended" from the next page.
From the next page:
- A. Give it a name.
- B. Set the Period as Month for this demo, but you can choose any other period.
- C. Set Budget renewal type as Recurring budget
- D. Choose Fixed from the Budgeting method.
- E. Give a real budget amount to your need, mine is a demo account so $10 is fine for me. Let me iterate again, choose a real amount.
- F. From the bottom-right of the page click "Next"
From the next page, click on the "Add an alert threshold. Going to set a couple of alerts.
Let's set an actual budget alert:
- A. Choose 80 from the threshold.
- B. Select Actual from the trigger
- C. Put as many real and active emails as necessary. (⚠️ please add active email to save yourself from disaster).
- D. Let's add another one. Actual budget alert only sends notifications when you spent threshold amount. For example, it will send an email to me after spending $8 (80% of 10$). So, we need something else together with it.
Now set a forecasted budget alert:

Everything is the same as the first alert, only change Trigger to Forecasted. So, it will send you an alert if AWS forecast that you will spend more than your threshold. After configuring it, click the Next button from the bottom of the page.
Click the Next again from the next page. And then click the Create Budget button from the final page.

And you are done with, you probably going to be saved from a disaster by email notifications.

...

2. Add Multi-Factor Authentication (MFA)

You should use MFA for all IAM accounts including the root account. If not possible, at least apply it to root and privileged accounts. Login to the account (root or IAM) and let's protect it.

Go to Security Credentials of IAM Console
Click on the "Assign MFA device"
Choose Virtual MFA device from the popup
You can choose either of these apps: Authy, Duo Mobile, LastPass Authenticator, Microsoft Authenticator, Google Authenticator, Symantec VIP. My personal choice is Authy.
Click on the "Show QR code" from the next popup and scan it from your MFA app. Then enter two codes (one after another). And then click the "Assign MFA" button.

...

(bonus) Set Alias for the Account

Account ID or Alias is required to log in IAM account. Account ID is a long number. Humans are bad with a long number. A meaningful alias can save you from the trouble of remembering the account ID. Let's set an alias for your account:

Go to IAM Console after login into the root account.
You will see a Create button in the right sidebar under the account alias.
Choose a unique and memorable alias for your account and click "Save changes".

Use the alias instead of account ID next time you are logging into an IAM account.

...

3. Create IAM account(s) for Day-to-Day tasks

You need the root account for a few tasks. Other than that you should not use the Root account. Perhaps, not for day-to-day tasks, lunching, and provisioning services.

I always create an IAM account with admin permission. I do all of the tasks from this account except those tasks that require the root account. So, let's create an admin IAM account.

Go to Users page of IAM Console.
Click on the Add users button from top-right corner.
You will be in this page:
- A. Choose an username
- B. Chosse access type. I need both CLI and AWS Console access. (UPDATE: Don’t choose CLI, if not necessary. You can create different IAM users for CLI actions.)
- C. You can enter your custom password or choose auto generated.
- D. It's better to require reset password to choose it's own.
- E. Click on the Next: Permission button from the bottom-right
You are in the permission page:
- A. Choose Attach existing policies directly from the top options
- B. Search for the AdministratorAccess role
- C. Choose the AdministratorAccess role
- D. Click on the Next: Tags button
You are in the tags page. You can add you tag(s) if you want but it's optional. Click in the Next: Review button.
Take a look on everything and check if it's matches to mine. And then click on the Create user button.
Now we are in the final page:
- A. The link to login to the console
- B & C. Username & Password to login to the console
- D & E. Access key & secret to login via CLI or App
- F. You can download the .csv file with credentails. You can save it in safer place for later use.

...

Conclusion

There are more things to do to save bills and tighten security of your account. But these are the first 3 things I do and doesn't require much technical knowledge.

You can subscribe to my newsletter in my blog. Follow me on Twitter, Dev.to, and Hashnode for regular updates. Don't hesitate to show your love by clicking Like 👍 button.

Install Cloud Watch Agent in Amazon Lightsail instance for Monitoring, Logging & Debugging

Harun R. Rayhan — Sat, 04 Jun 2022 04:51:10 +0000

Use the Cloud Watch agent to send logging and debugging to Cloud Watch. You can check logs without logging into instances. It's accessible even after the instance is terminated.

Life is far from perfect, and so does Amazon Lightsail instances. It can be frustrating if there is an error and you don't know how to debug it. The server itself and applications installed in it produces log files. And server stats can be helpful in monitoring, logging, and debugging.

Prerequisites

It would help if you had an existing Amazon Lightsail instance. You can create a new instance by following "Deploy WordPress App to Amazon Lightsail" or "Deploy Laravel Application to Amazon Lightsail".

Let's go 🏃‍♂️

Create an IAM User

Go to IAM Console.
Click on the Users menu from the navigation panel and then click on the Add Users button.
You will be in the Add user screen:
- A. Give it a name. For me, it's LightsailCloudWatchAgent.
- B. Check the Access key - Programmatic access option
- C. Click on the Next: Permissions button.
You will be in the Set permissions page:
- A. Select the Attach existing policies directly option.
- B. Search for CloudWatchAgentServerPolicy.
- C. Select CloudWatchAgentServerPolicy, and
- D. Click the Next: Tags button.
Tags are optional. You can choose your tags. Click on the Next: Review after you are done.
From review page you can check if everything alright and then click the Create user button.
Your account is created and you will be in this screen:
- A. & B. Copy Access Key ID and Secret access key in somewhere safe, or,
- C. Download the csv file. We will need this access later.

Configure AWS Access

Check if AWS CLI is installed by running aws --version. You should see the current version. If you get an error, CLI probably is not installed.
Ignore this if AWS CLI is already installed. Run this on ubuntu:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
sudo ./aws/install --bin-dir /usr/bin --install-dir /usr/local/aws-cli --update

Follow this instruction to install on other servers.

Now, we are going to configure the AWS access generated in the last step. Run this

sudo aws configure --profile AmazonCloudWatchAgent

Enter Access Key and Secret Like this:

Install the Cloudwatch Agent

You need to SSH into your instance. Here is how you can SSH into your Amazon Lightsail Instance.
Now, let's install the agent. I'm using ubuntu, you can just run this command

# Download the agent
wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
# Install the agent
sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

Now run this command configure the agent.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Input your choices as I did. Check the arrow mark. You can choose your own or follow mine.

Give AWS Access to the Agent. Run this:

sudo vim /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

Paste this in the bottom of the file:

[credentials]
shared_credential_profile = "AmazonCloudWatchAgent"

Now hit ESC and then :wq!.

Start the agent:

sudo amazon-cloudwatch-agent-ctl -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -a fetch-config -s

Check the status of the agent by using:

sudo amazon-cloudwatch-agent-ctl -a status

You should see something like this:

Verify metrics on CloudWatch

Go to the Cloudwatch Console
On the left navigation panel, choose Metrics.
Under “Custom Namespaces”, You should see a link for “CWAgent”.
Choose CWAgent.
Choose any ImageId, InstanceId, InstanceType.
Select the checkbox to display metrics on the graph. Here is mine:

Conclusion

Hope you enjoyed this article. This is the last article of the Amazon Lightsail series. You can ask me any Lightsail-related question in the comments or on social media.

You can subscribe to my newsletter to get blog updates every week. Follow me on Twitter, Dev.to, and Hashnode.

Connect Redis/Elasticache to application in Amazon Lightsail instance

Harun R. Rayhan — Sat, 28 May 2022 14:32:18 +0000

You can create Redis/Elasticache even though Amazon Lightsail doesn't offer it. You can create an Elasticache for Redis cluster in AWS and connect using VPC Peering.

Prerequisites

An Amazon Lightsail instance. To create a new instance follow Deploy Laravel Application to Amazon Lightsail or Deploy WordPress App to Amazon Lightsail
Be a little familiar with AWS Console.

Create Elasticache for Redis

Go to Elasticache for Redis console. Change the region that you want to launch it. I selected us-east-1 as my Lightsail Instance(s) are there.
Click the Create Redis cluster button from the center of the page (or wherever you see it)
You will land in this screen 👇
- A. Choose Configure and create a new cluster
- B. Choose cluster mode Disabled
- C. Give it a name
- D. Uncheck Multi-AZ Enable
- E. Uncheck Auto-failover Enable
- F. Select a free-tier node type. Currently it's "cache.t3.micro" and "cache.t2.micro".
- G. Do not need any replica for this demo. Make it "0".
- H. Give subnet groups a name
- I. Select your VPC
- J. And then clieck the Next button from the bottom.
You will be on the second page. Just uncheck Enable automatic backup and click the Next button from the bottom of the page.
Review everything on the 3rd and final page. And then click the Create button from the bottom of the page.
It will take a few minutes to get created. Wait for status changes to Available.
now, click on the Cluster name. And you will be on the details page.

Copy the Primary endpoint in your clipboard or somewhere safe.

Configure Security Group for Redis cluster

Go to the Security groups of VPS Console. Make sure you are in the same region as the Redis cluster. And then click the Create security group from the top-right corner.
You are in create Security Group page
- A. Give it a name
- B. Click on the Add rule button of the Inbound rules section.
- C. Select 6379 as Custom TCP Port
- D. Add your Amazon Lightsail VPC IP range. To generate it, check your instance's Private IP address. Mine is 172.26.3.157. Replace the last two sections with 0 and add /16. So, mine became 172.26.0.0/16.
- E. Optionally, you can add a description
- F. Then hit the Create security group button from the bottom of the page.
Now go to the Network and security tab of your Redis cluster and click the Modify button from the Security groups section.
Click on the Manage button from the next screen
Select the correct security group from the pop and then click on the Choose button.
Click Modify button from the bottom of the page.

It will take a few minutes to complete the modification. Wait until the status changes to Available.

VPC Peering

The important part of this tutorial. We need to peer VPC between Lightsail and Redis cluster's VPC to connect it.

Go to the homepage of the Amazon Lightsail console. Click the Account from the Account menu dropdown.
Click on the Advanced tab from the Account settings page. You will see a list of your VPCs. Connect the one that resides in your Redis cluster. For me, it's *Virginia (us-east-1)
*.

Connect to the Redis cluster from the Lightsail instance

Using CLI

Connect to the instance using SSH. Follow this to connect to SSH.
Install Redis CLI
- A. On Amazon Linux 2

  sudo amazon-linux-extras install epel -y
  sudo yum install gcc jemalloc-devel openssl-devel tcl tcl-devel -y
  sudo wget http://download.redis.io/redis-stable.tar.gz
  sudo tar xvzf redis-stable.tar.gz
  cd redis-stable
  sudo make BUILD_TLS=yes

B. on Ubuntu

  sudo apt install redis-server

Now connect to the Redis cluster. The moment of truth.

redis-cli -h <redis_endpoint> -c -p 6379

Replace <redis_endpoint> with your Primary endpoint of Redis you copied in the clipboard. Remove port (":6379") from the endpoint URL. You will see this screen if successful 👇

You can use the ping, set, and get commands to test it out.

Connect to Application

You will get packages for most of the Applications, languages, and Frameworks. Check docs. For Laravel, you just have to configure it in the .env file.

Conclusion

Hope you enjoyed this short article. You can subscribe to my newsletter, and follow me on Twitter, Dev.to, and Hashnode.

Add Lightsail Load Balancer to Application hosted in Amazon Lightsail instance(s)

Harun R. Rayhan — Sat, 28 May 2022 11:12:16 +0000

Lightsail offers Load Balancer. It's very easy to provision one and attach one or multiple instances to it. Let's create one and attach instances to it.

Prerequisites

You need one or more Amazon Lightsail instances to attach to the load balancer in the same region. You can create it by following Deploy Laravel Application to Amazon Lightsail or Deploy WordPress App to Amazon Lightsail

⚠️ Warning: Amazon Light Load Balancer going to cost you USD$18/month. It has no pre-tier available. Be sure before going to provision one. You can check the whole article before creating.

Provision a Load Balanced

Go to the homepage of the Amazon Lightsail console. Click on the Networking tab and click on the Create load balancer button.
You are in the Create a Load balancer screen like this 👇
- A. Change your region to where your instances are. My instances are in Virginia.
- B. Give it a name, and
- C. Hit the Create load balancer button from the bottom.
Load balancer will be created in a moment and you should land on the Load balancer page.

Attach instances to the Load balance

Go to the homepage again and then click the Networking tab. You should see Loadbalancer(s) including the one just created. Click on the name of the Load balancer.
Our load balancer doesn't have any attached instance(s) yet, we are going to attach one or more. Select your instance in the Target instances section.

I just have one in this region. You should see all of the instances you have in the region where the load balancer is provisioned.
Confirm the attachment of your instance
Do the same process until you attached all of your target instances.
Go to Load balancer URL (DNS name) and you should see your application.

ℹ️ If you don't see the desired page, check your application or other tools (Apache, Nginx, etc.) setting.

Add Domain or Sub-Domain

You can create DNS Zone in Lightsail for your domain and configure it. Follow this to add your domain/sub-domain.
You can add the Load balancer URL (DNS name) as A record in your domain's DNS control panel.

Enable HTTPS

Go to the Inbound traffic tab of your Load balancer setting.
- A. Create an SSL certificate for your domain by clicking the Create certificate + button.
- B. You should see all of the created certificates in the SSL/TLS certificate list and select the one you just created.
- C. Optionally, you can Redirect from HTTP to HTTPS. (After enabling the HTTPS).

Conclusion

Hope you enjoyed this short article. You can subscribe to my newsletter, and follow me on Twitter, Dev.to, and Hashnode.

Add Database to Application hosted in Amazon Lightsail

Harun R. Rayhan — Mon, 23 May 2022 14:04:33 +0000

Amazon Lightsail offers dedicated Databases. You can make use of it instead of an on-instance database. It's easy, managed, secure, and most importantly it's not tied to the instance. So, your database will be safe even if instance crashes.

In "Deploy WordPress App to Amazon Lightsail, the database is installed in the instance. In this article, we are to cover the dedicated databases.

⚠️ Warning: Dedicated Lightsail Databases are going to cost $$$. It has a 3-month free tier at this time of writing. Check your plan before creating a database.

So, what are you waiting for? Let's dive right in 🏃‍♂️

Create a Database Instance

Go to your Amazon Lightsail home page. And then click on the Databases tab.
Now click on the Create database button
Now you are on the database creation page:
- (A) Select AWS Region and Availablity Zone (AZ). for me Virginia and Zone A are perfect.
- (B) Choose the Database engine and version. I selected MySQL 8.
- (C) Click on Specify login credentials, and
- (D) Click on Specify the master database name.
Scroll down a little and configure this:
- (A) Choose Database username
- (B) Password. I'm okay with the generated password, but you can choose yours.
- (C) The first/master database. You can create more later.
- (D & E) Choose your plan type and the actual plan. You can choose the free-tier option like me.
- (F) Give it a name, and
- (G) Hit the Create database button Your database should be created within a few minutes. Please wait for it until it's Running.

Database Credentials

You already know your database username and master database. For everything else, go to your database and click on the Connect tab. You should see your credentials like this:

(A) DB Username
(B) DB Password
(C) DB Endpoint
(D) DB Port

Connect to the database

Instance CLI

You can SSH into your instance.

Install MySQL using the following command:

sudo apt install mysql-server -y
sudo systemctl start mysql.service

And then connect to MySQL:

mysql --host <db_endpoint> --user <db_username> --password

You will be asked for your DB password. If successful, you will see this screen:

SQL Client App over SSH (SSH Tunneling)

Most modern SQL Client applications (ex TablePlus, MySQL Workbench) have the option to connect Database via SSH. You can connect using that option.

Add Database to your application

Laravel and NodeJS

You can update your application's .env file with Database credentials.

WordPress

Update your wp-config.php file with the database credentials.

Any other app

Update your database credential in your application's configuration and/or environment file. Check your application's documentation for that. Or it's custom, you know where to put it.

Conclusion

Thanks for reading the article. Adding dedicated databases to Amazon Lightsail instance is not that hard. You can follow me to get instant updates on my posts. You can also subscribe to my newsletter.