DEV Community: Akarshan Gandotra

[Boost]

Akarshan Gandotra — Fri, 30 Oct 2026 06:32:07 +0000

Akarshan Gandotra

Oct 29

Zero Standing Access for AI Agents: Let Users Lend Access, One Session at a Time

#ai #security #architecture #identity

13 min read

Zero Standing Access for AI Agents: Let Users Lend Access, One Session at a Time

Akarshan Gandotra — Thu, 29 Oct 2026 07:07:11 +0000

Agents are doing real work in our products now. They read records, draft emails, file tickets and reconcile invoices. To do any of that they need access, and when it's time to give it to them, most teams reach for one of two tools. Both are wrong.

Option one: a service account. The agent gets its own credentials with a fixed set of permissions. It works, but it's standing access that no user ever agreed to, it isn't bound to any user, and every action shows up in the audit log as "the robot did it".

Option two: let the agent act as the user. Pass the user's token through, or mint one that impersonates them. Now the agent can do everything the user can do. A task that needed one key gets the whole keyring.

I think agents deserve to be a first-class principal in IAM, next to users and service accounts, built on one rule:

An agent owns no access. It borrows access from the person it works for, one request at a time, with that person's consent, for the current session only, and never more than that person has.

The rest of this post is what that rule looks like in practice: the entities IAM needs, agent templates, a permission catalog that IAM hosts as a tool, three user journeys with wireframes, and the parts I'm still arguing with myself about.

TL;DR

Every agent session starts with zero access.
The agent asks IAM for what it needs when it needs it, through tools, and gives a reason.
The user approves in the chat, the way AI coding tools ask before running a command: Allow once, Always allow, Deny.
Grants last for the session only. "Always allow" saves a rule that skips the prompt next time. It doesn't preload access.
An agent can never hold more than its user has right now, and some permissions are never available to agents at all.
Agent templates define what a kind of agent may ever ask for, plus the context it needs to ask well.
IAM hosts a permission catalog tool, so builders and agents can find out what exists, what's risky and what's off-limits.

Where the industry is today

As of October 2026 there's no finished standard for agent access. There is a lot of convergence, though.

Standards and guidance

The Model Context Protocol authorization spec builds agent tool access on OAuth 2.1. Each MCP server publishes which authorization server it trusts, and each token is bound to one server.
The IETF OAuth working group has a draft for Transaction Tokens: short-lived signed tokens that carry the user and the context along a chain of service calls. Several individual drafts propose agent-specific profiles (agent grants, agent instances, identity assertion grants), and none has been adopted by a working group yet.
The OpenID Foundation whitepaper Identity Management for Agentic AI says existing standards cover simple agents inside one organization, and recommends delegation through OAuth Token Exchange (RFC 8693).
NIST's NCCoE started a project on software and AI agent identity. Most of its 600+ public comments asked to build on existing standards, and agents that start other agents are flagged as an open problem.
The OWASP Top 10 for Agentic Applications lists identity and privilege abuse as a top risk. It recommends short-lived, narrowly scoped credentials per task and per agent, a fresh check before each privileged step, and human approval before an escalation.

What vendors ship

Microsoft Entra Agent ID gives each agent its own identity, created from a reusable blueprint and owned by a named human sponsor. High-risk permissions are blocked for agents even when someone consents.
Okta Cross App Access lets the company's identity provider decide which agent can reach which app. Its tokens carry the user as sub and the agent as act.
Auth0 for AI Agents adds a vault for third-party tokens, approval pushed to the user's phone (CIBA) for actions that need consent, and fine-grained checks on every action.
AWS Bedrock AgentCore Identity swaps the user's token for a narrower one per downstream service, carrying both identities.

The convergence is real: agents get their own identity, delegation goes through token exchange with the agent recorded in act, credentials are short-lived, and some permissions are blocked outright. What's still mostly missing is the part in the middle. What happens between "the agent asked" and "the agent got it"? Who consents, to what, for how long, and what do subagents inherit? That's the part this post is about.

Seven principles

Zero standing access. An agent, and every new session of it, starts with nothing.
Never more than the user. Every grant is checked against the user's current access, both when it's granted and at every token refresh.
Only the user can consent. Approval needs the user's own credential. The agent can ask, but it can't approve.
Session-scoped. Grants end when the session ends or is revoked.
Inherit down, never up. A subagent gets at most its parent's access, and what a subagent is granted stays with it.
Remember the consent, not the access. Saved rules let a user skip the prompt next time. Each session still starts at zero.
Everything is traceable. Every request, decision, grant and call records the agent, the user and the session.

A new entity in IAM: the agent

Here's how an agent compares with the two options we have today.

	Service account	Acting as the user	Agent
Acts for	Itself	A user	A user
Access when it starts	Whatever an admin granted	All of the user's access	None
Who decides its access	An admin, up front	Nobody	The user, per request
Bounded by the user	No	Equal to the user	Always at or below the user
How long access lasts	Until someone changes it	The token's lifetime	The session
Audit shows	The service account	The user	The agent, the user and the session
Stop one run	No	No	Yes

To support this, IAM and user management need a few new things: an agent template (what a kind of agent may ever ask for), an agent (a deployed agent with its own credential), an agent session (one run for one user), grants, access requests, consent rules and an activity log.

One design choice matters more than it looks: none of the existing grant tables (role assignments, group memberships, user permissions) ever point at an agent. Agent grants live in their own table and die with their session, so standing access for an agent can't exist even by accident.

What an agent can borrow

Most systems have more than one kind of permission, and an agent should be able to borrow each of them, with the same rules applied to all.

Kind	Controls	Example
API permissions	Which endpoints and actions it can call	Accounts API · Read
Data permissions	Which records and fields it can see	Accounts in my region
Asset permissions	Which objects it can open	Q3 pipeline dashboard · View
Tool and outbound permissions	Which tools it can run and where data can go	Send email to customers
Never delegable	Nothing an agent can ever get	IAM admin rights, emergency access, managing agents or consent

Your system may have more kinds, such as feature flags, billing actions or workflow approvals. The rules don't change. Each kind is a type of grant with its own enforcement point.

Agent templates: permissions plus context

An agent template is a reusable definition of one kind of job. It holds:

Purpose: what the agent is for.
Ceiling: the most it may ever ask for, by kind of permission. A request outside it is refused, whatever the user has.
Context: how the agent should reason about access.
Modes allowed, limits (session lifetime, subagent depth, request rate) and child templates it may start as subagents.
Sensitive items, which always prompt.
A sponsor, the human accountable for it, and a version.

The ceiling stops an agent from overreaching. It doesn't make the agent ask well. Left alone, an agent will often request its whole ceiling up front, because that's the easiest way to never be blocked. The context is there so the agent does the analysis every time: what's the smallest set of access this task needs, when is sensitive data actually justified, and when should it give access back.

Ask only for accounts in the renewal window, never the whole book. Prefer read access. Ask for contract pricing only when the user asks about price. Release access as soon as a brief is done.

Organizations can turn a template on and narrow it, but never widen it. A new version that widens the ceiling asks users for consent again. A version that only narrows it doesn't.

The permission catalog: an IAM tool for builders and agents

Here's a problem I didn't expect. People building agents don't know the permission model, and agents guess. IAM is the only system that knows what permissions exist, what each one unlocks, how risky it is, which data is sensitive and what's never delegable.

So IAM should publish that knowledge as a tool. iam.describe_permissions(task) searches the catalog for a task. iam.explain(permission) says what a permission unlocks, how risky it is, where it's enforced and whether an agent can have it. Builders use it to write ceilings. Agents use it at run time to plan the smallest request. Security reviewers use it to see what a new template version adds.

The same IAM tools give the agent its access during a session:

Tool	What it does
`iam.describe_permissions(task)`	Finds the permissions that fit a task, with risk and sensitivity
`iam.request_access(items, reason)`	Asks for access for this session. Each item is granted, refused or sent to the user
`iam.my_access()`	Lists what this session holds right now
`iam.start_subagent(template, read_only)`	Starts a subagent from an allowed child template
`iam.release_access(items)`	Gives access back early
`iam.end_session()`	Ends the session and every subagent, and revokes all grants

The app makes these calls for the agent and holds the tokens. The model sees each result, never a token.

Journey 1: building and publishing a template

A builder describes the job ("prepare renewal briefs for an account manager") and asks the permission catalog what fits.
They draft the template: the ceiling, the context and the limits. The catalog flags contract pricing as sensitive, outbound email as high risk, and billing admin as never available to agents.
Security reviews it. Because every item comes from the catalog, the review is a diff: what does this version add, and how risky is it?
The template is published as v3.
An organization admin turns it on and may narrow it, for example by removing contract pricing or turning off bypass mode for their users.

Journey 2: a user's session, end to end

The user opens a three-pane app: sessions on the left, the chat in the middle, and the session's access on the right.

Step 1: the session starts with nothing. The user types the task and picks a mode from the message box. The app exchanges the user's token, its own credential and the template for a session token that carries no permissions at all.

Step 2: the agent asks. To find this quarter's renewals, the agent calls iam.request_access for three items: an API permission, a data permission and an asset permission, with a reason. IAM checks each item: it's inside the template's ceiling, the user has it right now, and it isn't on the never-delegable list. No saved rule covers these, and the mode is Ask, so the call comes back as a prompt in the chat.

Step 3: the user approves. They choose Always allow. IAM records three grants on the session and three rules, then issues a fresh token with exactly those permissions. The decision only counts because it arrives with the user's own token. An agent can't approve its own request.

Step 4: the agent works, and starts a subagent. It calls the accounts API with its own token. Then it starts an email drafter subagent, read-only. The drafter inherits the three grants as read-only, asks for its own "Email templates · Write", and gets it once. That grant stays with the drafter and doesn't flow up to the parent.

Step 5: a sensitive request, and a change of access. The drafter asks for contract pricing. Sensitive items prompt in every mode and can never be saved as Always allow, and the user denies it. That afternoon the user loses access to the pipeline dashboard. At the next token refresh, 15 minutes at most, the agent loses it too. Nobody had to touch the agent.

Step 6: the session ends. The agent calls iam.end_session, or the user clicks End session, or it expires. The session and its subagents are revoked. The saved rules stay.

Journey 3: the next day, and the audit

The user starts a new session, and it holds nothing. The agent asks for the same three items. Two match saved rules and are granted with no prompt. The dashboard has a saved rule too, but the user no longer has access to it, so it's refused. A rule never grants more than the user has.

Later, someone in compliance asks what the agent accessed for this user last week. The activity log answers per session: what was asked, who decided and how (prompt or rule), what was granted and what was called. When the template ships a version that widens its ceiling, users are asked again once.

Modes and rules, like your AI coding tool

If you've used an AI coding tool, you already know the shape of this.

Session modes, picked when the session starts. Ask prompts for anything no rule covers. Auto also approves low-risk items, such as read-only API access, without a prompt. Bypass approves anything the user has except sensitive items, and an organization can turn it off.
Consent rules, saved from a prompt: allow, deny or ask, per template and per permission.

The order matters. The template's ceiling, the user's current access and the never-delegable list always win. Then deny, ask and allow rules apply, and only then the mode. The mode is stored on the server's session record. If the agent could send it with the request, a prompt-injected agent would simply claim bypass.

Under the hood

The mechanics are mostly standard. The app does an OAuth 2.0 Token Exchange (RFC 8693) when the session starts, and the agent's identity rides in the act claim, with one more nested act for each subagent level. Tokens live 15 minutes at most and are re-issued after every grant. The user's current access is re-checked at every issue, so losing a role removes it from the agent within one refresh. Revoking a session bumps a version number that every token carries.

Each kind of permission is enforced where it already lives:

Two lessons that weren't obvious until I worked through them:

Data restrictions come from the user, always. If an agent's data access were just the data permissions it asked for, it would skip the user's restrictions, such as masked fields or a block on certain sensitive records. The agent would see things the user can't. Grants come from the session, restrictions come from the user, and restrictions can't be requested at all.
Object access needs a live dual check. API permissions get recomputed every time a token is issued. Object permissions are often stored as relationships written once, at grant time. The only thing that notices the user lost access to a dashboard is a check that the session has it and the user still has it.

What can go wrong

Risk	Guardrail
Prompt injection makes the agent ask for broad access	Template ceiling, deny rules, the never-delegable list, the reason shown in every prompt, and a rate limit on requests
The agent approves its own request	Decisions only count with the user's own token
The agent claims a more permissive mode	The mode lives on the server
Escalation through admin tools	IAM admin rights are never delegable, so an agent can't raise its own user's access
Subagents combine into something no template allows	Review a template together with its child templates, and keep tool and outbound permissions in the ceiling
Data leaves through a tool IAM doesn't govern	Tool and outbound permissions are a kind of grant, enforced by a tool gateway
Background agents with nobody at the screen	Push approvals to the user's phone (CIBA) instead of letting saved rules quietly become standing access
Consent fatigue	Modes, saved rules, and templates whose context keeps requests small

The part I'm still arguing about: whose name goes in `sub`?

The convention, in RFC 8693 and in what Okta and AWS ship, is sub = the user and act = the agent. It reads naturally: the agent acts on behalf of the user.

I lean the other way: sub = the agent session, with the user as on_behalf_of. The reason is failure mode. In a large system, some service somewhere will look up permissions by sub. With the user in sub, that service hands the agent the user's full access. With the session in sub, it finds nothing and fails closed.

The cost is that standard libraries and MCP clients expect the convention. If you follow it, you need confidence that every service checks the token's own permissions and never re-derives them from sub. I'd love to hear which way you went and why.

Closing

Agents will soon outnumber the people in most identity systems. Least privilege has always been more of an aspiration than a default, because people's tasks are hard to bound. Agents are different: a session gives us a natural boundary, and a chat gives us a natural place to ask. Let's use both. Start every agent at zero, let the user lend what the task needs, and take it back when the session ends.

Sources

The CoreDNS Black Hole: how one dead DNS pod broke our API gateway

Akarshan Gandotra — Mon, 21 Sep 2026 11:53:12 +0000

Our API gateway started handing this back to roughly a third of all requests:

{"source":"auth","message":"Upstream Service Unavailable","error":"Upstream Service unreachable"}

Restart the gateway and everything is fine. A day later, it's back. No deploy, no code change, no traffic spike, nothing in the backend logs — the backends were healthy the whole time.

The cause was one open network socket pointing at a pod that no longer existed. The mechanism is genuinely strange, so let's start with an analogy.

The analogy: the receptionist who went home

Your gateway is a clerk at a busy counter. Before serving anyone, the clerk needs a phone number — "which desk handles billing today?" — so it calls the office directory.

A sensible clerk dials the directory's main switchboard every time. Our clerk did something subtly different: it dialled once, got through to Priya at the directory, and then kept that line open forever, because opening a line costs time and Priya always answers.

One night the building is reorganised. Priya's desk is removed. Her extension now goes nowhere.

Next morning the clerk picks up the same line, asks its question… and hears nothing. It waits. Still nothing. It gives up, tells the customer "billing is unavailable" — and then, crucially, asks the very same dead line again for the next customer. And the next. Forever.

Why doesn't it redial? Because the clerk only redials when the phone reports "this line is disconnected." Silence isn't an error. Silence is just silence, so it waits again.

That's the whole bug. Now swap in the real names:

Analogy	Reality
the clerk	an nginx worker process
the office directory	CoreDNS — Kubernetes' internal DNS
keeping one line open forever	nginx's resolver opens one long-lived UDP socket and reuses it
reaching Priya, not the switchboard	Cilium's socket load-balancer rewrites the connection to one specific CoreDNS pod
the building reorganisation	Karpenter replacing nodes, which replaces CoreDNS pods — about daily
silence vs. "disconnected"	a DNS timeout vs. a socket error. nginx reopens on error, never on timeout

If you haven't met these three before

CoreDNS is the cluster's phone book. In Kubernetes you don't address a pod by IP — you address a Service by name, like iamplatform.my-ns.svc.cluster.local. CoreDNS is the (usually two-pod) deployment that turns those names into IPs, and essentially everything in the cluster asks it, constantly.

Cilium is the networking layer. It does load-balancing inside the kernel using eBPF, instead of the older kube-proxy/iptables approach. The detail that matters here: when your app calls connect() on a Service IP, Cilium rewrites the destination there and then to one specific backend pod's IP. That's part of what makes it fast — and it means your socket is attached to that one pod, not to the Service as a whole.

Karpenter is the node autoscaler. It adds nodes when pods need room and — the part that matters here — removes under-used ones to save money. Pods on a removed node get rescheduled elsewhere, CoreDNS pods included. Completely routine behaviour.

Three reasonable components, each doing its job, combining into a permanent black hole.

One socket, glued to one pod. When Karpenter replaces that pod, the socket keeps pointing at nothing.

DNS lookup times out → nginx can't find the backend → 502 → our error handler turns that into the 503 the user sees. The backend was never even contacted.

Why it looked random

Each nginx worker has its own socket, so only the worker with the dead socket should fail — maybe 1/8 of traffic. We were seeing far more than that.

The reason was one line of config: accept_mutex off with no reuseport. Without reuseport, workers race to accept new connections and one worker reliably wins most of them. If that greedy worker is the one holding the dead socket, one broken worker is effectively the whole pod.

Without reuseport one worker wins most connections — so one dead socket looks like a dead pod.

That also explains the "intermittent" feel: whether a pod looked healthy or dead depended on which worker happened to be the greedy one.

We proved it instead of guessing

The temptation here is to ship a plausible fix and hope. Instead we wrote down predictions first — which pods would break, which worker in each, and roughly what failure rate — then deliberately deleted a live CoreDNS pod and checked.

The predictions matched. That mattered, because the first proposed fix (tune kernel socket settings, add more replicas) targeted a completely different theory — port exhaustion — that the evidence ruled out. Without the experiment we'd have shipped it and stayed broken.

Reading the sockets needs no special tooling:

# which CoreDNS pod is each worker's DNS socket glued to?
cat /proc/net/udp     # decode the hex peer address column

The fix: three layers

Failure rate during a deliberate CoreDNS pod deletion, at each layer of the fix.

1. Spread the traffic — listen 8080 reuseport. The kernel deals new connections evenly across workers, so a broken worker costs 1/N of a pod instead of all of it. This caps the damage; it doesn't prevent it.

2. Keep a spare line — list the same DNS name twice in the resolver directive. nginx opens one socket per listed address, so two entries means two independent sockets landing on (probably) two different CoreDNS pods. You must also raise resolver_timeout above nginx's hard-coded ~5s retry interval, or it never gets around to trying the second socket:

resolver  kube-dns.kube-system.svc.cluster.local
          kube-dns.kube-system.svc.cluster.local  valid=60s ipv6=off;
resolver_timeout 10s;   # was 3s — too short to ever reach socket #2

Honest cost: when the first socket is dead, requests wait ~5 seconds before the retry succeeds. That's a real, user-visible stall. This layer masks the problem, it doesn't heal it.

3. Take DNS off the request path entirely — the actual cure. Instead of looking up the address while a request waits, declare each backend as an upstream group that nginx resolves in the background and pools connections to (nginx 1.27.3+ in OSS; pooling explained two sections down, if the term is new):

upstream mt_iamplatform {
  zone   mt_upstreams 2m;
  server iamplatform.my-ns.svc.cluster.local:22036 resolve;  # background refresh
  keepalive 32;
}

The magic property: if a background refresh times out, nginx keeps the last known good address instead of failing the request. A DNS outage stops being an outage.

The numbers

Same experiment each time — delete a live CoreDNS pod:

Config	Requests that failed
Baseline	~37%
+ spread + spare socket	~3.4%
+ DNS off the request path	0 of 21,800

In the last run a worker was still holding a dead DNS socket the whole time. Nobody noticed, which is the point.

The rule we refused to break: nginx must always boot

Worth a detour, because it's the reason the fix looks the way it does.

If you name a backend literally — proxy_pass http://iamplatform.my-ns.svc.cluster.local:22036;, or a plain server line inside an upstream — nginx resolves that name once, at config load. Efficient. But if any one of those names fails to resolve — a service not created yet, a namespace being rebuilt, a typo in a new route — nginx prints [emerg] host not found in upstream and refuses to start at all. One missing backend takes down all ~50 routes, including the 49 that were fine.

So our routes deliberately use a variable:

set        $backend "iamplatform.my-ns.svc.cluster.local:22036";
proxy_pass http://$backend;        # a variable -> resolved per request

nginx can't resolve a variable at config load, so it doesn't try. The gateway boots knowing nothing about its backends and looks each one up at request time. A missing service now returns 502 on its own route, and nothing else notices.

A literal hostname makes every route depend on every backend at boot. A variable keeps the failure local.

Layer 3 had to preserve that. The obvious way to add pooling is a hard-coded upstream block — which hands straight back the boot-time dependency we'd avoided. Two details kept it safe:

server ... resolve tells nginx to skip the config-load lookup entirely and resolve in the background instead, so an unresolvable backend still can't block startup.
$mt_backend holds the upstream group's name, not a hostname — so proxy_pass is still pointed at a variable and never quietly became a literal.

set        $backend "mt_iamplatform";   # the group's name, not a host
proxy_pass http://$backend;             # this line never changed

A gotcha if you try this: nginx only matches a variable proxy_pass to an upstream group when the value contains no port. mt_iamplatform resolves to the group; mt_iamplatform:22036 is treated as a hostname and fails.

So we gained pooling and background DNS and kept "a broken backend is that backend's problem."

Connection pooling, in 60 seconds

Both the fix above and the bug below turn on this, so here's the short version.

By default, when nginx proxies a request it opens a fresh TCP connection to the backend, sends the request, reads the response, and closes it. Every request pays a three-way handshake, and every close parks a socket in TIME_WAIT on nginx for 60 seconds, holding one of ~28,000 ephemeral ports. At a few hundred requests a second, that adds up.

Pooling — nginx calls it upstream keep-alive — means: don't close it. Keep the connection and let the next request to the same backend ride the line that's already open.

Without pooling, every request pays a handshake and leaves a socket in TIME_WAIT. With it, one connection carries many requests.

Switching it on is three things, not one, and missing any of them leaves you with a pool that silently never engages:

upstream my_backend {
    server backend.my-ns.svc.cluster.local:8080;
    keepalive 32;              # how many idle connections to cache
    keepalive_requests 1000;   # reuse one this many times, then close it
    keepalive_timeout 60s;     # close it after this long idle
}

location /thing {
    proxy_pass http://my_backend;
    proxy_http_version 1.1;           # keep-alive needs HTTP/1.1
    proxy_set_header Connection "";   # don't forward the client's "Connection: close"
}

Four things that catch people out:

keepalive 32 is per worker process, not per pod. Sixteen workers means up to 512 idle connections from a single pod.
You need a named upstream block. A bare proxy_pass http://host:port; cannot pool at all.
Whoever closes first matters. Your keepalive_timeout has to sit below the backend's own idle timeout, or you'll eventually pick up a connection just as the backend's FIN arrives — upstream prematurely closed connection.
nginx only pools a connection whose response it finished reading. That one is a trapdoor, and it's the one we fell through.

Checking which you've got needs no extra tooling — count sockets to the backend from inside the pod:

# column 4 is the TCP state in hex: 01 = ESTABLISHED, 06 = TIME_WAIT
awk '$4=="01"' /proc/net/tcp | wc -l
awk '$4=="06"' /proc/net/tcp | wc -l

A healthy pool is a steady handful of ESTABLISHED and almost nothing in TIME_WAIT.

The bonus bug: a pool that never pooled

Measuring after a fix is how you find the next one. With DNS off the request path we looked at the gateway's sockets again: ~13,000 TIME_WAIT sockets per pod and zero pooled connections to our auth service — even though the config had declared keepalive 64 for it for years.

Back to the clerk. Before serving anyone, it also has to ask the auth desk "is this person allowed in?" The clerk only needs the answer written on the envelope — yes/no, and who you are. But the auth desk kept stuffing a full brochure inside every reply. And the clerk has a rule: I can only keep the phone line open if the reply is envelope-only. So it hung up after every call and redialled for the next customer.

In nginx terms: auth_request marks its subrequest header-only and never reads the response body — and nginx returns the connection to its keep-alive pool only when the response has no body (204, or a non-chunked Content-Length: 0). Our auth service answered with JSON, so the pool could never engage. Not just untidy: at ~470 connections per second to one backend pod you run out of ephemeral ports, and during an auth-service rollout — when there may be only one pod left — a busy gateway sits uncomfortably close to that.

The fix belonged in the auth service. But that service has its own ingress and other consumers who genuinely want the body, so stripping it for everyone would break them.

Tag the caller, not the endpoint

The gateway already stamps every auth subrequest with a header:

proxy_set_header X-Sent-From "nginx-pods";

So the auth service now drops only the body — never the status code, never the headers — for requests carrying that tag:

// middlewares.HeaderOnlyForGateway, wired on POST /auth
if !strings.EqualFold(strings.TrimSpace(c.GetHeader("X-Sent-From")), "nginx-pods") {
    c.Next()   // every other caller: unchanged, body and all
    return
}
c.Writer = &headerOnlyWriter{ResponseWriter: c.Writer}  // keep status + headers,
c.Next()                                                // discard the body

Nothing is lost, because the gateway reads only the response headers (X-Identity-*, X-Session-ID, X-Auth-Error-*) and builds its own client-facing JSON from them. Handlers keep calling c.JSON(...) exactly as before — they don't know the middleware exists.

A header tags the caller, so one shared endpoint can answer the gateway and everyone else differently.

One easy-to-miss detail: once pooling works, the two sides' idle timeouts must be ordered. Go's IdleTimeout is set to 120s, deliberately above the gateway's keepalive_timeout 60s, so nginx is always the side that closes an idle connection. If the server closed first, the gateway would sometimes pick one up just as its FIN arrived — surfacing as upstream prematurely closed connection.

Takeaways

A long-lived connection is a cached decision. Anything you connect to once and keep — DNS sockets, pooled connections, resolved IPs — is a bet that the thing on the other end won't be replaced. In Kubernetes, it will be.
Timeout ≠ error. Many clients recover from a refused connection and hang forever on silence. Ask which one your retry logic keys on.
Write your predictions down before you break something. It's the cheapest way to tell a real diagnosis from a plausible story — and it's what stopped us shipping the wrong fix.
Fail small, not early. Resolving every dependency up front feels safer, but it couples every route to every backend. Late binding — looking things up when you actually need them — is what keeps one missing service from becoming a full outage.

The gateway is nginx on Kubernetes with Cilium and Karpenter, but the shape of this bug isn't nginx-specific. Any client that holds one socket to one replica of a service has it.

A Practical Guide to Using Auth0 as an Identity Broker for Epic SMART on FHIR

Akarshan Gandotra — Mon, 03 Aug 2026 11:30:36 +0000

Your team already uses Auth0. Now someone wants to add Epic. On paper, it looks like one more enterprise connection — Epic speaks OAuth 2.0 and OpenID Connect, Auth0 brokers OIDC connections all day, so this should be a config change, not a project.

It isn't, and that has nothing to do with Epic being difficult. SMART on FHIR isn't just a login — it's a login wrapped around a second, quieter handoff: which FHIR server to talk to, which patient or encounter the user was looking at, and which scopes let your app read any of it. Auth0 has no idea any of that exists. Configure the connection the way you would for any other enterprise IdP, and the login will work perfectly in the demo, then quietly drop the one piece of context your application actually needed.

This guide is for teams in exactly that spot: Auth0 is already the broker, Epic is the new connection, and the question is how to get through that hop without losing what SMART on FHIR needs to carry.

TL;DR — Auth0 can broker Epic SMART on FHIR logins, but only if you stop treating it as a single login and start treating it as three separate relationships: app↔Auth0, Auth0↔Epic, and (if you call FHIR APIs) Epic-token↔FHIR API. The parts that actually break in practice: iss/aud/launch not surviving the trip through Auth0 unchanged, Epic's SMART discovery document not being the same thing as Auth0's OIDC auto-discovery, and the Epic access token never reaching your backend unless you go get it deliberately.

The architecture looks simple on a whiteboard:

Application → Auth0 → Epic

The application integrates with Auth0, and Auth0 handles authentication with Epic through an OIDC enterprise connection. That works well, especially when your platform already supports multiple identity providers through Auth0 — but only once the SMART-specific pieces below are made explicit instead of assumed to ride along for free.

Understanding the authentication flow

A brokered login is really three relationships stacked on top of each other:

Application ↔ Auth0 — the login relationship your app already knows how to build.
Auth0 ↔ Epic — the enterprise connection Auth0 brokers on your app's behalf.
Epic access token ↔ Epic FHIR API — only relevant if your application calls FHIR APIs directly, not just authenticates users.

Keeping these three boundaries separate is the most important design decision in this whole integration. Concretely, for an EHR launch, the hops look like this:

Step	Hop	What happens
1	User → Application	EHR launch lands on your app with `iss` and `launch`
2	Application → Auth0	Redirect to Auth0's `/authorize`
3	Auth0 → Epic	Redirect to Epic's `/authorize` — the upstream leg of the enterprise connection
4	User ↔ Epic	User authenticates (often silent — they're already inside the EHR)
5	Epic → Auth0	Redirect to Auth0's callback with an authorization code
6	Auth0 ↔ Epic	Back-channel code exchange, server-to-server, using Auth0's own client credentials
7	Auth0 → Application	Redirect to your app's callback with Auth0's own code
8	Application ↔ Auth0	Your app exchanges that code for its own tokens

By default, your application only ever sees the result of step 8. Steps 5–6 happen entirely inside Auth0, unless you deliberately configure a way to retrieve Epic's token afterward — more on that later, in Plan how the backend will obtain the Epic token.

Start by defining the purpose of the integration

Before you touch the Auth0 dashboard, decide what the application actually needs from Epic — this one decision determines almost everything else in this guide.

Using Epic only for authentication

In the simpler model, Epic is used only to identify the user. The application may need claims such as sub, name, email, and fhirUser. Once the user has authenticated, Auth0 issues its own tokens and the application creates its own session. The application never touches the Epic access token.

This is the easiest model to support through an Auth0 enterprise connection, and if it's all you need, most of the friction in this guide disappears.

Calling Epic FHIR APIs

The second model is more involved. The application needs to authenticate the user, obtain an Epic access token, preserve SMART launch context, request appropriate FHIR scopes, use the Epic token against the correct FHIR server, and potentially refresh or renew that token.

Here, the Auth0 token and the Epic token serve different purposes and are not interchangeable. An Auth0 access token is generally intended for your own application APIs. An Epic access token is intended for Epic FHIR APIs. Mixing them up is the single most common design mistake in this kind of integration.

Register the callback correctly in Epic

When Auth0 acts as the broker, Epic redirects the authorization response back to Auth0, not directly to your application. The callback registered in Epic follows this exact pattern:

https://<auth0-domain>/login/callback

That's Auth0's fixed callback path for enterprise connections, regardless of which upstream IdP you're connecting — https://<tenant>.us.auth0.com/login/callback on a standard domain, or https://<your-custom-domain>/login/callback with a custom Auth0 domain.

Your application has a separate callback configured in Auth0:

https://<application-domain>/callback

Registering your application's callback directly in Epic will not work here — Epic has no way to reach your application. It only knows about Auth0.

Before testing, verify that the callback registered in Epic exactly matches the one Auth0 generates: protocol, domain, path, trailing slash, and custom Auth0 domain if you use one. OAuth callback matching is generally exact, and this is the single fastest way to lose an afternoon to a one-character mismatch.

Configure Epic as an OIDC enterprise connection

Epic can be added to Auth0 as an OpenID Connect enterprise connection. A typical configuration needs an issuer, authorization endpoint, token endpoint, JWKS endpoint, client ID, client secret, scopes, and connection type.

Use discovery metadata instead of guessing endpoint URLs — but be precise about which discovery document you're reading. Epic's FHIR servers publish SMART-specific metadata at:

{fhir-base-url}/.well-known/smart-configuration

(available on Epic releases from roughly August 2021 onward — older instances may only expose the FHIR CapabilityStatement at {fhir-base-url}/metadata). It looks like this:

{
  "issuer": "https://<epic-host>/...",
  "authorization_endpoint": "https://<epic-host>/.../authorize",
  "token_endpoint": "https://<epic-host>/.../token",
  "jwks_uri": "https://<epic-host>/.../jwks"
}

Gotcha: this is not automatically the same document as the generic OIDC discovery Auth0 expects when you point its connection setup at an issuer for auto-discovery — {issuer}/.well-known/openid-configuration. Auth0's "fetch from issuer" convenience option quietly failing against a raw Epic FHIR base URL is one of the most common first-hour surprises. Plan to fetch the SMART configuration document yourself and paste authorization_endpoint, token_endpoint, and jwks_uri into Auth0's manual fields.

Different Epic environments use different issuers, OAuth endpoints, FHIR base URLs, signing keys, and client registrations — and each Epic app record needs the relevant SMART capabilities (launch-ehr, client-confidential-symmetric, sso-openid-connect, and so on) enabled for the flow you're building. A working sandbox configuration tells you almost nothing about whether a hospital's production Epic instance will behave the same way.

Use the authorization-code flow

Configure the Auth0 enterprise connection to use the back-channel authorization-code flow. This isn't just a conceptual choice — it's a literal Type setting on the connection (Auth0 Dashboard → Authentication → Enterprise → your OIDC connection), toggled between Front Channel and Back Channel:

{
  "strategy": "oidc",
  "options": {
    "type": "back_channel"
  }
}

Back-channel makes Auth0 send an upstream request with response_type=code, receive the code from Epic, and exchange it at Epic's token endpoint using Auth0's own client credentials.

Note: PKCE, if you use it, protects only the browser-to-Auth0 leg. It does not extend to the Auth0-to-Epic exchange — that hop is authenticated with Auth0's client secret, not your user's proof-of-possession.

A front-channel configuration instead generates a request with response_type=id_token and response_mode=form_post — closer to an implicit flow, where Epic returns tokens directly in the browser response instead of a code Auth0 exchanges server-side. That's not what a SMART application built around Authorization Code Flow expects.

When validating the setup, inspect the actual Epic authorization URL your browser is redirected to and confirm response_type=code is present. Don't trust the dashboard setting alone — the generated browser request is the source of truth.

Enable the connection for the Auth0 application

Creating an enterprise connection doesn't automatically make it available to every Auth0 application. After creating the Epic connection, enable it for the specific application that will initiate the login.

With Auth0 Organizations in the picture, the connection may also need to be enabled separately for the relevant organization. That's up to four layers, all of which have to be valid before login completes:

Connection exists
Connection enabled for application
Connection enabled for organization
User allowed in organization

Miss any one of these and the failure mode looks identical from the outside: the user just can't log in.

Configure the required SMART scopes

A standard OIDC login might request openid profile email. SMART applications need more. A common starting point for an Epic EHR-launched application is:

openid fhirUser launch

Each scope earns its place.

`openid`

Enables OpenID Connect and allows an ID token to be returned.

`fhirUser`

Requests the FHIR identity associated with the authenticated user — a reference such as Practitioner/123, Patient/456, or RelatedPerson/789. This is usually more useful than email for linking the authenticated identity to a healthcare user, and it's the scope most teams forget until they need it.

`launch`

Signals that the application expects EHR launch context. The actual launch value travels separately, as launch=<opaque-launch-value>.

`profile`

Still worth including when you need standard profile claims, but it does not replace fhirUser — they answer different questions.

A reasonable starting configuration:

openid profile fhirUser launch

The final scope list depends on the FHIR resources the application actually needs — patient/Patient.read, patient/Observation.read, user/Practitioner.read, and so on. Request the minimum that lets the app do its job; every extra scope is one more thing to justify during Epic's app review.

Preserve the SMART launch parameters

An EHR-launched SMART application is normally opened with parameters like:

iss=https://epic.example.com/api/FHIR/R4
launch=opaque-value

Treat launch as opaque. Don't decode it, modify it, generate a replacement, or assume anything about its internal format — store it temporarily and forward it as-is. iss represents the FHIR server that initiated the launch.

A simplified request flow looks like this:

GET /smart/launch?iss=https://epic.example.com/api/FHIR/R4&launch=opaque-value

The application then starts authorization with:

scope=openid fhirUser launch
aud=https://epic.example.com/api/FHIR/R4
launch=opaque-value

The mapping that matters is aud = iss received during the SMART launch. That value is not the Auth0 audience, the Epic client ID, the Auth0 client ID, the OIDC issuer, or the application callback URL — it's specifically the FHIR resource server the application wants to access, and it's worth saying that plainly because it's the single most common source of confused config in this whole flow.

Validate the incoming `iss` value

In a multi-tenant healthcare platform, don't trust the iss parameter just because it showed up in the browser — it arrives via an unauthenticated redirect, so treat it like any other user-controllable input. Validate it against a configured list of supported Epic environments:

SUPPORTED_EPIC_ISSUERS = {
    "https://hospital-a.example.com/api/FHIR/R4",
    "https://hospital-b.example.com/api/FHIR/R4",
}

Before starting the authorization flow:

if iss not in SUPPORTED_EPIC_ISSUERS:
    raise InvalidIssuerError()

Skip this check and you've built an open redirector that will happily start an authorization flow against whatever FHIR server shows up in a query string — including one an attacker controls. For a larger platform, store the trusted configuration per tenant:

Tenant A → Epic issuer A, Epic client ID A, Epic connection A
Tenant B → Epic issuer B, Epic client ID B, Epic connection B

Resolve the tenant first, from something you trust, then validate the Epic issuer against that tenant's configuration — never derive the tenant only from an untrusted browser parameter.

Separate EHR launch from standalone launch

SMART supports multiple launch styles. Don't treat them as the same flow wearing different clothes.

EHR launch

The application is launched from within Epic. It receives iss and launch, and requests the launch, openid, and fhirUser scopes. The launch context may include patient, encounter, tenant, need_patient_banner, and smart_style_url.

Standalone launch

The application starts outside the EHR. There's no incoming launch value. Instead, the application requests context through scopes like launch/patient and launch/encounter.

Give these two flows separate entry points — /smart/launch for EHR launch, /login/epic for standalone — rather than one endpoint branching on whether a launch parameter happens to be present. It's easy for EHR-specific assumptions to leak into the standalone path when they share code, and hard to notice until a standalone user hits an edge case nobody tested.

Pass the correct parameters upstream

Auth0 sits between the application and Epic, so whatever it receives has to actually reach Epic's authorization endpoint — aud, launch, scope, and login_hint are the ones that matter most, and whether they pass through dynamically depends entirely on how the enterprise connection is configured. Test this early; it's not something you want to discover during a hospital pilot.

Capture the authorization request Auth0 actually sends to Epic and confirm it contains:

client_id=<epic-client-id>
redirect_uri=https://<auth0-domain>/login/callback
response_type=code
scope=openid fhirUser launch
aud=<epic-fhir-base-url>
launch=<original-launch-value>

If aud or launch is missing, the user can still reach Epic's login page and authenticate successfully — the resulting session just won't contain the SMART context your application needs. A successful login is not the bar here. The final token response has to contain the context, not just a valid identity.

Use `state` to correlate the complete flow

state does two jobs: protecting the authorization flow from CSRF, and correlating the returning response with the original launch. For a SMART integration, that means associating the attempt with the tenant, the Epic issuer, the launch value, the original application URL, the Auth0 organization, and the requested connection.

Don't put any of that directly into a browser-visible state string. Generate a random value, store the metadata server-side against it, and send only the random value in the authorization request:

{
  "state": "random-value",
  "tenant_id": "tenant-a",
  "epic_issuer": "https://hospital-a.example.com/api/FHIR/R4",
  "launch": "opaque-value",
  "return_to": "/patient-chart"
}

The browser only ever sees state=random-value. On callback, the application looks up the stored metadata and validates it. That state entry should expire quickly, be single-use, be bound to the browser session where possible, and be deleted the moment the callback finishes processing it.

Understand how Auth0 Organizations affect login

Organizations add an authorization layer after Epic authentication. A user can authenticate with Epic perfectly and still get rejected by Auth0 because they're not permitted in the requested organization.

Before relying on an organization-based login, verify: the organization exists, the Epic connection is enabled for it, the connection is enabled for the Auth0 application, the user can become a member (with automatic membership configured if that's the intent), and the application is passing the correct organization ID.

Don't assume every access_denied is Epic saying no — check where in the flow it actually happened. An error that surfaces after the browser has already returned from Epic to Auth0 is usually about organization membership, an Auth0 Action, a connection policy, or user linking, not Epic.

Recognize this exact message: if every connection enabled for an organization is an enterprise connection, and all of them are hidden from the login prompt, Auth0 returns "no connections enabled for the organization are visible." That's a pure Organizations misconfiguration. Don't spend an afternoon staring at Epic's side of the fence looking for a bug that isn't there.

Decide how users will be linked

Epic's identity may not map cleanly onto identities you already have. Candidates include the Epic OIDC sub, a FHIR user reference, email, employee ID, practitioner ID, or some existing external-identity mapping.

Don't link accounts by email alone unless your security model explicitly allows it. Email changes, goes missing, differs in casing, gets shared, and often doesn't match between Epic and the enterprise directory anyway.

A composite key holds up better:

tenant-a + epic + oidc-subject-123

That gives you a stable external identity key. Store fhirUser as related healthcare identity metadata alongside it, not as the primary login identifier — it answers "which FHIR resource is this person," not "which account is this."

Keep the Auth0 token and Epic token separate

After the brokered flow, you may be holding an Epic ID token, Epic access token, and Epic refresh token; an Auth0 ID token and Auth0 access token; and your own application session token. Each has a different issuer, audience, and job:

Auth0 ID token — tells the application who the authenticated Auth0 user is.
Auth0 access token — calls an API protected by Auth0.
Epic access token — calls Epic FHIR APIs.

Validate issuer, audience, expiry, scope, and intended API before using any of them. Never send the Auth0 access token to the Epic FHIR API. Never send the Epic access token to an internal API expecting an Auth0 token. Your APIs should reject tokens minted for a different audience — don't rely on discipline alone to keep these apart.

Plan how the backend will obtain the Epic token

Here's the part that surprises people: when Auth0 performs the upstream code exchange, Auth0 receives the Epic access token — not your application. Getting it from there to your backend takes a deliberate decision, not a default.

Auth0 gives you two supported paths:

Read it from the user's identity record. The upstream token shows up at user.identities[0].access_token (and, for a short allowlist of named providers, a refresh token alongside it). Getting it out requires a Management API call to the Get a User endpoint with the read:user_idp_tokens scope — your frontend can't read this directly, nor should it be able to.
Use Auth0's Token Vault, where available, which stores upstream access/refresh tokens in a dedicated secure tokenset instead of on the identities array.

Test this before you design around it: refresh-token persistence for a custom OIDC connection like Epic is not guaranteed the way it is for a short list of named providers (Google, Azure AD, Bitbucket, SharePoint). Confirm Auth0 is actually retaining Epic's refresh token before you build a refresh strategy that assumes it'll be there — finding out otherwise during a production incident is a bad way to learn this.

If neither mechanism reliably gives you what the application needs — particularly long-lived refresh capability — your remaining options are performing the SMART authorization-code exchange directly in your own backend (bypassing Auth0 for the Epic-specific leg), or building a separate backend service purely responsible for Epic token lifecycle: refresh, revocation, expiry, per-tenant isolation. This is usually the fork in the road between "Auth0 brokers everything" and the hybrid architecture covered later in Should Auth0 sit in the middle?

Whatever you choose, don't put the full Epic access token inside an Auth0 custom claim. That token would then ride along inside browser-visible ID tokens, and from there into logs, session storage, frontend debugging tools, and analytics — none of which should ever see a FHIR access token. Keep it backend-only.

Store launch context separately from the user identity

SMART context belongs to a specific authorization session. It should never quietly become permanent user profile data.

User identity:  Epic subject, FHIR user, tenant
SMART session:  patient, encounter, FHIR server, scopes, Epic access token, token expiry, launch context

A clinician launches the same application for a different patient every hour. If the patient ID lives on the permanent user identity, the second launch corrupts the first. Attach it to the session instead:

{
  "session_id": "session-123",
  "user_id": "user-456",
  "tenant_id": "tenant-a",
  "fhir_server": "https://hospital-a.example.com/api/FHIR/R4",
  "patient": "patient-789",
  "encounter": "encounter-101",
  "expires_at": "..."
}

Delete or expire it when the session ends. It shouldn't outlive the reason it existed.

Validate the token response

Once Epic completes the authorization-code exchange, actually check what came back.

For the ID token: signature, issuer, audience, expiry, issued-at time, nonce, and subject. For the Epic access token and SMART context: that the access token and its expiry exist, the scope matches what you requested, the FHIR server matches the trusted issuer, and required patient/encounter context and fhirUser are present when your flow needs them.

Don't assume every successful token response includes patient context — what comes back depends on launch type, requested scopes, Epic's app configuration, user authorization, and whatever EHR context was actually available. If the application requires a patient and none arrives, stop and show an appropriate message. Don't fall back to making unscoped FHIR requests just because the login technically succeeded.

Add structured logging around each redirect boundary

Brokered OAuth flows are hard to debug precisely because the browser hops across multiple domains and you can't just attach a debugger to the whole journey. Add structured logs at each major step:

{"event": "smart_launch_received", "tenant_id": "tenant-a", "issuer": "https://hospital-a.example.com/api/FHIR/R4", "has_launch": true}
{"event": "auth0_authorization_started", "tenant_id": "tenant-a", "organization_id": "org_123", "connection": "epic-hospital-a"}
{"event": "auth0_callback_received", "tenant_id": "tenant-a", "state_valid": true}
{"event": "smart_session_created", "tenant_id": "tenant-a", "has_patient": true, "has_encounter": false, "scope_count": 5}

Never log the authorization code, access token, refresh token, client secret, full ID token, or opaque launch value.

Check your middleware, not just your handlers: application-level discipline about redaction is easy to undo one layer down. A blanket "log every request" middleware — the kind teams add for general observability — will often capture the Authorization header, the full query string (which contains code= and launch= on a redirect), or a raw JSON request body at a token-exchange endpoint. Audit the entire path a request takes through your stack, not just the function that handles it.

Test each integration layer independently

Don't debug the complete brokered flow as one unit — isolate it:

Epic discovery — the discovery, token, and JWKS endpoints are reachable.
Direct Epic authorization request — the Epic client registration accepts the intended response_type, redirect_uri, scope, aud, and launch.
Auth0 enterprise connection — Auth0 can authenticate against Epic with no organization restrictions in play yet.
Auth0 Organizations — add the organization and membership rules only after the base connection works.
SMART launch propagation — iss, aud, and launch actually reach Epic intact.
FHIR API access — the Epic-issued access token works against the intended FHIR endpoint.

Six stages, six separate places a bug can hide. Collapsing them into one end-to-end test is how you end up guessing.

Production checklist

Epic registration

[ ] Correct Epic environment
[ ] Correct client ID and client secret
[ ] Auth0 callback registered
[ ] Authorization Code Flow enabled
[ ] Required FHIR APIs and scopes approved

Auth0 connection

[ ] OIDC enterprise connection created
[ ] Back-channel flow configured
[ ] Issuer, token, and JWKS endpoints configured (manually, if Epic doesn't expose standard OIDC discovery)
[ ] Connection enabled for application
[ ] Connection enabled for organization

SMART flow

[ ] Incoming iss validated against a trusted, per-tenant allowlist
[ ] Launch value treated as opaque
[ ] aud matches the FHIR server
[ ] Correct scopes requested
[ ] EHR and standalone launch kept as separate entry points
[ ] State stored securely, single-use, and expiring
[ ] Nonce validated

Identity

[ ] Stable external identity key selected (not email alone)
[ ] Account-linking rules defined
[ ] fhirUser handled as metadata, not a primary key
[ ] Organization membership configured
[ ] Tenant isolation enforced

Tokens

[ ] Auth0 and Epic tokens stored separately
[ ] Epic token remains backend-only
[ ] Token audiences validated on every API
[ ] Refresh strategy defined and tested against the real connection, not assumed
[ ] Revocation behavior and expiry handling defined

Session

[ ] Patient and encounter context stored per session, not on the user record
[ ] Launch context expires
[ ] Logout behavior defined
[ ] Audit logs added, with sensitive values excluded — including at the middleware/framework logging layer, not just application code

Should Auth0 sit in the middle?

Auth0 earns its place when your platform already runs on it, Epic is one of several enterprise identity providers, you need centralized organization and session management, Epic is mainly there for authentication, the SMART parameters forward reliably, and the Epic access-token lifecycle has a clear owner.

A direct SMART integration starts looking better when the application serves many Epic customers with different FHIR issuers, launch parameters are highly dynamic, the app leans heavily on patient and encounter context, you need direct control over PKCE, refresh-token management is central to the product, or getting SMART-specific parameters through Auth0 keeps fighting you.

Auth0 can also stay part of the broader platform without brokering the Epic SMART flow itself:

Epic SMART authorization
  → Application validates Epic identity
  → Application links internal user
  → Internal application session

The right architecture comes down to one question: is Epic acting as your identity provider, your FHIR authorization server, or both? The answer isn't the same for every product, and it's worth answering deliberately instead of defaulting to whatever pattern your last OIDC integration used.

Final thoughts

Using Auth0 as an identity broker for Epic SMART on FHIR is possible, but it's not a standard enterprise OIDC login wearing a different logo. Keep the responsibilities explicit:

Epic manages — SMART authorization, FHIR scopes, FHIR launch context, FHIR access tokens.
Auth0 manages — enterprise identity brokering, organizations, Auth0 sessions, Auth0 tokens.
Your application manages — tenant selection, launch validation, state correlation, SMART session context, FHIR API usage, token isolation.

Once those boundaries are explicit, the integration becomes something you can actually design, test, and operate — instead of something that works until the first real hospital points a real EHR launch at it. The question was never whether Epic supports OIDC. It does. The question is whether every SMART-specific value your application depends on can move through the identity broker without losing its meaning along the way.

Part 10 — Lessons learned building a Kubernetes Auth Gateway

Akarshan Gandotra — Mon, 04 May 2026 18:42:44 +0000

We're at the end of the series. Nine chapters of mechanism. One chapter of opinion.

Building the Auth Gateway took roughly two years from "what if NGINX did the auth?" to "this thing handles every authenticated request in production." A lot of what's in the previous chapters wasn't obvious to us at the start. This is the post-mortem on our own architecture: what worked, what hurt, what we'd build earlier, and what we'd warn the next team about.

What worked

A few decisions held up cleanly. We'd make all of them again.

`auth_request` as the primitive

NGINX's auth_request directive is, with no exaggeration, the single most leveraged design choice in the platform. One directive, well-understood, supported across NGINX versions. We don't need a service mesh. We don't need a custom Envoy filter. We don't need a Lua module compiled into NGINX.

If you can do your auth in HTTP-status terms (200/401/403), auth_request is the right tool. If you can't, you probably want a sidecar or mesh-level enforcement and this whole architecture doesn't apply.

Endpoint metadata as data

Storing endpoint type and required permissions in Postgres, refreshed via Pub/Sub, was the right call. It means:

We can change auth without redeploying the gateway.
We can audit auth ("what protects this URL?") with a SQL query.
Admin tooling and the gateway share a single contract.

The cost — a small DB lookup at boot, an in-memory trie, a refresh mechanism — was tiny compared to the operational flexibility we got back.

One structured log line per decision

AUTH_DECISION is the contract between the Auth Service and oncall. Every field, every time, every request. A year of operations later, this is the artifact we reference most often. Every alert we've built points at it. Every incident postmortem references it.

Resist the temptation to add INFO/DEBUG lines around it. Resist the temptation to omit fields when they're "not relevant." One line. Same shape. Forever.

Fail-closed by default at the edge

error_page 502 503 504 = @auth_unavailable; was a one-line change that defines our security posture. When the Auth Service is unhealthy, NGINX returns 503 to the client instead of letting the request through. We've had a few incidents where this caused brief platform-wide outages. We have never regretted the choice.

The principle: the cost of a 5-minute outage on rare occasions is much, much less than the cost of one cross-tenant data leak ever.

Caches in the auth process, not in NGINX

auth_request is intentionally not cacheable, and we leaned into that. Every cache lives inside the Auth Service: JWT verify, RSA keys, route lookup, policy bitmap, revocation map, SA versions. Each is invalidated through its own channel. The gateway's hot path makes zero Redis calls in steady state.

This kept the architecture honest. The auth pod is the unit of correctness. Scale it, monitor it, debug it as one thing.

Pub/Sub-driven trie reload

Push-based invalidation for the endpoint trie was the right shape. Periodic-only would have given us a ~30 minute window where new admin routes were unprotected. Pub/Sub-only would have been brittle (events get lost). Both, with periodic as the safety net, gives us seconds of staleness in the common case and bounded staleness even when the message is lost.

Most caches we'd default to TTL. The trie was worth the special case.

The bitmap fast path

Encoding permissions as bit indexes paid off. Smaller tokens, faster checks, cleaner metrics. The legacy path we kept around for safety has earned its keep — version skew is real, and fall-through is graceful.

Shadow mode for two months before flipping the switch was the right rollout pattern. Catching three real bugs in shadow with zero impact on production is the gold standard for a sensitive change like authorization logic.

What hurt

Now the harder list. Things that cost us time, sleep, or trust.

Tenant resolution living in two places

NGINX resolves the tenant. The Auth Service also checks tenant binding (token tenant matches request tenant). The two places do different checks for a reason — but the reason isn't obvious, and we've watched several engineers add tenant logic to a third place because they didn't realize it was already covered.

What we'd do differently: write a single tenant-resolution doc that explicitly enumerates which layer owns what and what each layer assumes about the others. A "tenancy contract" page. We have it now (chapter 5 is a recovered version of it); we should have had it on day one.

The "first segment is the slug" rule

For a long time, the Auth Service split the URI on / and treated the first segment as the service slug. This worked until services started nesting each other or grouping under shared prefixes. We had to retrofit X-Service-Slug and X-Request-Path headers — backward-compatibly, with fallback to the old rule. The retrofit is fine; it took longer than it should have because the old rule was buried in three places.

What we'd do differently: explicit slug headers from day one. Don't infer slugs from the URI structure. The path inside a service is the service's business; the slug is a separate concern.

Migration to the bitmap took longer than expected

The bitmap fast path was a six-week project that took five months. The math was straightforward. What ate the time:

Coordinating bit-index assignments with the token issuer team (different repo, different rollout cadence).
Fixture data in our test suites was hardcoded with old permission strings; updating it for the bitmap registry was a long tail of small PRs.
The shadow comparison logic exposed three subtle bugs (Chapter 6) that each required investigation.

What we'd do differently: assume cross-team auth changes are 3x what you estimate. Build the shadow harness first, then the new path. The shadow harness paid for itself five times over.

Cache invalidation was an afterthought

The first version of the JWT cache was a map with time.AfterFunc evictors. We covered it in Chapter 8. It seemed fine. It fell over in production within a week.

The lesson generalizes: a cache without a written-down invalidation channel is a memory leak. Every cache should have:

A bounded size (entries or bytes).
A clear invalidation event ("token expired", "trie reloaded", "revocation event consumed").
A staleness window we can articulate ("up to 30 seconds late").

If you can't write those three down, don't add the cache.

The default tenant we shipped on day one

For the first quarter we had a default tenant. "If no X-Tenant-ID and no host match, fall through to default-tenant." It was added because it made local dev easier.

It cost us in two ways. First, removing it took longer than building it — every misconfigured client started 400'ing once we removed the fallback. Second, while it was live, it produced exactly one near-miss data leak (a service-account request without a tenant header writing into the wrong tenant). We caught it before it left staging.

Shipping that default tenant was the worst single decision in the whole project. We'd remove it from every future system before it ever boots.

Per-pod alert spam, twice

Twice we shipped alert code that fired per-request rather than per-state. The first time was during a Redis outage in our second month (lit up Slack with ~10k messages in 90 seconds). The second was during an RSA misconfig rollout (a few hundred messages per minute per pod, fleet-wide).

Both were the same bug: alerting from a request handler instead of from a state-transition observer. Both were "fixed" with atomic.Bool swaps. Now we apply that pattern aggressively.

What we'd do differently: write the alert dedup helper first, before the first alert. Have it baked into the codebase before there's anything to alert on.

What we'd build earlier

In the order we'd add them:

1. The structured `AUTH_DECISION` log

On day one. Even before fancy auth logic. The log structure outlives every other choice — every dashboard, every alert, every postmortem reads from it. Build the contract first.

2. Slack alert dedup helper

Before the first alert. Five lines of code to wrap an atomic.Bool around a Slack send. Ship it before you have anything to alert on.

3. Fail-closed posture

Before the gateway sees a single request in production. Don't even try permissive defaults. The "we'll tighten it later" path becomes "we shipped a permissive-by-default thing for 18 months." Just ship it tight.

4. Endpoint metadata in DB

Skip the YAML-of-routes phase. Skip the in-code decorator phase. Go straight to the database table with refresh mechanism. The transitional architectures cost more to migrate off than they cost to skip.

5. Gap probe on revocation streams

The probe (Chapter 7) catches data loss between the stream and consumers. It costs almost nothing to run. Without it you don't know if you're losing events; you just hope.

6. Shadow harness for sensitive changes

Comparing old-vs-new in production with the new path muted is a powerful pattern. Build the harness as a reusable thing. We re-implemented variants of it for three different rollouts before realizing it should be a library.

7. Tenancy contract document

One page. Owns: which layer resolves the tenant, which layer validates token-tenant binding, which layer scopes queries, what the failure modes are. Required reading before anyone touches request handling. Should have existed before the gateway shipped.

The maturity progression

Looking back, the gateway evolved through identifiable stages. They're worth naming because if you're starting fresh, knowing the destination shape lets you skip steps.

v1 — per-service auth libs. Where most teams are. Each service has its own JWT decode, its own permission check. Inconsistent, drift-prone, slow to fix CVEs. Don't stay here.

v2 — auth_request + minimal /auth. A simple gateway that decodes a token and returns 200/401. Static list of "open" routes. Enough to centralize the decision; not enough to scale.

v3 — trie + classification + Pub/Sub. Endpoint metadata in a DB. Trie in memory. Refresh kicks. Now adding a route doesn't require a redeploy.

v4 — revocation + caching. Logout works. Admin disable works. Each cache layer in place. Hot path is sub-millisecond.

v5 — bitmap + structured logs + degraded mode. The mature gateway. Fast, observable, alertable, recoverable.

Most of the value lives between v2 and v3. If you're at v1, that's the migration to plan for. v3 to v5 is iteration; v1 to v3 is the project.

Five pieces of advice for teams building this

If you're starting from scratch with a similar problem, here's what I'd hand off in five bullets:

1. Start with `auth_request`. Don't shop architectures.

Service mesh, custom Envoy filter, Lua plugin, sidecar — they all promise more flexibility. They all cost more in operations. auth_request is enough for the 90% case, and the 10% is rarely worth the complexity.

2. Make the gateway HA before anything else.

Two replicas minimum, HPA, graceful shutdown, retries to upstream auth pod, circuit-breaker semantics in NGINX, fail-closed posture. If any one of these is missing the gateway will take down your platform during a normal degraded event. This isn't optional.

3. The log is the API.

The AUTH_DECISION log is a public contract with everyone who ever debugs your gateway. Treat it like a schema. Don't change field names without a migration. Don't add free-form strings to enum fields. Have one version-controlled doc that defines every field and every value of every enum.

4. Cache invalidation has to be explicit.

Every cache: bounded size, explicit invalidation channel, articulable staleness window. If a cache doesn't have all three, it's a bug-in-waiting. We learned this twice.

5. Build observability before you build features.

Dashboards, alerts, trace context, the structured log — all of these come before you ship the cool feature you're excited about. A clever new permission model that you can't observe is worse than a boring permission model you can.

What we'd build next

A few things on our list that didn't fit this series:

NGINX otel module compiled in. Right now NGINX traces are limited; the Auth Service has full spans, but the NGINX hop is a black box from the trace's point of view. Worth fixing.
Per-tenant rate limiting. Currently we rely on upstream services. The gateway is the natural place.
WAF integration. We have an external WAF. Closer integration so WAF events show up in AUTH_DECISION would help triage.
Token introspection cache. Some integrations issue opaque tokens that we have to introspect with the issuer. Caching that lookup is its own caching problem; we haven't tackled it.
A formal "tenancy contract" page. Yes, the same one I told you to build on day one. We're catching up.

Each of these is a future series, probably.

Final architecture

For posterity, the picture of where we ended up:

Everything in this picture has been earned by an outage, a postmortem, or a near-miss. None of it is decoration. If you're building something similar and one of the boxes seems extra to you, it's because you haven't had the incident that justifies it yet.

Closing

Centralizing auth at the edge is one of those decisions that looks obviously correct in hindsight and is genuinely hard to convince a team to invest in beforehand. The wins are diffuse — slightly less drift, slightly fewer CVEs, slightly faster security responses. The pain is concentrated and visible — one new service to operate, one extra hop, one more place that has to be HA.

But every six months we look back and the gateway has paid for itself again. A library upgrade we did once instead of thirty times. A revocation feature that shipped in a week instead of being negotiated across teams. A multi-tenant isolation guarantee we can actually defend in audits.

If you take one thing from this series, take this: auth is not a problem you solve once and ignore. It's a problem you solve somewhere, well, and operate with care. Pick that somewhere to be the edge, build it small and observable, and the rest of your platform gets to focus on actual product work.

Thanks for reading. If you build one of these — or are stuck somewhere mid-build — drop a comment. The hardest part of operating an Auth Gateway is realizing that other people have built the same thing and hit the same rocks. There's no reason for each team to find them independently.

Part 9 — Operating the gateway: logs, traces, health, and degraded mode

Akarshan Gandotra — Mon, 04 May 2026 18:42:27 +0000

The first eight chapters of this series have been about building an Auth Gateway. This one is about living with one.

A gateway in front of every authenticated request is a force multiplier — for both your platform and any oncall page. If something is broken, it's broken everywhere at once. So observability isn't a Chapter 9 thing. It's a Chapter 0 thing. We just describe it last because there's enough mechanism to talk about that you need the rest of the series first.

This chapter covers the four things you need to be able to do at 3 AM:

Read a single log line and understand what happened.
Trace a slow request from edge to upstream.
Tell whether a pod is alive, ready, or in deep trouble.
Get an alert once — not once per pod per second — when something degrades.

The log line

There are exactly two structured log lines per protected request: one from NGINX, one from the Auth Service. They share request_id, so you can join them.

The NGINX line

Every request is logged as JSON to stdout via NGINX's log_format main:

log_format main escape=json '{
  "logType":"NGINX_LOGS",
  "request_id":"$request_id",
  "time_local":"$time_iso8601",
  "remote_addr":"$remote_addr",
  "request_method":"$request_method",
  "request_uri":"$request_uri",
  "request_path":"$uri",
  "slug":"$location_path",
  "product":"$product",
  "microservice":"$microservice",
  "status":"$status",
  "status_class":"$status_class",
  "request_time_ms":"$request_time_millis",
  "service_request_time_ms":"$upstream_response_time_millis",
  "service_connect_time":"$upstream_connect_time",
  "auth_request_time_ms":"$auth_time_millis",
  "auth_connect_time":"$auth_connect_time",
  "body_bytes_sent":"$body_bytes_sent",
  "http_referer":"$http_referer",
  "http_user_agent":"$http_user_agent",
  "http_x_forwarded_for":"$http_x_forwarded_for",
  "http_host":"$http_host",
  "tenant_id":"$tenant_id",
  "tenant_namespace":"$tenant_namespace",
  "identity_id":"$identity_id",
  "identity_type":"$identity_type",
  "auth_error_message":"$auth_error_message",
  "auth_error_code":"$auth_error_code"
}';

A few fields that matter more than they look:

auth_request_time_ms — how long the auth subrequest took. We graph this. We page on the p99 going above 50 ms.
service_request_time_ms — how long the upstream took, excluding the auth subrequest. Sequential, not overlapping (Chapter 2).
status_class — 1xx/2xx/3xx/4xx/5xx. Faster than parsing status for dashboard breakdowns.
tenant_id — the resolved tenant. Always grep by tenant first.
auth_error_code + auth_error_message — populated on deny. Empty on allow.

Two extra log formats for the fail paths

When NGINX hits @auth_unavailable or @upstream_unavailable, we log to a different format:

log_format auth_unavailable     '...auth-specific fields...';
log_format upstream_unavailable '...upstream-specific fields...';

location @auth_unavailable {
  internal;
  access_log /dev/stdout auth_unavailable;
  return 503 ...;
}

The reason: when something is on fire, you want it isolated in its own log stream. Dashboards built off main get drowned by 200s; a dashboard against auth_unavailable shows you exactly the broken bucket without filtering.

The Auth Service line

The Auth Service emits exactly one AUTH_DECISION log per request:

{
  "level":"info",
  "ts":"2026-05-01T12:34:56.789Z",
  "logger":"AUTH_DECISION",
  "request_id":"550e8400-e29b-41d4-a716-446655440000",
  "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id":"00f067aa0ba902b7",
  "uri":"/api/v1/users",
  "method":"GET",
  "slug":"user-service",
  "tenant_id":"mt_prod",
  "endpoint_type":"ACCESS_CONTROLLED",
  "identity_type":"USER",
  "identity_id":"user@example.com",
  "auth_method":"bearer_token",
  "decision_reason":"ACCESS_LEVEL_MATCH",
  "outcome":"allow",
  "status":200,
  "duration_ms":2.5,
  "authn_ms":1.8,
  "authz_ms":0.7,
  "jwt_cache_hit":true,
  "bitmap_authz_used":true,
  "granted_access_levels":["product:admin"],
  "token_revoked":false,
  "slow":false
}

This is the single most important artifact in the whole gateway. It is, in dry terms, our auth audit log. In practical terms it's the thing oncall greps when anything goes weird.

Design rules we apply to it religiously:

Exactly one line per request. No "starting auth", "authenticated", "authorizing", "decided" — those make the storyline split across N entries that you have to stitch back together. One line, one decision, every field.
Every field, every time. If a field doesn't apply (e.g., bitmap_authz_used for an OPEN endpoint), it's false or empty, not omitted. Optional fields make ad-hoc queries painful.
decision_reason is enum-only. Free-form strings here would be the death of dashboards. New reasons require code review (Chapter 3).
Trace IDs are present. trace_id and span_id are pulled from the OpenTelemetry context, so the log line stitches to traces in our backend without join logic.

Joining NGINX and Auth Service

request_id is generated by NGINX ($request_id) and forwarded to the Auth Service via the subrequest header X-Request-ID. Both log lines carry it. A typical investigation:

1. user reports 401 at 12:34:56 UTC
2. grep tenant_id="mt_prod" identity_id="..." in NGINX logs around the time
3. capture request_id
4. grep that request_id in Auth Service logs
5. read decision_reason — full story

The whole graph of "client → ingress → NGINX → Auth Service → upstream" stitches back together by request_id.

Tracing

OpenTelemetry instrumentation runs across both NGINX (via the otel module if compiled in; we're tracking that as a future improvement) and the Auth Service. The Auth Service span looks like:

SPAN: pth-auth-service POST /auth
├── ATTRIBUTES
│   ├── http.route = "/auth"
│   ├── auth.tenant_id = "mt_prod"
│   ├── auth.endpoint_type = "ACCESS_CONTROLLED"
│   ├── auth.outcome = "allow"
│   └── auth.decision_reason = "ACCESS_LEVEL_MATCH"
├── EVENTS
│   ├── jwt.cache.hit
│   ├── route.cache.hit
│   └── bitmap.match
└── DURATION 2.5ms

The trace context propagates to upstream services via standard W3C trace headers. So a single trace shows: client → ingress → NGINX (eventually, via otel module) → Auth Service span → upstream service span(s) → DB calls inside the upstream. The whole story.

We don't turn on full sampling. 1% sampling at edges, 100% sampling for spans tagged auth.outcome="deny". The deny path is where the interesting investigations happen; sampling it fully gives us forensic detail without exploding storage.

Health probes

Three K8s probe endpoints, each with a different purpose.

`/livez` — process is alive

func Liveness(c *gin.Context) {
    c.JSON(http.StatusOK, gin.H{"status": "ok"})
}

Returns 200, always. The contract: as long as this handler runs, the process isn't deadlocked. K8s only kills the pod if the request times out (handler doesn't run at all).

What /livez does not check:

It does not check the trie.
It does not check Redis.
It does not check Postgres.

This is intentional. A pod whose Redis connection died can still serve cache-hot requests correctly. Killing it on a Redis outage is exactly the wrong thing to do — you turn a cache-hit-100%-but-Redis-down state into a cluster-wide rolling restart.

`/readyz` — pod can take traffic

func Readiness(c *gin.Context) {
    if !trieLoaded() {
        c.JSON(503, gin.H{"status": "trie not loaded"})
        return
    }
    if revocationExpected() && !revocationServiceReady() {
        c.JSON(503, gin.H{"status": "revocation cache not ready"})
        return
    }
    c.JSON(200, gin.H{"status": "ok"})
}

Two gates:

Trie loaded. Without it, we can't classify any endpoint. The pod is useless until the trie is in memory.
Revocation cache ready (if revocation is enabled). Without it, fail-closed designs would deny everything; fail-open designs would miss every revocation. Either way, not ready.

Notably not gated on Redis health: a pod that lost Redis after readiness flips green stays ready. Refreshes fail loudly via Slack, but live traffic isn't disrupted.

`/healthz` — deep health

func Health(c *gin.Context) {
    pgErr := pingPostgres(c.Request.Context())
    rdErr := pingRedis(c.Request.Context())
    if pgErr != nil || rdErr != nil {
        c.JSON(503, gin.H{
            "status": "degraded",
            "postgres": errString(pgErr),
            "redis":    errString(rdErr),
        })
        return
    }
    c.JSON(200, gin.H{"status": "ok"})
}

This one does depend on Redis and Postgres. It's not used by Kubernetes — it's used by external monitoring (Pingdom, status pages). The distinction matters:

K8s probes determine traffic routing. They should be tolerant — every false-positive failure pulls a pod out of service.
Monitoring probes determine alerting. They should be strict — they tell humans something is wrong, not the load balancer.

A common mistake is wiring /healthz to readinessProbe. Don't. You will pull pods out of the rotation on a transient Redis blip and convert a degraded state into an outage.

stateDiagram-v2
    [*] --> Booting
    Booting --> NotReady: trie empty
    NotReady --> Ready: trie loaded AND<br/>(revocation disabled OR revocation cache warm)
    Ready --> Degraded: Redis stream XREAD failure
    Degraded --> Ready: reconnect
    Ready --> Live: /livez always 200
    Ready --> DeepCheck: /healthz pings PG + Redis

Degraded mode

The gateway is built to tolerate three specific kinds of trouble:

Redis is slow or down

Revocation stream consumer fails. streamDegraded atomic flips true. Slack alert fires once. Hot path keeps working — local cache is in memory.
Pub/Sub subscriber reconnects in the background. On reconnect, the subscriber re-subscribes. The periodic cleanup goroutine resyncs the ZSET when it next runs.
SA version sync fails. Local SA version map is unchanged. Tokens continue to validate against the last known versions until the next successful sync. Slack alert fires.

Postgres is slow or down

Trie reload fails. Existing trie remains in memory. Slack alert. Hot path is unaffected.
New pods cannot start (initial trie load blocks readiness). Existing pods serve.
This is a partial outage: scaling up is broken, current capacity still works.

Slack is slow or down

Alerts are fire-and-forget goroutines. We don't block the hot path on Slack.
If Slack is down, alerts are queued in goroutines for a configured timeout (5s) then dropped. We don't retry forever and OOM the pod.

The alert tree

flowchart TD
    Boot[startup] --> A1{revocation Redis OK?}
    A1 -->|no| Alert1[Slack: TokenRevocationService Redis client not initialized<br/>readyz=503]
    Run[runtime] --> A2{XREAD error?}
    A2 -->|yes| Alert2[Slack: stream degraded once]
    A2 -->|recover| Alert3[Slack: stream recovered]
    Run --> A3{localCache size > MAX?}
    A3 -->|yes| Alert4[Slack: cache overflow once<br/>fall back to ZSCORE]
    Run --> A4{RSA key missing for tenant?}
    A4 -->|yes| Alert5[Slack: per-tenant dedup]

The Slack-alerter pattern

Three rules we apply rigorously to alert code, because we learned them the hard way:

Rule 1: Atomic-bool dedup per state transition

type degradeFlag struct {
    on atomic.Bool
}

func (d *degradeFlag) MarkDegraded(reason string) {
    if !d.on.Swap(true) {
        slack.Alert("auth.degraded", reason)
    }
}

func (d *degradeFlag) MarkRecovered() {
    if d.on.Swap(false) {
        slack.Alert("auth.recovered", "")
    }
}

The Swap returns the previous value. If it was already true, we don't re-alert. We alert once on the transition, and once on the recovery.

Without this, a Redis outage produces thousands of Slack messages per pod per minute. The first time it happened, oncall threw their phone across the room.

Rule 2: Per-(pod × cause) dedup, not per-cause

A 100-pod deployment hitting the same RSA key misconfiguration alerts 100 times. That's correct: each pod is a separate runtime, each could have its own state, each is a separate alert source. We tag every alert with the pod's hostname so you can see in Slack whether it's a single-pod or fleet-wide problem.

Rule 3: Deployment-tag prefix

Every alert is prefixed with [customer-env]:

[acme-prod] auth.degraded TokenRevocationService Redis stream degraded

This makes a single Slack channel viable for many environments. Without the prefix, you can't tell at a glance whether the alert is from staging or prod.

What we don't alert on

Individual auth failures. Login throttling, expired tokens, denied requests — those are normal and high-volume. They're in dashboards, not Slack.
High latency on a single request. Latency alerts go on rolling p99, not on individual outliers.
Anything below "the gateway behaves correctly but degraded." If the system self-heals quietly, we don't page humans.

NGINX-specific operations

A few NGINX-isms worth calling out.

Graceful shutdown

The chart's deployment does this on preStop:

lifecycle:
  preStop:
    exec:
      command:
        - /bin/sh
        - -c
        - |
          echo "[preStop] draining 15 seconds..."
          sleep 15
          echo "[preStop] nginx -s quit..."
          nginx -s quit
          while pgrep -x nginx > /dev/null; do sleep 1; done

15 seconds of drain, then nginx -s quit (graceful — drain in-flight requests, then exit), then wait for all worker processes to finish. Combined with terminationGracePeriodSeconds: 60, we have ~60 seconds total budget for clean shutdown. Without this, rolling deploys produced visible spikes of 502s in client logs.

Worker tuning

worker_processes  auto;
worker_rlimit_nofile 65535;
worker_shutdown_timeout 30s;

events {
  worker_connections 10240;
  use epoll;
  multi_accept on;
  accept_mutex off;
}

worker_processes auto scales to CPU count. accept_mutex off is the modern default — let kernel epoll handle accept distribution. worker_connections 10240 is per worker, so a 4-core pod handles ~40k concurrent connections.

Upstream keepalive

upstream auth_service {
  server auth-service-golang.<ns>.svc.cluster.local:80
         max_fails=3 fail_timeout=30s;
  keepalive 64;
  keepalive_requests 1000;
  keepalive_timeout 60s;
}

keepalive 64 is per worker. A 4-worker NGINX with 64 keepalive maintains 256 keepalive connections to the Auth Service. Without keepalive, every subrequest opens a fresh TCP connection — fatal at any real RPS. The first time we deployed without it, p99 auth time was 80 ms. With it: 3 ms.

max_fails=3 fail_timeout=30s marks the upstream "down" after 3 consecutive failures and stops sending traffic for 30 seconds. Combined with the retry config in auth.conf, this gives smooth failover when one auth pod is sick.

A picture of where the time goes

gantt
    title Single request as seen by NGINX
    dateFormat  X
    axisFormat %s ms
    section NGINX
    receive + route match : 0, 1
    auth subrequest       : 1, 5
    proxy upstream        : 6, 18
    log JSON              : 24, 1

The auth subrequest is the smallest thing on the timeline. That's not by accident. Chapter 8's caches are the reason it stays small. Chapter 9 is what tells you when it stops staying small.

What's next

Chapter 10 is the retrospective: what we'd build differently if we did it from scratch, what tools we wish we'd added on day one, and the maturity progression from "auth library in every service" to "production-grade gateway." It's the chapter you write after operating the thing for two years, and it's the one I'd want to read if I were starting from scratch.

Part 8 — Making It Fast: Caching, Hot Paths, and Avoiding DB Calls

Akarshan Gandotra — Mon, 04 May 2026 18:42:14 +0000

The Auth Gateway sits in front of every authenticated request in the platform. Its latency isn't just its own latency — it's the floor for every service behind it. If auth takes 50ms, every request to every upstream service starts 50ms in the hole.

Our internal target is sub-millisecond on cache-hot paths. The way we hit it isn't clever algorithms — it's a stack of small caches, each one handling a different kind of state, each invalidated through a different channel. This post walks through all of them.

The principle that shapes everything

Before the individual layers: a rule we hold as policy.

Redis is allowed to influence the hot path. Redis is not allowed to block it.

Every cache in the system is in-process. Redis feeds them asynchronously — pushing revocation events, triggering trie reloads, syncing SA versions. But a pod whose Redis connection is dead can still answer requests correctly, for the duration of its staleness window.

That's the difference between "Redis is down, the platform is down" and "Redis is down, the platform is slightly stale." One is a severity-1 incident. The other is a degraded mode we can tolerate for minutes while someone fixes it.

With that framing, here's how a warm request flows through the cache stack:

Six layers. Five are pure in-process memory. The sixth — revocation — is in-process too, but fed asynchronously from Redis. No layer blocks on a network call.

Layer 1: JWT verify cache

The single biggest win in the stack. RSA signature verification is expensive — a few hundred microseconds per call — and at 50,000 RPS that cost is real.

We wrap the entire decode-and-verify path in a Ristretto cache. The key is a 64-bit FNV hash of the raw token string; the value is the decoded JWT claims. On a cache hit, we skip RSA verification entirely.

A few choices worth explaining:

Why Ristretto over a plain LRU. Ristretto uses TinyLFU — it tracks access frequency and uses it to decide what to evict. Under burst traffic, a pure LRU can evict frequently-used tokens just because they weren't the most recent. TinyLFU keeps the hot tokens and evicts the cold ones. The behavior under load is meaningfully better.

Why hash the token string. Two reasons. Memory: a JWT is 500–2000 bytes; a uint64 is 8. And defense-in-depth: if the cache state ever ends up in a log or heap dump, the tokens themselves aren't exposed.

Why cap TTL at 30 seconds. The cache stores the decoded token, not the auth decision. Revocation is checked separately on every request. But capping TTL at 30 seconds keeps the staleness window honest — a token that's been revoked won't ride a warm cache entry for an hour.

Layer 2: RSA public key cache

Per-tenant RSA public keys are loaded from environment config at boot. Parsing PEM is not free — a few hundred microseconds — and we don't want to pay it on every cache miss.

We cache the parsed key per tenant using sync.Once. The first request for a given tenant parses the key; every request after that gets the cached result, including if the first parse failed.

Two operational details that matter:

A misconfiguration fires a Slack alert once per tenant per pod, not once per request. Without this guard, a single bad key config generates a Slack message for every request that hits that tenant, which during a deploy is thousands of messages in seconds.

Key rotation requires a pod restart. We considered hot-reloading. We chose deploy-to-rotate — the operational simplicity of a predictable restart beats the complexity of a file watcher and the failure modes it introduces.

Layer 3: route cache

The trie lookup is already fast — O(depth), with depth typically 3–5 segments. But re-walking the same paths 50,000 times a second is wasteful. A TinyLFU cache sits in front of the trie, keyed by slug, HTTP method, and path.

The platform has around 3,000 distinct route tuples in production. Sized at 10,000 entries, the cache fits the entire steady-state working set with room to spare. Misses are new endpoints, cold starts, and post-reload warm-up.

Invalidation is bulk. On any trie reload — whether triggered by a periodic interval or a Redis Pub/Sub kick — we drop the entire route cache. We considered partial invalidation (only drop entries for changed slugs) and rejected it. Trie reloads are rare. The cache refills in milliseconds. The bookkeeping complexity of partial invalidation isn't worth the seconds of warm-up time it would save.

Layer 4: the trie

The trie is a cache too, just an unusual one. It's an in-memory mirror of the endpoint table from Postgres. No request ever touches Postgres on the hot path.

Invalidation has two channels:

Periodic: every hour by default. A safety net.
Push: via Redis Pub/Sub on auth:trie:refresh. Admin tooling publishes this after any write to the endpoint table. Pods reload within milliseconds.

The push channel exists because endpoint changes are operationally significant. A new admin route that's meant to be protected shouldn't have a one-hour window where it's open because the trie hasn't refreshed. The push channel closes that window.

Layer 5: policy bitmap snapshot

The permission bitmap (covered in the previous chapter) is loaded alongside the trie. It's an in-memory structure mapping permission names to bit indexes, with a version number.

The snapshot is never partially updated. It's swapped atomically — a background process builds a new snapshot when the registry changes, then stores it via an atomic pointer swap. Readers grab the pointer at the start of a request and work with that exact snapshot throughout. No locks, no torn reads.

This pattern shows up repeatedly in the codebase: when state changes as a whole unit, an atomic pointer is simpler and faster than a read-write mutex around a map. It's worth internalizing.

Layer 6: revocation map and SA version map

These were covered in depth in the previous chapter. In the context of the cache stack:

The revocation map is bounded at 50,000 JTIs, fed by a Redis Stream, and fails open — if a JTI isn't in the map, we treat it as not revoked. The staleness window is low single-digit seconds in steady state.

The SA version map has the opposite posture: fail closed. If the map isn't ready, the pod doesn't pass readiness. If a service account token's version is behind the current version in the map, it's denied.

Same underlying shape — in-memory map fed asynchronously from Redis — but different risk tolerance based on what's being protected.

How all the invalidation channels fit together

Three patterns across the stack:

TTL-based (JWT verify cache). Simple, no coordination. Best when the cached value has a natural expiry built into it — which JWTs do.

Push-based (trie, revocation stream, SA version). Required when a staleness window has real cost. Needs a degraded-mode plan for when the push channel is unavailable.

Capacity-based eviction (route cache, JWT cache). Bounded memory by design. What gets evicted matters more than when — which is why TinyLFU beats LRU for this workload.

When in doubt, start with TTL. Push-based caches are powerful but bring failure modes — lost events, stalled consumers, cursor races. Use them only when a TTL window is genuinely unacceptable.

The cache we got wrong

Our first JWT cache used a plain Go map with a mutex and a time.AfterFunc per entry to handle expiry.

It worked in tests. It fell over in production within a week. Two problems:

Goroutine pressure. Every cached token spawned a timer goroutine. At a million live tokens, the Go scheduler handled it — but GC pauses got ugly and unpredictable.

No cap. There was no size limit. Memory grew until pods OOM-killed.

Switching to Ristretto solved both: timers are amortized into a small internal worker, and MaxCost enforces a hard ceiling.

The lesson: a cache is a copy of state. If there's no mechanism to bound or invalidate it — TTL, push, or capacity — it's not a cache. It's a memory leak.

Cold start vs. warm

A pod's first requests are slower. The trie loads from Postgres before readiness flips — that's the only DB call in the pod's lifetime. After that, every lookup is in-memory.

The JWT cache starts empty on a fresh deploy and fills up within seconds as real tokens come through. We don't pre-warm it — the cost of cold RSA verifications for a few seconds after a deploy is acceptable.

The revocation cache we do pre-warm, synchronously, before readiness. A pod that's marked ready must have the current revocation set. Otherwise it would fail-open on every request until its first Redis sync — meaning any logouts from the past hour would be invisible to it.

What to actually graph

For each cache, the metrics that matter:

Hit rate — the most important number. A cache with a stable size but falling hit rate is broken.
Eviction rate — meaningful only if the cache is bounded. High eviction with high hit rate is fine; it means the cache is doing its job under pressure.
Size — useful for capacity planning, not for alerting.

The JWT verify cache runs at 95%+ hit rate in steady state. A fresh deploy drops it to zero and it climbs back within seconds. Anything else warrants investigation.

Don't alert on cache size. Alert on hit rate.

Next up: Chapter 9 — operating the gateway. The structured auth decision log, OpenTelemetry tracing, the three Kubernetes probes, degraded-mode behavior, and the Slack alert pattern that keeps on-call sane during a Redis outage.

Part 7 — Token Revocation Without Killing Performance

Akarshan Gandotra — Mon, 04 May 2026 18:41:57 +0000

JWTs have a hard problem hiding inside them: they're stateless. The whole point of a JWT is that the verifier can check a signature and make a decision — no database, no round-trip. That's what makes them fast. It's also what makes "log this user out right now" not work out of the box.

We had to solve this. Users log out. Admins disable accounts. Service accounts rotate. Each one of those events has to invalidate live tokens immediately, not at the next expiry tick.

This post is about how we did it without giving up the performance properties that made JWTs worth using in the first place.

The constraints that ruled out the obvious answers

Three numbers shape the design:

50,000 RPS of authenticated requests.
Sub-millisecond auth budget on the hot path.
Single-digit-second propagation — when a user logs out, every pod must know within a few seconds.

The obvious approaches each break one of these:

Query Redis on every request. Adds a network round-trip to every auth decision. Median latency explodes. Redis also becomes a hard single point of failure — if it's slow or down, every request fails.

Push revocation events via websockets or long-poll to every pod. Works at low scale. Gets fragile when pods churn, restart, or drop events during a network blip.

Short-lived tokens with fast refresh. A 5-minute expiry reduces the window, but doesn't close it — and 5 minutes is too long when an account is disabled for a security reason.

What worked: a two-layer design.

Redis is the propagation layer. It holds the authoritative revocation state and a live event feed.
Local memory is the decision layer. Each pod keeps an in-memory map of revoked JTIs. The hot-path check is a single map lookup — no I/O.

Two Redis structures, one job each

Two Redis keys do the heavy lifting, and they serve different purposes — which is why both are necessary.

revoked_access_tokens is a sorted set. Each member is a JTI; the score is the token's expiry timestamp. This is the source of truth at any point in time — you can ask it "give me everything currently revoked" with a single range query.

revoked_access_token_events is a stream. Each entry carries the JTI, expiry, and metadata about the revocation. This is the live feed — pods subscribe to it and learn about new revocations as they happen.

The ZSET answers "what is the state right now?" The Stream answers "what has changed since I last checked?" You need both because they're good at different things: the ZSET is for bulk reads at startup, the Stream is for incremental updates during steady state.

The startup problem — and two races hiding in the obvious solution

When a pod boots, it needs to populate its local map before it serves traffic. The tempting approach: read the ZSET to get current revocations, then subscribe to the Stream for updates.

Two races hide here.

Race 1: What if a revocation arrives between the ZSET read and the Stream subscription? The event is in the Stream, but the pod's cursor is positioned after it. The JTI never makes it into the local cache.

Race 2: What if you start the Stream consumer from the very beginning (0-0) to avoid missing anything? Now you replay every event ever emitted — potentially thousands. Worse: if the stream has been trimmed, you'll silently miss events older than the trim window.

The fix is to reverse the order: capture the Stream tip before reading the ZSET, then start the consumer from that captured tip.

func (s *TokenRevocationService) WarmCache(ctx context.Context) error {
    // 1. Capture the stream tip first.
    tipID, err := s.captureStreamTip(ctx)
    if err != nil { return err }

    // 2. Read the current ZSET snapshot.
    now := time.Now().Unix()
    members, err := s.redis.ZRangeByScoreWithScores(ctx,
        "revoked_access_tokens",
        &redis.ZRangeBy{Min: strconv.FormatInt(now, 10), Max: "+inf"},
    ).Result()
    if err != nil { return err }

    s.mu.Lock()
    for _, m := range members {
        s.localCache[m.Member.(string)] = int64(m.Score)
    }
    s.mu.Unlock()

    // 3. Start the consumer at the tip captured before the ZSET read.
    // Anything that arrived between tipID and now replays through the consumer.
    go s.consumeStream(tipID)
    return nil
}

The order matters precisely. By capturing the tip first, anything that arrives while we're reading the ZSET will replay through the consumer. Anything already in the ZSET when we read it is loaded directly. If the same JTI appears in both — a revocation that landed right on the boundary — setting the same map entry twice is harmless.

The pod's lifecycle from boot to steady state:

The hot path: deliberately boring

The actual check on every request is about as simple as it gets:

func (s *TokenRevocationService) IsJTIRevoked(jti string) bool {
    s.mu.RLock()
    expiresAt, found := s.localCache[jti]
    s.mu.RUnlock()
    if !found { return false }
    if time.Now().Unix() > expiresAt { return false }
    return true
}

A read lock, a map lookup, a comparison. No Redis, no network. Hundreds of nanoseconds.

The !found → false branch is a deliberate fail-open choice: if a JTI isn't in the local cache, we treat it as not revoked. The risk is that a freshly revoked token might be accepted for the few seconds between the revocation being published and the local cache being updated. We accept that window. The alternative — failing closed — would mean denying every request whose JTI we haven't explicitly loaded, which at startup means denying all traffic until the cache is fully warm. That's worse.

The gap probe: catching what the Stream misses

The Stream consumer keeps a cursor — the ID of the last event it processed. Periodically, the stream gets trimmed to bound its size. If the consumer's cursor falls behind the trim window (because of a slow handler, a GC pause, or a network blip), the next XREAD will silently skip the trimmed events.

We detect this with a gap probe that runs every 5 minutes:

If the oldest event currently in the Stream is newer than the consumer's cursor, we missed something. When that happens, we resync from the ZSET (which is the authoritative source of truth and doesn't get trimmed the same way) and snap the cursor to the stream tip.

This probe has fired exactly twice in production since we added it — both times during planned Redis maintenance — and both times the recovery was automatic. The value isn't that it fires often. It's that without it, you'd never know you missed events at all.

Service accounts: the same idea, different risk tolerance

User token revocation fails open — a freshly revoked token might slip through for a few seconds. That's acceptable: the window is small, bounded, and observable.

Service-account rotation fails closed. When a service account is rotated, the old credentials must be denied immediately, even if that means a slightly degraded startup path.

The mechanism is different too: instead of JTI revocation, service accounts carry a version number. The gateway keeps a local map of current SA versions loaded from Redis. If the token's version is less than the current version for that service account, it's denied.

The pod won't pass readiness until this cache is loaded. If Redis is unavailable at startup, the pod doesn't serve traffic. That's intentional — we'd rather have fewer pods than pods that can't correctly enforce SA rotation.

The 60-second sync window is our exposure. We reduce the effective risk by having the rotating system hold the old version live for a grace period, only promoting the new version once enough gateways have synced.

What revocation doesn't do

A few things that seem natural but aren't in scope:

Revoke by user ID. The cache is JTI-indexed. To revoke all of a user's tokens, the issuer enumerates their live JTIs and revokes each one. The Auth Service sees only individual JTIs.

Cross-region propagation. We run regional auth services with regional Redis instances. Revocations published in one region don't automatically appear in another. Most revocations are tenant-bound, and tenants are region-bound, so this rarely matters in practice.

Shared Redis. This Redis instance is auth-only. The corner cases in revocation are complex enough that sharing infrastructure with rate limiters or session stores would make debugging much harder.

What we'd do differently on day one

Add the gap probe immediately. It's a small amount of code and it's the difference between "we silently lose a logout event occasionally" and "we always know when propagation breaks."

Test the warm path with a slow or unavailable Redis. Most bugs we found were in error handling during startup, not steady-state operation. The warm path runs once per pod lifetime; staging rarely exercises it unless you deliberately inject failures.

Bound everything from the start. The local cache, the stream length, the sync interval. Unbounded growth in any of them becomes an incident.

Next up: Chapter 8 — every cache in the hot path, together. JWT verify cache, RSA key cache, route cache, policy bitmap, revocation map, SA version map. Each one is fast individually; together they're how the gateway fits inside its latency budget. We'll cover TTL strategy, invalidation, and the one cache where we got eviction wrong.

Part 6 — Authorization at Scale: Access Levels, Roles, and Compact Decisions

Akarshan Gandotra — Mon, 04 May 2026 18:41:42 +0000

Authentication answers "who are you?" Authorization answers the harder question: "are you allowed to do this?"

By the time a request reaches this stage, we've already validated the token and confirmed the tenant. Now we need to decide — before the request touches any upstream service — whether this specific identity has permission to call this specific endpoint. That decision runs hundreds of millions of times a day. It needs to be fast, correct, and cheap to reason about when something goes wrong.

This post is about the model we use, the simpler approach that served us for a year, and the optimization we eventually built — and why we kept the old path around anyway.

The model: three layers, one question

Our authorization model has three layers:

A role is what a user is granted — something like clinic_admin or billing_specialist. An access level is a coarse permission — something like user:admin or schedule:write. An endpoint declares which access levels are sufficient to reach it.

Roles are bags of access levels. Endpoints are protected by lists of access levels. A user can call an endpoint if any of their roles' access levels appear in the endpoint's required list. That's the whole model.

We deliberately keep it coarse. There are dozens of access levels in the system, not thousands. Questions like "can this user delete this specific patient record?" belong to the upstream service that owns that data — it has the context the gateway doesn't. The gateway's job is to filter on the order of "is this user even an admin at all?" — a check that catches the vast majority of misuse and runs at edge speeds.

Who defines access levels and endpoints?

Here's something that might seem surprising: the gateway doesn't own the access level definitions. Individual product services do.

Each service ships a access_levels.json alongside its code. This file declares what access levels it recognizes and which endpoints require which levels. A scheduling service owns schedule:write. A billing service owns billing:read. The gateway is a consumer — it doesn't make editorial decisions about what permissions mean.

// access_levels.json — owned and maintained by the upstream service
{
  "access_levels": [
    { "name": "schedule:write", "description": "Create and modify appointments" },
    { "name": "schedule:read",  "description": "View appointments" }
  ],
  "endpoints": [
    { "path": "/api/appointments",  "method": "POST", "requires": ["schedule:write"] },
    { "path": "/api/appointments/*", "method": "GET",  "requires": ["schedule:read", "schedule:write"] }
  ]
}

The publish flow runs through CI/CD. When a service merges a change to its access level definitions, a pipeline step pushes the updated file to a well-known S3 path. The gateway picks up the change on its next refresh cycle — no gateway deploy required, no manual registry edits.

service-repo/
  access_levels.json   ← owned by the service team
  .github/workflows/publish.yml

# publish.yml (simplified)
- name: Publish access levels
  run: |
    aws s3 cp access_levels.json \
      s3://registry/services/${{ env.SERVICE_NAME }}/access_levels.json

This keeps ownership aligned: the team that builds the feature decides what permission protects it. The gateway team owns the mechanism; product teams own the policy. Changes are auditable through git history, reviewable via pull request, and rollbackable the same way code is.

What the token carries — and what it doesn't

The JWT includes the user's granted access levels as a bitmap — a compact byte slice — along with the version of the registry used when the token was issued. It does not contain the full permission graph, and it does not contain endpoint requirements. Those live in the database, loaded into memory at boot.

A decoded JWT payload looks like this:

{
  "sub": "user_abc123",
  "tenant": "acme-health",
  "policy_bitmap": "__________________8H",
  "policy_bitmap_version": 114,
  "exp": 1714000000
}

policy_bitmap is a base64url-encoded byte slice — each bit position corresponds to one access level in the registry at version 114. policy_bitmap_version tells the gateway exactly which registry snapshot to use when interpreting the bits. If the gateway's current registry is at version 114, it uses the fast bitmap path. If the versions differ, it falls back to string matching (more on that below).

This is a deliberate tradeoff. The stateless alternative — put everything in the token, make every decision without a database — sounds clean until users accumulate permissions. Tokens balloon to 4–8 KB. Cookies start failing at network edges. Mobile clients cache tokens aggressively and get stuck with stale permission sets. Every role change requires re-issuing every affected token immediately.

The compromise: the JWT carries coarse access levels (a small, stable set encoded as a bitmap), and the database carries endpoint requirements (queried once at startup, refreshed on demand). Per-request authorization is a fast in-memory lookup on both sides.

The payoff on token size is significant. Before the bitmap rework, heavily-permissioned admins had tokens approaching 3 KB. After:

The original approach: string matching

The first implementation is what you'd sketch on a whiteboard. Take the user's access levels, take the endpoint's required access levels, check if they overlap.

It's O(n + m) — linear in the number of user permissions and required permissions. With typical values (a user might have 20–80 access levels, an endpoint usually requires 1–3), this runs in nanoseconds. It's correct, it's readable, and it worked fine in production for over a year.

The reason we eventually replaced it had nothing to do with speed.

The first reason was token size. As the platform grew and senior users accumulated more access levels, tokens stretched. We had admins with tokens approaching 3 KB. That's uncomfortable but manageable — until it isn't.

The second reason was density of signal. String matching tells you that the user was authorized, but the log entry just says granted: ["user:admin"]. We wanted richer per-permission metrics — which access levels are actually being exercised, which ones are granted but never hit anything — without adding another pass over the data.

The bitmap approach: compress the representation, keep the logic

The idea is simple: assign every access level a stable integer index. Represent a user's granted permissions as a bit vector — one bit per access level. Represent each endpoint's requirements the same way. Authorization becomes a bitwise AND.

If the result is nonzero, the user has at least one of the required permissions. Allow. If the result is zero, deny. That's the entire hot path.

The anchoring snippet — the intersection check at the core of it:

func intersects(a, b []byte) bool {
    n := len(a)
    if len(b) < n {
        n = len(b)
    }
    for i := 0; i < n; i++ {
        if a[i]&b[i] != 0 {
            return true
        }
    }
    return false
}

For a typical 32-byte bitmap (covering 256 possible access levels), this is a handful of CPU instructions. Decision time dropped from ~3 microseconds in the worst legacy case to under 200 nanoseconds. Not visible to end users. Very visible in CPU costs at 50,000 requests per second.

Token size dropped too — from ~3 KB for a heavily-permissioned admin to under 1 KB. The access levels that used to be a long string array became the policy_bitmap field: a base64url-encoded byte slice.

The two paths side by side:

The version problem — and why we kept the old path

Here's the catch: bit indexes have to be stable. If access level user:admin is bit 0 today, it must still be bit 0 when old tokens are being validated. This is managed through a versioned registry — each snapshot of the bit assignments carries a version number, and the JWT records which version was used when it was issued via policy_bitmap_version.

{
  "policy_bitmap": "__________________8H",
  "policy_bitmap_version": 114
}

When the gateway boots, it loads the current registry — say, version 114 — and builds an in-memory lookup from version number to bit-index map. When a token arrives, the gateway reads policy_bitmap_version and checks:

Version matches current registry (114 == 114): decode the bitmap, run intersects(), done.
Version is older (e.g., 112): fall back to string matching against the access level names.
No policy_bitmap_version field: legacy token predating the bitmap feature — fall back to string matching.

The fallback uses the access level names embedded in the token (carried as a separate claim for exactly this purpose) and checks them against the endpoint's required list. Same outcome, no bitmap needed.

This fallback isn't a temporary measure. It's load-bearing. Long-lived service account tokens might be weeks old. We can't deny them just because they predate a registry update. The string-based check is version-agnostic: it doesn't care about bit indexes at all. As long as both sides agree on what the access level strings mean, it works.

New registry versions are created whenever a service publishes new access level definitions through the CI/CD pipeline. The version number increments, new bit positions are assigned to new access levels, and existing assignments are preserved verbatim. Old tokens stay valid — they just take the slightly slower path until they expire naturally.

We track the fallback rate with a metric. When it's near zero, things are healthy. A spike tells us something is wrong — maybe a token issuer is behind on registry versions, maybe a test fixture has stale data, or maybe a new service published access levels without updating the issuer to match.

A few things we deliberately didn't do

Some approaches we considered and rejected:

Caching the authorization decision. "Same token plus same endpoint equals same answer" feels right. It's wrong — role changes, revocation, and tenant changes all invalidate it. We cache the token decode result (the identity and access levels), not the decision.

Per-tenant access level definitions. Letting each tenant define what user:admin means sounds flexible. In practice, it means the registry forks and every cross-tenant reasoning breaks. Access levels are platform-wide; role assignments are per-tenant. That's the line. Individual services define access levels globally — they don't get per-tenant variants.

Hierarchical permissions. "user:admin implies user:read" is elegant on paper. It complicates bitmap encoding and makes rollback harder. We grant both explicitly. A few extra access levels per role is not a real cost.

A central registry team as the bottleneck. Early on, a single team owned all access level definitions. This created a queue — every new feature needed a registry PR to land before it could ship. Moving ownership to service teams via the CI/CD publish flow eliminated that queue entirely. The gateway team reviews the mechanism; service teams review each other's access level semantics in their own PRs.

The pattern underneath the optimization

The bitmap is a performance and density win. But the deeper idea is the same one from the last chapter: make the implicit explicit and keep the decision structure visible.

String matching and bitmap intersection both produce the same outcome — allow or deny. What the bitmap adds isn't correctness, it's compactness: a cheaper wire representation, a faster runtime check, and a version-aware fallback that degrades gracefully instead of breaking.

The CI/CD publish flow adds a different kind of compactness: it removes the coordination overhead of centralized registry management. Services declare what they need. The pipeline handles the distribution. The gateway consumes whatever's in the registry. No tickets, no handoffs.

The fallback is worth lingering on. Most optimizations in auth systems are irreversible — once you commit to a new token format, old tokens become a problem. Keeping the legacy path as a first-class citizen, with its own metrics and log fields, meant we could ship the optimization without a flag day. Old tokens kept working. New tokens got faster. The two paths converged over time on their own.

What the deployment actually looked like

Theory is one thing. Here's the Datadog dashboard from the bitmap deployment on Apr 27 at 17:30.

The real win shows up in p99 latency: it drops from a spiky 5–14 ms pattern to a stable ~4–5 ms, eliminating GC-induced variance from string allocations.

Execution time stayed in the 100–400 µs range, with a one-time spike during in-memory bitmap rebuild. Fallback usage naturally decayed as tokens rotated, validating seamless migration.

Next up: Chapter 7 — token revocation. JWTs are stateless by design, which makes "log this user out right now" genuinely hard. We solved it with a Redis-backed revocation list, an in-process cache, and two startup races we had to fix the painful way.

Part 5 — Multi-tenant auth and routing in Kubernetes

Akarshan Gandotra — Mon, 04 May 2026 18:41:26 +0000

In the first four chapters of this series I've talked about what the Auth Gateway decides. This chapter is about who it decides for.

We run a multi-tenant platform. Every request, on every endpoint, belongs to one tenant. Get tenant resolution wrong and you don't have a security incident — you have a cross-tenant data leak incident, which is a category of bad you don't recover from.

This chapter is the boring, careful, paranoid story of how NGINX and the Auth Service cooperate to never let a request through without a clear tenant identity.

The two questions

Every multi-tenant request raises two questions:

Which tenant is this for? (resolution)
Where does the request go for that tenant? (routing)

We answer #1 at the NGINX layer, before auth. We answer #2 partly at NGINX (path-based routing) and partly inside the upstream service (tenant-scoped queries). The Auth Service sits between them: it makes sure the token's tenant matches the request's tenant before either service sees the request.

Resolution: two valid inputs, one explicit failure mode

We accept two ways to identify a tenant:

X-Tenant-ID header. Explicit. Used by service-to-service calls and SDKs that know who they're for.
Host header (mapped via X-Tenant-Host). Implicit. Used by per-tenant DNS like tenant1.example.com.

We do not accept a third way: a default tenant. There is no fallback. If both inputs are missing or unknown, NGINX returns 400 before the Auth Service is even called.

Why so strict? Because a default tenant is the most expensive bug you can ship. Every "wait why is data showing up in tenant X?" post-mortem starts the same way: somebody added a fallback "for convenience" and somebody else's request hit it without a tenant header.

We removed our default tenant on day 90 and have never looked back.

Per-tenant SNI server blocks

For tenants with their own DNS, NGINX uses server blocks to short-circuit resolution. The Helm chart templates one server per tenant from global.tenants:

{{- range .Values.global.tenants }}
server {
  listen {{ $.Values.containers.containerPort }};
  server_name {{ .tenant_dns }};

  set $tenant_id        {{ .tenant_id }};
  set $tenant_namespace {{ .tenant_namespace }};

  include /etc/nginx/auth.conf;
  include /etc/nginx/locations.conf;
  include /etc/nginx/custom_error_locations.conf;
}
{{- end }}

A request to tenant1.example.com matches server_name tenant1.example.com, lands in this block, and $tenant_id is already set before any other directive runs. There is no header parsing, no map lookup, no opportunity for ambiguity. Tenant identity is pinned at SNI time.

This is also nice for TLS: the per-tenant ingress can attach per-tenant certificates if you want them, and the SNI selection happens before any HTTP processing.

The default server block: header-based fallback

Not every tenant has its own DNS. Many service-to-service calls hit a shared in-cluster ingress with X-Tenant-ID set explicitly. For those, the default server block handles resolution:

server {
  listen {{ .Values.containers.containerPort }};
  server_name _;                            # match anything not matched above

  set $tenant_id "";

  if ($http_x_tenant_id) {
    set $tenant_id $http_x_tenant_id;       # priority 1
  }

  if ($tenant_id = "") {
    set $tenant_id $tenant_id_from_host;    # priority 2: map of X-Tenant-Host
  }

  if ($tenant_id = "") {
    return 400 "Tenant not specified";      # priority 3: hard fail
  }

  include /etc/nginx/auth.conf;
  include /etc/nginx/locations.conf;
  include /etc/nginx/custom_error_locations.conf;
}

$tenant_id_from_host is a map populated from the same global.tenants list:

map $http_x_tenant_host $tenant_id_from_host {
  {{- range .Values.global.tenants }}
  "{{ .tenant_dns }}" "{{ .tenant_id }}";
  {{- end }}
}

A few subtleties worth highlighting:

The order matters. Header beats host. We picked header priority because programmatic clients should be explicit.
$tenant_id_from_host defaults to empty string if the host isn't in the map. We then 400 — same as if the header was missing entirely.
if directives in NGINX are deeply weird. We confined them to this block and resisted the temptation to put ifs anywhere else in the config.

Tenant binding inside the JWT

Once NGINX has set $tenant_id, it forwards it as X-Tenant-ID to the Auth Service. But the token also carries a tenant claim. The Auth Service must check they match:

if userToken.TenantID != log.TenantID {
    c.fail(ctx, log, 401, ReasonTenantMismatch, "tenant mismatch")
    return
}

This is the line that saves you when a malicious actor copies a token from tenant A and replays it against tenant B's hostname. The token signature is valid. The token isn't expired. The token isn't revoked. But the tenant in the token is tenantA and the request is for tenantB. We 401.

Three things make this work:

The token is bound to a tenant at issuance. Our token issuer puts tid: "tenantA" in the JWT claims when it mints the token. We sign with the per-tenant RSA key (Chapter 3), so a token from tenant A can't be re-signed for tenant B without the private key.
The gateway picks the verification key by X-Tenant-ID. If the request says it's for tenant B, we verify the token's signature with tenant B's public key. A tenant A token signed with tenant A's key fails signature validation, not tenant binding — but either way it's denied.
The tenant claim is also checked. Even if the keys were the same, the explicit userToken.TenantID != log.TenantID check would catch reuse.

Belt and suspenders. We've never regretted having both.

Sequence: tenant flow end-to-end

Routing: MT vs ST upstreams

Once we know the tenant, we have to route the request. We have two upstream models:

MT (multi-tenant) services. One deployment, serves all tenants. Tenant comes in as X-Tenant-ID, the service queries data with WHERE tenant_id = ?.
ST (single-tenant) services. One deployment per tenant, in the tenant's own Kubernetes namespace. The service doesn't even need to know about other tenants — it can't see them.

This is purely a per-service architectural choice. Some products are happy with MT; some have stricter isolation requirements (or run heavy per-tenant data) and want ST.

The location loop in locations.conf handles both with one branch:

proxy_pass http://{{ if eq $type "ST" }}{{ $serviceDict.SERVICE_HOST }}.$tenant_namespace.svc.cluster.local{{ else }}{{ $serviceDict.SERVICE_HOST }}{{ end }};

Unrolled:

MT: proxy_pass http://user-service — the bare service name. CoreDNS resolves it to the service's ClusterIP in whatever namespace the gateway lives in.
ST: proxy_pass http://api-service.tenant1-ns.svc.cluster.local — the FQDN includes the tenant namespace. Each tenant has its own copy of the service in their own namespace.

A few subtleties:

For ST, the tenant namespace is part of the DNS name. NGINX's resolver kicks in at request time, not at config-load time. Adding a new tenant means deploying its services in tenantN-ns, then adding it to global.tenants. NGINX picks it up on the next config reload.
For MT, all tenants hit the same upstream IP. The upstream service is responsible for tenant scoping. We trust it because we forward X-Tenant-ID and the upstream service's auth library re-checks the header against the token's tenant. (Yes, double-checking. After the first cross-tenant near-miss, we added it.)
ST is more expensive operationally — N deployments of every service — but radically simpler to reason about. Two services, two answers; pick what your compliance team can defend.

Headers we propagate to upstream

After auth passes, NGINX sends a defined set of headers to the upstream. From locations.conf:

proxy_set_header X-Identity-ID       $identity_id;
proxy_set_header X-Identity-Type     $identity_type;
proxy_set_header X-Identity-Name     $identity_name;
proxy_set_header X-Session-ID        $session_id;
proxy_set_header X-Tenant-ID         $tenant_id;
proxy_set_header X-Tenant-Namespace  $tenant_namespace;
proxy_set_header X-Request-ID        $request_id;

The upstream contract:

X-Identity-ID is the principal. Treat it as the user's primary key.
X-Identity-Type is USER or SERVICE_ACCOUNT. Some endpoints reject service accounts, some require them.
X-Tenant-ID is the tenant. Always scope queries by it.
X-Tenant-Namespace is the Kubernetes namespace, useful for diagnostics and per-tenant Kafka topic naming.
X-Session-ID is an opaque session correlation ID. Useful for logging, never for auth.
X-Request-ID is the trace correlation ID. Forward it to your downstream calls so the whole graph stitches together.

We do not forward Authorization. The upstream service has no business looking at the JWT. If it needs to know who's calling, it uses X-Identity-ID. If it needs to make a downstream call, it gets a fresh service-account token — it does not replay the user's token.

Stripping Authorization was one of those changes that everyone agrees is a great idea in principle and fights tooth-and-nail when their service breaks during the rollout. Worth the fight.

Skipping tenant for a few endpoints

A handful of endpoints genuinely don't have a tenant: NGINX /healthz, public OAuth callbacks, JWKS endpoints, version probes. For these, tenant resolution must not run.

In NGINX:

location /healthz {
  access_log off;
  return 204;
}

This location is matched before the tenant-resolving if block in the default server, because NGINX processes more-specific locations first. /healthz returns 204 without ever evaluating $tenant_id.

Inside the Auth Service, the equivalent pattern shows up: the trie has rows with endpoint_type=OPEN and no tenant requirement. Even if NGINX did pass through, the Auth Service would allow without checking the tenant. Belt and suspenders again.

The Ingress regex: another layer of opt-in

Our cluster's Ingress controller routes some paths through the Auth Gateway and some paths around it. The chart's multi-tenant Ingress uses a negative-lookahead regex to express "send everything to NGINX except these specific exempt paths":

- path: "/(?!{{ $exemptedPattern }}).*"
  pathType: ImplementationSpecific
  backend:
    service:
      name: {{ $shortName }}
      port:
        number: 80

$exemptedPattern is a long alternation built from values.yaml:

ingress:
  exemptedPaths:
    - api/
    - login/
  exemptedPrefixes:
    - ui
  exemptedExtensions:
    - js
    - css
    - ico
    # ... static assets

Anything matching the regex bypasses the gateway and goes to a legacy ingress. This is how we rolled the gateway out one service at a time — and how we keep it manageable today as services migrate at different speeds.

What we'd warn future-us about

A few real lessons:

Default tenants are forever. If you ship one, every subsequent design decision will assume it exists, and removing it later is a multi-quarter project.
Tenant-aware logging is non-negotiable. Every log line must carry tenant_id. We don't grep by user — we grep by tenant first, then narrow. Chapter 9 has the log format.
Keep the tenant model boring. "What is a tenant?" should have a 1-sentence answer. The moment "tenant" starts meaning different things in different services, your isolation guarantees evaporate.
MT and ST are different operating models, not different security models. The same auth contract should hold. If your ST services can be looser because "they're isolated anyway," you have a problem.
Never derive tenant from the user. "User belongs to tenant X, so I'll use tenant X" sounds reasonable until you have users in multiple tenants. The tenant comes from the request, not from the user.

What's next

Chapter 6 stays on tenant boundaries but zooms in on the authorization side: roles, access levels, the role → access-level → endpoint mapping, and the bitmap fast path that replaced our original string-set matching. We'll see why a JWT should not be the source of truth for a user's full permission set, and how to encode permissions densely enough that the gateway can decide in O(1) bitwise.

Part 4 — Endpoint classification: OPEN, AUTHENTICATED, ACCESS_CONTROLLED

Akarshan Gandotra — Mon, 04 May 2026 18:41:07 +0000

In Chapter 3 the controller branched on something called the "endpoint type":

switch endpointType(perms) {
case "OPEN":            ...
case "AUTHENTICATED":   ...
case "ACCESS_CONTROLLED": ...
}

That branch is the most important conditional in the entire gateway. It decides whether a request even gets a token check, and whether to run authorization. This chapter is about how that decision is data, not code, and the trie that powers it.

Three kinds of endpoint

Every endpoint in our platform falls into one of three buckets:

OPEN — no auth required at all. Health checks, public OAuth callbacks, JWKS, version, docs. The request is allowed without a token.
AUTHENTICATED — token required, no specific permission. "Get my own profile," logout, list-my-stuff endpoints. Anyone with a valid token can call it.
ACCESS_CONTROLLED — token required and a specific permission. Admin operations, deletes, anything that crosses a user boundary.

The Auth Service runs different pipelines for each:

The brilliance — and we say this honestly, because we did not design it this way the first time — is that an endpoint's classification is a column in a database row, not a hardcoded route. Adding a new admin route means inserting a row, not deploying the gateway. We rebuild the in-memory data structure on demand.

The data structure: a trie

When NGINX hits /auth, it forwards X-Original-URI: /user-management/v1/users/abc123 and X-Original-Method: GET. We need to turn that into an endpoint metadata record.

The naïve approach is a big map with (method, full_path) keys. That works until you have wildcards: /users/{id} should match both /users/abc and /users/xyz. Once you have wildcards you want a trie.

Our trie node looks like this:

type TrieNode struct {
    children   map[string]*TrieNode  // exact-match segments
    wildcard   *TrieNode             // catch-all child for {id}-style segments
    Permissions map[string][]string  // method -> required permissions
    EndpointType string               // OPEN | AUTHENTICATED | ACCESS_CONTROLLED
    BitmapMask   uint64               // pre-computed for bitmap fast-path (Chapter 6)
}

A trie key is a slash-segmented path. /users/{id}/roles becomes the path ["users", "{id}", "roles"]. Walking the trie is one segment at a time:

func (t *Trie) Lookup(method string, path []string) (*TrieNode, bool) {
    node := t.root
    for _, seg := range path {
        if next, ok := node.children[seg]; ok {
            node = next
            continue
        }
        if node.wildcard != nil {
            node = node.wildcard
            continue
        }
        return nil, false
    }
    if _, ok := node.Permissions[method]; ok {
        return node, true
    }
    return nil, false
}

O(depth) worst case, where depth is the number of segments. In practice a typical endpoint is 3-5 segments. We're talking nanoseconds.

A toy view of what one slug's trie looks like:

Notice me and {id} are both children of users. users/me resolves first (exact match) and gets AUTHENTICATED. users/abc123 falls through to the {id} branch and gets ACCESS_CONTROLLED + user:read. Order matters: exact wins over wildcard.

Service slug: scoping the trie

We don't have one giant trie. We have one trie per service slug. Why?

Different services own different paths.
A single global trie collides on ambiguous paths (multiple services exposing /v1/users for different reasons).
Cache locality and refresh granularity are better when each service's routes are isolated.

For a long time the slug came from "the first segment of the URI." That was fine until services nested each other (/api/v2/user-management/users). So we added two optional headers:

X-Service-Slug — explicit slug.
X-Request-Path — the path inside that slug, with the prefix already stripped.

The route resolver uses them when present:

func (r *RouteResolver) ResolveEndpointWithMetrics(ctx *gin.Context, log *AuthDecisionLog) (
    slug, trieKey string, perms []string, trieExists, found bool,
) {
    slug = ctx.GetHeader("X-Service-Slug")
    path := ctx.GetHeader("X-Request-Path")

    if slug == "" || path == "" {
        // Legacy: split URI on first segment
        slug, path = splitFirstSegment(log.URI)
    }

    trie := globalTrieRegistry.Get(slug)
    if trie == nil {
        return slug, path, nil, false, false
    }

    node, ok := trie.Lookup(log.Method, segments(path))
    if !ok {
        return slug, path, nil, true, false
    }
    return slug, path, node.Permissions[log.Method], true, true
}

Two return flags, not one: trieExists distinguishes "the service slug isn't registered" from "the service slug is registered but this path doesn't match." The first is a server problem (deploy mismatch). The second is a client problem (404). Different decision reasons, different alerts.

Loading the trie from Postgres

The source of truth is two tables:

-- Each row is one (service, method, pattern) combination
CREATE TABLE endpoint (
    id            uuid PRIMARY KEY,
    service_slug  text NOT NULL,
    method        text NOT NULL,
    pattern       text NOT NULL,
    endpoint_type text NOT NULL  -- OPEN | AUTHENTICATED | ACCESS_CONTROLLED
);

-- Permissions an endpoint requires (only for ACCESS_CONTROLLED)
CREATE TABLE endpoint_policy (
    endpoint_id      uuid REFERENCES endpoint(id),
    general_policy_id uuid REFERENCES general_policy(id)
);

-- The actual policy table — name + bit_index for the bitmap path (Chapter 6)
CREATE TABLE general_policy (
    id        uuid PRIMARY KEY,
    name      text UNIQUE,
    bit_index int  -- nullable; assigned for bitmap-eligible permissions
);

At startup, LoadTrieAndRegistry runs one query per active service slug, joins the three tables, and builds a Trie per slug:

func LoadTrieAndRegistry(ctx context.Context, db *sql.DB) (
    map[string]*helpers.Trie, *policybitmap.Snapshot, error,
) {
    rows, _ := db.QueryContext(ctx, `
        SELECT e.service_slug, e.method, e.pattern, e.endpoint_type,
               coalesce(array_agg(gp.name) FILTER (WHERE gp.id IS NOT NULL), '{}'),
               coalesce(array_agg(gp.bit_index) FILTER (WHERE gp.bit_index IS NOT NULL), '{}')
          FROM endpoint e
          LEFT JOIN endpoint_policy ep ON ep.endpoint_id = e.id
          LEFT JOIN general_policy  gp ON gp.id = ep.general_policy_id
         GROUP BY e.id
    `)

    tries := map[string]*helpers.Trie{}
    snap  := policybitmap.NewSnapshot()

    for rows.Next() {
        var (
            slug, method, pattern, etype string
            permNames []string
            bitIdxs   []int
        )
        rows.Scan(&slug, &method, &pattern, &etype, pq.Array(&permNames), pq.Array(&bitIdxs))

        if _, ok := tries[slug]; !ok {
            tries[slug] = helpers.NewTrie()
        }
        node := tries[slug].Insert(segments(pattern))
        node.EndpointType = etype
        node.Permissions = map[string][]string{method: permNames}
        node.BitmapMask  = computeMask(bitIdxs)  // Chapter 6
    }
    return tries, snap, nil
}

A single SQL query, one pass to build N tries. We measure load time and emit it as a startup log: trie_load_duration_ms. On a healthy database, hundreds of tries with thousands of routes load in well under a second.

Refreshing without a restart

Endpoint metadata changes — we onboard a new service, add a new permission, deprecate a route. We don't want to roll the gateway every time.

There are two refresh mechanisms:

1. Periodic. Every TRIE_REFRESH_INTERVAL_SECS (default: 1 hour) we re-run the loader. This is a safety net. If the live channel ever misses an event, periodic catches it within an hour.

2. Live. A Redis Pub/Sub channel called auth:trie:refresh. When admin tooling changes endpoint metadata, it PUBLISHes to that channel. Every Auth Service pod is subscribed, and refreshes within milliseconds.

The Pub/Sub message itself carries no payload. It's purely a kick. Each pod queries the database itself. Two reasons:

We don't have to keep the message in sync with the schema. New columns, no message change.
A pod that just booted does the same load as a pod that received a refresh kick. One code path.

The downside: every refresh kick = one query per pod. If you have 100 pods and someone bulk-edits 1000 endpoints with 1000 publishes, that's 100,000 queries. We added a debounce (coalesce events within a 200 ms window). Onboarding a service still hammers the DB once, briefly, but it doesn't spiral.

Caching the lookup

Even with a fast trie, re-walking the same path on every request is wasteful. We layer a TinyLFU cache (W-TinyLFU, via a Go port) on top of the resolver:

type RouteCache struct {
    inner *tinylfu.Cache[string, RouteResult]
}

func (c *RouteCache) Get(slug, method, path string) (RouteResult, bool) {
    return c.inner.Get(slug + "\x00" + method + "\x00" + path)
}

The key is service_slug + NUL + method + NUL + path. The NUL byte is a delimiter — paths can't contain it, so we can't accidentally collide a slug with the start of another slug.

The cache is bounded by entry count, not by RAM. Default: 10,000 routes with 100,000 frequency counters. TinyLFU keeps the frequently used routes hot and evicts cold ones — better than LRU when traffic has a long tail of rarely-hit paths.

Cache invalidation: on any trie reload (periodic or kick), the cache is fully cleared. We considered partial invalidation. We chose not to — the cache fills back up in milliseconds because the same handful of paths drives 99% of the traffic.

Why this beats per-route decorators

In a Python or Node service you might write:

@app.route("/users/<id>", methods=["GET"])
@requires_permission("user:read")
def get_user(id): ...

Three problems with that:

Service-by-service drift. Each service maintains its own decorators. Different services use different permission strings, different exception types, different log shapes. The contract is informal.
Fragile to refactor. Move the route, lose the decorator. Now anyone can call it. We've seen this happen at zero, two, and twelve months in.
Audit is impossible at the service level. "What permission protects this URL?" requires reading every service's source.

By making endpoint metadata a database row, we get:

One contract, in one schema.
Decoupled from code — no rebuild to add a route.
Auditable: a single SELECT answers "what does this URL require?"
Reusable: the same metadata drives the gateway and the admin UI that lets product managers tweak it.

The cost is that you can't read the auth requirements next to the handler in the upstream service's code. We mitigate that with code generation: a CI job dumps endpoint rows for each service into a routes.yaml checked into the service's repo for reference. The DB stays the source of truth.

Readiness depends on the trie

There's a subtle interaction with Kubernetes probes worth calling out. Our /readyz returns 503 if the trie isn't loaded:

func CheckTrieReadiness(tries map[string]*helpers.Trie) (int, gin.H) {
    if len(tries) == 0 {
        return http.StatusServiceUnavailable, gin.H{"status": "trie not loaded"}
    }
    return http.StatusOK, gin.H{"status": "ok"}
}

A pod that boots before Postgres is reachable will fail readiness, which keeps it out of the Service load balancer until the trie is loaded. A pod that was serving traffic and then loses access to Postgres keeps its existing trie in memory and stays ready — refreshes fail loudly via Slack, but live traffic isn't disrupted.

This split — "load fails block readiness, refresh failures don't" — is on purpose. A booting pod with no trie can't make decisions; pull it out. A serving pod with a stale trie can still make decisions correctly for endpoints that haven't changed; keep it in service while we fix Postgres.

Anti-patterns we avoided

Worth listing what we considered and rejected, because they're tempting:

Storing the endpoint type inside the JWT. Tempting because then you don't need a trie. Wrong because a token outlives configuration changes — we'd cache stale auth requirements.
A single hard-coded list of public paths. A previous iteration had a YAML file shipped with the gateway. Updating it required a deploy. The trie + DB replaced it.
Per-tenant route metadata. We talked about it. We rejected it: the same service exposes the same routes for every tenant. Tenant-specific differences belong in the access-level model (Chapter 6), not the route model.
Letting the upstream service register its own routes via API at startup. Looks elegant. Falls apart in chaos: a buggy service can register away its own auth.

What's next

Chapter 5 takes the same metadata-as-data philosophy and applies it to multi-tenancy. The trie tells us what an endpoint requires; tenant resolution tells us who it's for. We'll see how NGINX server blocks, headers, and tenant maps cooperate to never let a request through without a clear tenant identity.

Part 3 — Inside the Auth Service: From Token Validator to Policy Decision Point

Akarshan Gandotra — Mon, 04 May 2026 18:40:48 +0000

Most auth services start simple — verify the token, return 200 or 401. Then requirements accumulate. Tenant isolation. Service accounts. Token revocation. Access levels per endpoint. And suddenly what was a lightweight validator is carrying a lot of weight, without a clear structure to hold it.

This post is about how we structured ours — the ideas that shaped it, and the ones we got wrong before landing here.

One job, lots of supporting infrastructure

The Auth Service does exactly one thing from the outside: receive a subrequest from NGINX, inspect the headers, and return a decision. Under a millisecond, every time.

But a single HTTP handler that does that reliably at scale has a lot underneath it — caching, revocation checks, routing logic, identity propagation. The structural challenge is keeping the handler small while the infrastructure grows. We landed on a controller that reads like a flowchart:

Extract the request metadata (URI, method, tenant).
Resolve the endpoint to find out what kind of auth it needs.
Based on that: allow it openly, run authentication only, or run authentication and authorization.

That's the whole thing. Everything else is a service the controller delegates to.

The insight that changed how we think about routing: endpoint classification is data, not code

Early on, we made auth decisions in code. A route was open because someone wrote if path == "/health" { return 200 }. Access control lived in conditionals scattered across handlers.

This breaks the moment your product team adds a new endpoint, or you need to temporarily open a route for a partner integration, or you realize a route that was open should have been authenticated all along.

We flipped it: every endpoint in the system has a classification stored in the database — OPEN, AUTHENTICATED, or ACCESS_CONTROLLED — along with a permission list if it's access-controlled. The auth service resolves the incoming request to an endpoint record and reads that classification. The decision logic then becomes a simple switch:

OPEN: allow, log it, done.
AUTHENTICATED: run token validation.
ACCESS_CONTROLLED: run token validation, then check permissions.

The consequence is that we never recompile or redeploy the Auth Service to change how a route is protected. That's a database update. It also means non-engineers can reason about the access model without reading code.

Naming every failure: the decision-reason contract

The second structural idea that shaped everything else: every outcome has an explicit name.

We maintain an enumerated list of decision reasons — constants like MISSING_TOKEN, TENANT_MISMATCH, TOKEN_REVOKED, SA_VERSION_MISMATCH, OPEN_ENDPOINT, ACCESS_LEVEL_MATCH. Every code path in the service must set one before returning. There's no exit that doesn't produce a named reason.

const (
    ReasonOpenEndpoint       = "OPEN_ENDPOINT"
    ReasonMissingToken       = "MISSING_TOKEN"
    ReasonTokenRevoked       = "TOKEN_REVOKED"
    ReasonTenantMismatch     = "TENANT_MISMATCH"
    ReasonSAVersionMismatch  = "SA_VERSION_MISMATCH"
    ReasonTokenTypeMismatch  = "TOKEN_TYPE_MISMATCH"
    // ... and so on
)

This sounds like a minor logging detail. It isn't.

When a token fails, why it fails tells a completely different story depending on the reason. TOKEN_REVOKED means the user logged out or was disabled. SA_VERSION_MISMATCH means a service account was rotated and the calling service hasn't caught up. TOKEN_TYPE_MISMATCH means something is trying to authenticate with a refresh token where it should use an access token — usually a buggy SDK, occasionally something worth investigating.

If all of these collapsed into a generic 401 Unauthorized, you'd lose all of that signal. Dashboards would be useless. On-call would be guessing.

The list itself is a contract with the log pipeline. New reasons go through code review. Old reasons can't be deleted without checking dashboards and alerts first. It's one of the few places in the codebase where "this is more rigid than it needs to be" is actually correct.

One log line per request — and why that matters more than it sounds

Our first approach was to emit log lines at each stage of the pipeline — one when we resolved the route, one when we validated the token, one when we made the authorization decision. We could stitch them together by request ID.

We abandoned this. The stitching was always slightly wrong. Correlation IDs got dropped. Fields you needed were in a different log line than the one you found first. Debugging a production incident meant reconstructing a timeline from fragments.

Now there's one structured log record per request. It's built up incrementally — every handler in the pipeline writes into the same struct. By the time the response goes out, the record has every field: URI, method, tenant, identity, cache hit status, decision reason, outcome. It emits once, at the end.

The operational improvement was immediate. Grepping for a user's identity ID gives you a complete picture of every request they made — what was allowed, what was denied, and exactly why. No joining, no reconstruction.

If you're designing an auth service, this is the first structural decision we'd recommend getting right. Everything else can be refactored. The logging model tends to calcify early.

How we handle JWT verification at scale

Validating a JWT sounds cheap. For HS256 with a shared secret, it mostly is. For RS256 with asymmetric keys — which is what we use for user-facing tokens — the RSA verification step sits in the hundreds of microseconds. At meaningful request volume, that becomes a real CPU cost.

Our solution is a cache in front of the decode step. The cache key is a hash of the raw token string (not the string itself — the hash is 8 bytes versus potentially hundreds, which adds up at scale). The TTL matches the token's expiry. When a token comes in that we've already verified recently, we skip the RSA verification entirely.

A few things we were careful not to cache:

Revocation state. Whether a token has been revoked can change at any moment, independent of the token's validity. We cache the decode result — the claims, the identity — but we always check revocation live. These are different questions.

The auth decision itself. The decision depends on the endpoint, the tenant, and the required access level, none of which the token cache knows about. Caching decisions would mean a user who got their access level changed mid-session would still see stale decisions until cache expiry. Unacceptable.

The principle here generalizes: cache the facts (what the token says), not the decisions (what we're going to do about it).

The boundary the Auth Service deliberately doesn't cross

The clearest sign a service is well-designed is what it refuses to do.

Our Auth Service handles coarse-grained access: does this identity have the level of access required to reach this endpoint category? That's it. It does not answer questions like "can this user delete this specific record?" or "does this account have permission to access this tenant's billing history?"

Those are business policy questions. They belong in the services that own that data, where the full context exists.

Every time we've been tempted to push business logic into the Auth Service — usually because it would be convenient, or because a product requirement seemed auth-adjacent — we've regretted it. Business policy changes frequently. Auth infrastructure should be boring and stable. Keeping them separate means changes to one don't put the other at risk.

The Auth Service also doesn't store sessions, doesn't issue tokens, and doesn't look up users. Tokens carry enough identity for upstream services to do that themselves. The Auth Service only validates.

The pattern underneath all of this

Looking back, the decisions that held up over time share a common shape: make the implicit explicit.

Endpoint classification pulled auth rules out of code and into data. Decision reasons named every outcome instead of letting them collapse into status codes. The single log line made the request lifecycle visible as a single artifact instead of scattered fragments. The cache/decision boundary separated "what the token says" from "what we're going to do about it."

None of these are particularly novel ideas. But they compound. A service where every decision is named, every outcome is logged atomically, and every boundary is deliberate is a service you can actually operate.

That's the goal.

Next up: Chapter 4 — the path trie that resolves incoming URIs to endpoint records in O(path length), without a database call on the hot path.

DEV Community: Akarshan Gandotra

[Boost]

Zero Standing Access for AI Agents: Let Users Lend Access, One Session at a Time

Zero Standing Access for AI Agents: Let Users Lend Access, One Session at a Time

Where the industry is today

Seven principles

A new entity in IAM: the agent

What an agent can borrow

Agent templates: permissions plus context

The permission catalog: an IAM tool for builders and agents

Journey 1: building and publishing a template

Journey 2: a user's session, end to end

Journey 3: the next day, and the audit

Modes and rules, like your AI coding tool

Under the hood

What can go wrong

The part I'm still arguing about: whose name goes in sub?

Closing

The CoreDNS Black Hole: how one dead DNS pod broke our API gateway

The analogy: the receptionist who went home

If you haven't met these three before

Why it looked random

We proved it instead of guessing

The fix: three layers

The numbers

The rule we refused to break: nginx must always boot

Connection pooling, in 60 seconds

The bonus bug: a pool that never pooled

Tag the caller, not the endpoint

Takeaways

A Practical Guide to Using Auth0 as an Identity Broker for Epic SMART on FHIR

Understanding the authentication flow

Start by defining the purpose of the integration

Using Epic only for authentication

Calling Epic FHIR APIs

Register the callback correctly in Epic

Configure Epic as an OIDC enterprise connection

Use the authorization-code flow

Enable the connection for the Auth0 application

Configure the required SMART scopes

openid

fhirUser

launch

profile

Preserve the SMART launch parameters

Validate the incoming iss value

Separate EHR launch from standalone launch

EHR launch

Standalone launch

Pass the correct parameters upstream

Use state to correlate the complete flow

Understand how Auth0 Organizations affect login

Decide how users will be linked

Keep the Auth0 token and Epic token separate

Plan how the backend will obtain the Epic token

Store launch context separately from the user identity

Validate the token response

Add structured logging around each redirect boundary

Test each integration layer independently

Production checklist

Should Auth0 sit in the middle?

Final thoughts

Further reading

Part 10 — Lessons learned building a Kubernetes Auth Gateway

What worked

auth_request as the primitive

Endpoint metadata as data

One structured log line per decision

Fail-closed by default at the edge

Caches in the auth process, not in NGINX

Pub/Sub-driven trie reload

The bitmap fast path

What hurt

Tenant resolution living in two places

The "first segment is the slug" rule

Migration to the bitmap took longer than expected

Cache invalidation was an afterthought

The default tenant we shipped on day one

Per-pod alert spam, twice

What we'd build earlier

The part I'm still arguing about: whose name goes in `sub`?

`openid`

`fhirUser`

`launch`

`profile`

Validate the incoming `iss` value

Use `state` to correlate the complete flow

`auth_request` as the primitive

1. The structured `AUTH_DECISION` log

1. Start with `auth_request`. Don't shop architectures.

`/livez` — process is alive

`/readyz` — pod can take traffic

`/healthz` — deep health