DEV Community: Fernando Azevedo

One listing, many legal entities: AWS Marketplace subsidiaries in practice

Fernando Azevedo — Fri, 30 Oct 2026 17:14:07 +0000

Anyone who has operated a group with one entity per country knows the hidden cost of AWS Marketplace: the same listing recreated in every seller account, with descriptions and pricing dimensions that drift silently. On 10/29/2026 AWS made it possible to publish the product once in the primary account and authorize subsidiaries to sell it through direct private offers, each under its own tax identity, currencies and bank account. The announcement sells this as the end of duplication. I read it differently: duplication leaves the catalog and becomes an authorization problem, with states, defaults and a revocation that does not do what the name suggests.

The situation: the catalog was never the problem

In a group with a US entity, a Brazilian Ltda and an Indian subsidiary, the reason for three seller accounts is not technical, it is fiscal. Each entity has to be the seller of record in its market: it holds the customer contract, invoices in local currency, pays its own Marketplace fees and gets paid into its own bank account. The documentation is explicit about this, and the new capability does not change that premise. It only removes the obligation to maintain the product inside every account.

The expensive part was never creating the listing. It was keeping three copies in sync for years: every new pricing dimension, every description tweak, every product version multiplied by the number of entities, and every drift found by a customer before you found it. That is the maintenance cost this capability attacks, and it is real.

What it puts in its place deserves the same care. The primary account now issues authorizations, one per product and per subsidiary, stating in which countries and with which pricing dimensions that entity can sell. That is access control over revenue. In a regulated environment, anything that decides who sells what, to whom and where belongs in the same bucket as an IAM role: it needs an owner, an audit trail and periodic review.

One detail that tempers the optimism of anyone who already has duplicate listings: live agreements on a subsidiary's own listing stay there and cannot be moved to the primary account's product. The migration applies to new offers, not to the installed base. You will live with both models until the last legacy contract expires.

The model: two layers and nothing shared by default

Authorization has two layers, and mixing them up is the first mistake I would expect to see in production.

Connection: done in AWS Partner Central. The primary sends the request, with up to 10 account IDs per submission, and an administrator in the subsidiary accepts it. Acceptance has a side effect that needs legal sign-off: it records the subsidiary's consent to share partner attributed revenue and its Marketplace insights reports with the primary. The connection alone authorizes no sales at all.

Authorization: a SubsidiaryAuthorization@1.0 entity in the Catalog API, created by the primary with StartChangeSet and the CreateSubsidiaryAuthorization change type. It starts in Draft, becomes Active through UpdateAvailability and ends in Restricted. It carries the product ARN (immutable), the Dimensions list, the SubsidiaryAccountId (also immutable, it becomes the SubsidiaryTargetingRule) and a BuyerTargetingTerm with CountryCodes and OfferTargeting, which today accepts only PrivateOffer.

Service role: both accounts need AWSMarketplaceSubsidiaryAuthorizationServiceRole, which AWS uses to create the resource share between them. Without it, in the console the authorization is created and never becomes active, and the subsidiary sees nothing. Through the API the error shows up as INCOMPATIBLE_ACCOUNT_STATE on the primary or INCOMPATIBLE_SUBSIDIARY_ACCOUNT_STATE on the subsidiary, and only in DescribeChangeSet, because validation is asynchronous.

Human IAM: separate from the service role, and you need both. To authorize, the primary uses AWSMarketplaceSellerFullAccess or AWSMarketplaceSellerProductsFullAccess; to create an offer from the Private offers page, the subsidiary uses AWSMarketplaceSellerOfferManagement. Those managed policies are broad. In production I would write a custom policy restricted to the Catalog API actions each team actually uses.

Who controls what: product in the primary, contract and money in the subsidiary

The diagram answers one question: where the control boundary sits between accounts. The primary owns the product and the authorization; the subsidiary is the seller of record, sets the price and gets paid. The dashed line is reporting visibility only and moves no money. Simplification: the Partner Central connection and the service role are implicit in the sharing arrow.

Primary account

Authorization repo Catalog API (ci)
Product listing SaaSProduct (data)
SubsidiaryAuthorization Draft, Active, Restricted (security)
Marketplace insights consolidated (data)

Subsidiary account

Private offer seller of record (compute)
Disbursement local bank, currency (storage)

Buyer

Buyer account tax address (user)

Flows

pipe -> authz: StartChangeSet
authz -> prod: references ARN
authz -> offer: scopes countries and dimensions
offer -> buyer: private offer
buyer -> bank: pays via AWS
offer -> insights: reports

Where the risk lives: revoking is not switching off

The announcement says that revoking an authorization immediately stops the subsidiary from creating new offers. Correct, and incomplete for anyone designing a control. The documentation details four behaviors that change the runbook.

Published offers survive: revoking does not cancel offers already published. The buyer can still accept an offer the subsidiary published before revocation, and existing agreements continue under their agreed terms, including renewals and disbursements to the subsidiary. If the reason for revoking is the divestiture of the entity, a restriction in a country or a corporate dispute, the block does not reach the exposure already out in the market.

Restricted is terminal: there is no reactivation. To authorize again, you create a new authorization, which gets a new ID. Any system using the authorization ID as a business key breaks on the first accidental revocation.

The connection takes everything down: ending the connection in Partner Central ends every authorization between the two accounts. An analyst who removes a connection to fix revenue attribution takes down the selling capability of the whole entity.

Narrowing breaks drafts: removing a country or a dimension makes publishing fail for draft offers that used that scope. The offer stays parked and the local seller only finds out when trying to publish, usually the day before quarter close.

The practical lesson: revocation is a process, not a click. Revoke the authorization, list the subsidiary's published offers not yet accepted, decide what happens to each one, and only then touch the connection. To shorten the exposure window, use short expiration dates on the private offers subsidiaries publish.

Authorization as code: `UpdateInformation` replaces everything

With more than two subsidiaries, console-made authorizations drift within weeks. The Catalog API lets you treat each authorization as a versioned resource, and three details make that mandatory, not cosmetic.

Full replacement: UpdateInformation replaces Name, Description, Resource and Terms wholesale; anything you omit is removed. Synchronous validation returns 422 if Dimensions is empty, but a BuyerTargetingTerm without CountryCodes passes, and an empty list means all countries. A script that only meant to change the description can open the Brazilian entity's sales to the whole world. The only safe approach is to always send the complete state, read from a versioned file.

The default sits on the wrong side: buyer eligibility is evaluated against the buyer's tax address. An authorization with no countries contradicts the fiscal premise that justified the local entity in the first place. In the pipeline, I would block the merge of any authorization with an empty CountryCodes.

Products do not propagate: adding a dimension to the product does not reach the subsidiary; you have to call UpdateInformation with the new list. If the product has Metered or ExternallyMetered dimensions, all of them must be in the authorization, otherwise the change set fails with INCOMPATIBLE_PRODUCT_DIMENSIONS.

The rest is idempotency and observability. StartChangeSet is asynchronous: store the ChangeSetId, poll DescribeChangeSet until SUCCEEDED or FAILED, and read the ErrorDetailList. Two authorizations for the same product and subsidiary cannot overlap countries and dimensions (DUPLICATE_SUBSIDIARY_AUTHORIZATION), which gives you a natural key for your state: product, subsidiary and country set, not the ID. Reconcile daily with ListEntities using EntityType: SubsidiaryAuthorization and diff against the repository.

Playbook for tomorrow

Map entity, country and currency: Build the entity by country by currency table and confirm every subsidiary has completed its tax interview, KYC and payment setup, with a bank account and at least one disbursement currency. Without payment setup the subsidiary cannot create offers, even with an active authorization. In India, the account must be standalone, outside AWS Organizations, disburses only in INR and sells only to buyers in India.
Create the service role on both ends: In each account, in Partner Central, Settings, Service roles tab, create AWSMarketplaceSubsidiaryAuthorizationServiceRole before the first authorization. Without it the authorization never activates and the console gives no warning.
Accept the connection with legal sign-off: Acceptance records consent to share attributed revenue and insights reports with the primary. In a group with minority shareholders or a joint venture, that is a corporate decision, not an IT one.
Version every authorization: One file per product and subsidiary, with explicit CountryCodes and the complete Dimensions list. The pipeline sends the whole state via UpdateInformation, polls DescribeChangeSet and fails the build on any non-empty ErrorDetailList.
Release in two steps: Create in Draft, review DescribeEntity against the file and only then call UpdateAvailability with Active. Activation takes a few minutes; do not promise the local team the offer goes out the same minute.
Write the revocation runbook: Revoke the authorization, list published and unaccepted offers, decide what happens to each one, and only then touch the connection. Include who approves, because Restricted cannot be undone.
Reconcile daily: ListEntities with SubsidiaryAuthorizationFilters by status and account, diffed against the repository. Use the primary's consolidated Marketplace insights to cross-check offers, agreements and renewals per subsidiary.

The authorization name is an interface: The subsidiary picks the authorization by its Name (1 to 100 characters) when creating an offer. Put the product and scope in the name, for example Security Suite | BR, AR | users, api, and generate it from the versioned file. A hand-written name drifts from the real scope on the first change, and the local seller picks the wrong authorization without noticing.

Three ways to sell through more than one entity

Criterion	Subsidiary authorization	Resale authorization (CPPO)	Duplicate listing
Who sells	Another seller account of your own group	Third-party Channel Partner	Each account, with its own product
Price	Subsidiary sets it; the authorization carries no price	Markup over the wholesale set by the ISV	Each account sets it
Currency	Those configured in the subsidiary itself	The one the ISV defines in the authorization	The account's own
Offer types	Direct private only	Private through the partner	Public and private
Maintenance cost	1 listing + N versioned authorizations	1 listing + one authorization per partner	N listings synced by hand
Use when	Local entity must be seller of record and sells through private offers	A third party resells and needs margin	The local entity needs a public offer or its own CPPO

Anti-patterns I would expect in the first quarter

Authorization without CountryCodes: an empty list means all countries, and the entity created for one fiscal market can now sell to any tax address.
Treating revocation as a kill switch: offers already published remain acceptable and agreements keep renewing; the block applies only to new offers.
Partial updates by script: UpdateInformation removes everything not sent, so a description patch can wipe countries and dimensions.
Removing the connection to fix attribution: ending the connection ends every authorization between the two accounts at once.
Authorization ID as a business key: Restricted is terminal and re-authorizing creates a new ID; key on product, subsidiary and countries.
Shipping a new dimension and forgetting the authorization: the subsidiary keeps selling the old product version until someone calls UpdateInformation.

Questions that will come up

Can the subsidiary create a CPPO for a local channel with the authorized product?

No. The Catalog API rejects a resale authorization created from a product the primary authorized. The subsidiary can only create CPPOs for products it owns.

Can I migrate agreements from the duplicate listing to the primary's product?

No. They stay on the original listing. Plan for coexistence until expiry and use the authorization at renewal time, as a new offer.

Can the primary create offers or complete KYC on the subsidiary's behalf?

No. Each account creates its own offers and completes its own tax interview and KYC. The service role also gives the primary no access to the subsidiary's offers, agreements or financial data.

Are Marketplace fees consolidated in the primary?

No. Listing and transaction fees are charged to the transacting subsidiary and are not pooled, netted or rebilled across accounts. Your transfer pricing model still lives outside the Marketplace.

What does it cost and where is it available?

No additional cost, in all Regions where AWS Marketplace is available, according to the 10/29/2026 announcement.

Curator's note: I would start with the lowest-volume subsidiary, a single product and two countries, and only move to the next one after running the revocation runbook end to end in a test setup. Before the first active authorization, the pipeline would already be blocking empty CountryCodes and partial updates. The hard-won lesson I bring from financial platforms: the control that causes the most trouble is not the one that blocks, it is the one that looks like it blocks and does not. Here, that control is called revocation, and its name promises more than it delivers.

Verdict

Adopt subsidiary authorization when the group has local entities that must be seller of record, sells mostly through private offers and maintains duplicate listings today: the maintenance gain is direct. Do it with authorizations in a repository, CountryCodes always explicit and a revocation runbook that handles published offers. Keep the local listing when the entity needs a public offer or its own CPPO, because neither is supported in this release.

Rating: Adote com runbook

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

AWS Builder Loft São Paulo: a new home for building together

Fernando Azevedo — Fri, 30 Oct 2026 17:14:04 +0000

São Paulo has a new home for builders. I left thinking about what we could build together. On the evening of October 29, the talks, conversations and community encounters at the AWS Builder Loft São Paulo opening left me with a welcome feeling: many people are ready to learn, share what they know and help an idea become something real.

Fernando Azevedo

What I took home from that evening

As a solutions architect and an AWS Community Builder in AI Engineering, I spend much of my time connecting real problems with solutions that work. Good architecture takes questions, context and a willingness to reconsider choices. A good community needs those things too.

That was what excited me most that evening. Being with people who want to create makes room for conversations that are difficult to fit into a presentation or a quick exchange of messages. A question reaches someone who has faced the problem before. One person's experience gives someone else the confidence to begin. An idea gains a second perspective.

I left happy to have been part of this beginning and keen to keep contributing. The collage accompanying this piece preserves some of that memory: the talks, people gathering, the space and the small details that make an encounter memorable.

A place designed for continuity

The idea behind the Loft was already present in the official AWS announcement of the expansion to Berlin, Hyderabad and São Paulo: permanent spaces for learning, collaboration and community activities. For São Paulo, the announcement highlights connections with local communities, universities and startup networks.

That continuity matters. Having somewhere to return gives a conversation a next chapter. Someone can arrive with a question, try an approach and come back with what worked, or what needs to change. Knowledge travels with the people involved.

I see particular value in bringing beginners together with people who have production experience. Beginners ask questions that make us explain things better. People who have dealt with incidents and difficult decisions can share the precautions that do not always appear in a demonstration.

A conversation can become a partnership. A question can open the way to a solution.
Fernando Azevedo

Why this matters for Brazil

In my view, a space like this can connect people working on similar challenges who have not yet met. Developers, architects, students, data professionals, security specialists and founders have much to learn from one another. Bringing those perspectives together makes it easier to understand the problem before choosing the technology.

I also see an opportunity to create more knowledge grounded in our own circumstances. Costs, operations, accessibility, team development and the needs of Brazilian users belong in technical discussions. Sharing those experiences in Portuguese helps more people participate; publishing them in English as well broadens the exchange with the global community.

The work can reach beyond people who are able to be physically present in São Paulo. A gathering can lead to a public repository, an accessible guide, a recording or an account that explains a decision. That is how I imagine the Loft's energy reaching other cities too: through useful materials, collaboration between groups and people willing to carry ideas forward.

What I would like to bring to the table

I want to contribute what I have been studying and building in architecture, cloud and artificial intelligence, and learn from people with different experiences. In AI Engineering, I see room for sessions where we examine a real solution: the problem it solves, its limits, how to test it, what it costs to operate and what happens when it fails.

One idea would be a hands-on session about agent tools, using a small example that anyone can reproduce. Another would be an architecture discussion where the group compares alternatives and records the decision. These are ideas I would like to develop with the community, not activities already scheduled at the Loft.

My open AWS architecture MCP, architecture studies and blog articles can provide starting points. The best outcome would be someone testing an example, finding a limitation, asking a better question and helping us improve the material.

From an encounter to a next step

For someone just starting out, my invitation is simple: bring a real question. You do not need a finished project or knowledge of every service. Explaining what you tried and where you got stuck is already a good start to a conversation.

For those with more experience, it is worth bringing the difficult choices too: the alternative you rejected, the unexpected cost, the failure that changed the design. These accounts help others move forward with a clearer understanding.

When a conversation produces an idea, we can choose a small next step: write an example, review a diagram, test a hypothesis or open a repository discussion. Recording what we learn and crediting contributors keeps the encounter going. AWS Builder Center and AWS User Groups offer ways to continue that exchange. To attend future in-person activities, check the announcements and registration details for the local edition.

Thank you for being part of this beginning

My thanks to the AWS team for the invitation and organization, to the speakers, AWS Heroes, community leaders and everyone who took part in that evening. It was a pleasure to be there with you.

I left AWS Builder Loft São Paulo grateful and eager to keep the conversation going. I want to continue contributing alongside AWS builders, users and customers, sharing what I learn and listening to the problems we still need to solve.

What idea, challenge or project would you bring to the table? It could be an unanswered question, a prototype that needs another perspective or a topic you would like to teach. Let's start there.

Let's learn, share and build together. There is much to look forward to in the AWS community in Brazil.

Background and ways to stay involved

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Your agent crashed after the write. Should you retry?

Fernando Azevedo — Tue, 27 Oct 2026 16:46:39 +0000

The hardest moment in an automation often sits between two acknowledgements: the target accepted the change, but the executor has not recorded completion. Restarting the agent restores process availability; it does not resolve uncertainty about the previous effect. In this lab, I terminate the process in that exact interval and inspect what survives. The next decision depends on persistent evidence, not the model's confidence.

What was verified

Implementation: lab-security-remediation, tag v0.4.0; local tests executed on October 27, 2026.
Observed result: 22 tests passed. In the receipt exercise: version 2, one receipt, and no second change after recovery and resubmission.
Scope: Local SQLite, synthetic target, simulated review. No credentials or AWS calls required.
Not demonstrated: Does not establish exactly-once execution across remote APIs, power-loss resilience, or production-ready authorization.

Start with the question the log cannot answer

Imagine a tool that fixes a configuration. It receives a reviewed proposal, checks the target version, and applies the change. Before returning success, the process terminates. The client sees an interruption. The target may already have changed. A log line saying the call started cannot distinguish those possibilities.

In this exercise, the target is only a SQLite row representing a synthetic bucket. The blockPublicAccess property is a teaching device; this is not an S3 call. The proposal identifies the operation and the expected version. An intent record, called the ledger, stores the reviewed request. A receipt records that this request produced this target version.

That separation creates a testable question: what evidence connects this operation to this change? Seeing the desired configuration does not tell us who changed it. Finding a restarted process does not tell us whether its predecessor finished. A client error describes the client's observation, not necessarily the target's reality.

Where evidence must survive

Local lab architecture. The tables share one SQLite file, but intent, effect, and completion use separate transactions. The target update and its receipt are atomic with each other.

Local CLI

Reviewed executor (compute)
Recovery reader (compute)

SQLite · separate commits

Operation ledger (storage)
Target + receipt (data)
Audit events (storage)

Flows

executor -> ledger: INTENT
executor -> target: EFFECT
target -> recovery: EVIDENCE
ledger -> recovery: REQUEST
recovery -> ledger: CLASSIFIES
recovery -> events: DECISION

Prepare a run another person can reproduce

Use Node.js 22.13 or later with node:sqlite available. The tagged source is the reference for this exercise; do not assume the published npm version is identical. Installing dependencies is not required to run this lab. Clone into a new directory, check the tag, and run the tests before changing a scenario.

Choose fresh database names for each complete repetition. A reused database retains history, and that history changes the expected result. Preserve the original proposal too: generating another after the change can produce a different digest for the same identifier and should cause a conflict, not implicit permission to execute again.

The intentional failures are separate from inspection commands. In a script using set -e, explicitly capture exit code 86 so the expected termination does not end the investigation. The uncertainty exercise returns 2, another expected outcome. These codes are evidence; do not erase them with an indiscriminate || true.

Setup: pinned source and local tests

git clone https://github.com/fernando-moretes/lab-security-remediation.git
cd lab-security-remediation
git checkout v0.4.0
node --version
npm test

Experiment 1: the effect exists; the acknowledgement was lost

Create the database and export the proposal. Running with --approve simulates explicit review and records intent. The after-target crash point terminates the process after the commit that updates the target and writes its receipt, before marking the operation complete. This is not an exception caught inside one function: the CLI exits, and subsequent commands open the database in other processes.

After the crash, inspect three things together: operation status, target version, and receipt existence. In the October 27 run, the operation was pending, the target was at version 2 with blockPublicAccess: true, and one receipt matched the proposal digest. That combination demonstrates why retrying solely because the ledger is pending would be an uninformed decision.

Intentional crash: run exits 86; continue with inspect

node bin/cli.mjs durable init ./receipt.sqlite
node bin/cli.mjs durable plan ./receipt.sqlite > receipt-proposal.json
node bin/cli.mjs durable run ./receipt.sqlite receipt-proposal.json --approve --crash after-target
# Expected exit: 86
node bin/cli.mjs durable inspect ./receipt.sqlite
node bin/cli.mjs durable recover ./receipt.sqlite synthetic-finding-001
node bin/cli.mjs durable run ./receipt.sqlite receipt-proposal.json --approve
node bin/cli.mjs durable inspect ./receipt.sqlite

Recovering the record does not require repeating the effect

The recover command reads the operation, receipt, and current state. Here it changes the classification to reconciled and records the decision in the event history. It does not mutate the target. Resubmitting the original proposal also returns reconciled, with retryable: false. In the final inspection, the version remains 2 and the receipt count remains 1.

Those are the invariants worth checking. The test is not simply searching for the word success in the output. Compare the observable effect before and after recovery. Compare it again before and after resubmission. If the version increased again, an apparently correct status would hide a second execution.

A receipt is historical evidence, not a permanent health certificate. If the target changes later, the earlier operation's history can remain valid while current state diverges. The implementation exposes driftAfterReceipt to distinguish those questions. Restoring a desired configuration, when necessary, requires a new reviewed decision about current state.

Observed results from both experiments

Inspection	With receipt	Without receipt, with drift
After interruption / drift	pending; version 2; 1 receipt	pending; version 2; 0 receipts
Recovery	reconciled; exit 0	uncertain; exit 2
Original proposal resubmission	reconciled; version remains 2	uncertain; version remains 2
Cloud calls	0	0

Experiment 2: the state looks right; ownership is unknown

Now use another database. Interrupt execution at after-intent, before applying the effect. Then use the teaching command drift to change the target through another path. During recovery, the desired property is true and the version is 2, just as in the first experiment. The decisive difference is the missing receipt.

The observed result is uncertain, retryable: false, with exit code 2. Resubmitting the original proposal preserves that classification and does not mutate the target again. The system does not infer ownership from matching values. This behavior trades automatic progress for a decision that can be investigated.

Do not delete the operation to unblock a queue. That removes the memory used to recognize duplicates. Preserve the database, proposal, and results; investigate the change and define the next step as a new decision. The lab offers no magic administrative resolution button and should not suggest that manually editing the ledger is a safe operating procedure.

The final configuration matches, but the receipt is missing

node bin/cli.mjs durable init ./uncertain.sqlite
node bin/cli.mjs durable plan ./uncertain.sqlite > uncertain-proposal.json
node bin/cli.mjs durable run ./uncertain.sqlite uncertain-proposal.json --approve --crash after-intent
# Expected exit: 86
node bin/cli.mjs durable drift ./uncertain.sqlite true
node bin/cli.mjs durable inspect ./uncertain.sqlite
node bin/cli.mjs durable recover ./uncertain.sqlite synthetic-finding-001
# Expected exit: 2, status: uncertain
node bin/cli.mjs durable run ./uncertain.sqlite uncertain-proposal.json --approve
# Expected exit: 2; no additional target change

Three commits explain the failure windows

The implementation persists reviewed intent in one transaction. In another, it compares the expected version and writes the target together with its receipt. In a third, it marks completion in the ledger. The receipt and change are atomic with each other; the whole execution is not one transaction. That choice deliberately creates the window we just tested.

BEGIN IMMEDIATE serializes writers to the local file with a bounded contention wait. Storage uses WAL and synchronous=FULL. These choices give the exercise transactional semantics, but process-termination tests are not power-loss, disk, corruption, or network-filesystem tests. The cost of serialization matters too: one local file does not represent distributed scalability.

The suite covers other boundaries: crashes before the target commit, before the completion commit, and after completion; concurrent processes; repeated identifiers with different payloads; unreviewed proposals and state changes. The 22 passing tests check these properties in the lab. They are not an availability percentage or a production agent success rate.

C4

A person runs a recovery lab and inspects synthetic evidence. The model calls no AWS services and requires no LLM.

Containers

CLI (Node.js): Independent processes for plan, run, inspect, and recover.
Persistent store (SQLite local file): Operations, target, receipts, and history persisted in one file.

Components

Proposal validation: Validates the request, simulated review, and expected version.
Conditional mutation: Updates target and receipt in the same commit.
Recovery classifier: Reads evidence and records the classification without executing the mutation.

What changes when the tool calls AWS

In a real integration, recording intent in a database and calling a remote API are operations in different systems. The local commit does not encompass the API. The lab deliberately keeps target and receipt together so evidence is inspectable; moving the design to the cloud requires reassessing that boundary.

The AWS Builders' Library explains the value of an explicit client-provided identifier for recognizing retry intent. Check the concrete operation's idempotency contract: scope, token lifetime, behavior with changed parameters, and repeated-call responses. Do not infer a universal guarantee because another service accepts a token.

For each mutable tool, document which remote evidence is available, how long it remains queryable, and who resolves inconclusive outcomes. When the target cannot provide sufficient proof, retain a visible uncertainty state. Adding a queue, an agent framework, or an MCP server does not independently close the gap between effect and acknowledgement.

Separate current authorization from historical evidence too. In this lab, --approve is only a local review simulation. A real service must verify identity, scope, and validity before mutation; revoked approval must not become permission for a new execution just because an old record exists. A recovery read and a mutation attempt require different decisions.

Decision: uncertainty is an operational outcome

Status: accepted

Context

The client may lose the response after a committed effect at the target.

Decision

Use persistent evidence to reconcile; block resubmission when the effect cannot be attributed to the operation. In this lab, recover never mutates the target.

Consequences

Requires an owner for investigating uncertain states.
Receipt and proposal retention is part of the recovery contract.
May require intervention even when current state looks desirable.

Checklist for reviewing your own tool

Define intent: Choose a stable identifier for the same operation and an explicit rule for conflicting payloads. Save the original proposal and contract version.
Locate the real commit: Mark where the target changes and where acknowledgement is persisted. Inject an interruption between them, away from production resources.
Compare effects: Observe version, receipts, and history before and after replay. A success status alone does not demonstrate the absence of duplicates.
Exercise ambiguity: Have another actor change the target without the operation's receipt. Verify that matching values do not become assumed ownership.
Define retention and operations: Document the evidence retention window, investigation owner, and consistent database-copy procedure. Close processes before copying the lab file.

A Well-Architected reading

operational-excellence: Turn uncertain into an investigation queue with an owner and minimum evidence. Recovery needs a runbook and an exercise as well as observability.
security: Local review authenticates nobody. In production, protect the ledger, minimize sensitive data, and revalidate authorization for the next mutation.
reliability: Test lost acknowledgement, not only unavailability before a call. Separate duplicate recognition, reconciliation, and a newly authorized attempt.
performance: Measure contention and reconciliation latency in the real architecture. SQLite writer serialization is not evidence of distributed throughput.
cost: The exercises make no AWS calls. For a remote implementation, estimate evidence storage and recovery queries; do not invent savings from this local test.
sustainability: Avoiding redundant effects and attempts reduces unnecessary work, but the lab measures neither energy nor emissions. Do not turn that hypothesis into a number.

Traps the exercise makes visible

Retrying every interrupted call without consulting effect evidence.
Treating the desired final value as proof that this operation produced it.
Generating a new proposal after the change and calling it a retry of the original.
Deleting receipts or ledger records without a policy that preserves the duplicate-recognition window.
Presenting a local SQLite transaction as an exactly-once guarantee across remote services.

Diagnostic replay and mutation retry are different operations

To share a reproducible case, include the code version, synthetic initial state, proposal, interruption point, exit codes, and before-and-after inspections. A terminal screenshot helps communicate; the files and commands make verification possible. Remove credentials and real data before opening an issue.

The public evidence associated with this article contains results from both scenarios. The invitation is to discuss a concrete contract: which observation permits reconciliation, which permits another attempt, and which requires investigation? A small contribution, such as adding a failure case with an expected outcome, can be more useful than adding another tool to a catalog.

If you want to experiment, run both scenarios in the repository and describe which failure boundary you would test next. To bring this into your AWS architecture, start with one mutable operation and document its idempotency and recovery contract. The test then becomes a property the team can review, beyond a generic agent demonstration.

References

The question to bring to your review

Before authorizing another attempt, ask for evidence that distinguishes a confirmed effect, an absent effect, and an unknown effect. In this lab, a receipt allows reconciliation without another execution; its absence in the presence of drift requires stopping. The value of the exercise is making that decision reproducible and reviewable by someone else.

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Migrating agents from GPT-5.6 Terra to GPT-6.1 Terra on Bedrock

Fernando Azevedo — Tue, 27 Oct 2026 16:46:37 +0000

The GPT-6.1 Terra announcement on Amazon Bedrock, on 10/26/2026, sells the switch as obvious: 15 percentage points above GPT-5.6 Terra on OpenAI's Agents' Last Exam and token prices at least 50% lower. The question I care about is not whether the new model is better, it's what breaks in your AWS account the day you change the model ID. I read the model card line by line against the 5.6 Terra card, and the answer has names: source Region, API choice, caching and quota. This is the path I would follow to migrate a production agent without learning any of that on call.

The starting point

The scenario I use as a reference is common in financial environments: an internal agent that reads policies, queries two or three systems through tool-use and returns structured JSON to a Step Functions workflow. It runs on bedrock-runtime with the us.openai.gpt-5.6-terra profile, because the risk team does not accept global routing, and the US geographic profile, sourced from N. Virginia, stays within us-east-1, us-east-2 and us-west-2.

A typical run has about 40 thousand input tokens, 30 thousand of which are a stable prefix (instructions, tool definitions, policy excerpts), and 3 thousand output tokens. At 5.6 Terra list price on US CRIS (US$ 2.20 per million input, US$ 0.22 per million cache read and US$ 13.20 per million output), that run costs roughly US$ 0.068 in steady state. It looks small until you multiply by 200 thousand runs a month: about US$ 13.6 thousand in inference alone.

These numbers are arithmetic on published prices, not customer measurements. They exist to size the decision. What actually matters is the calendar: the 5.6 Terra model card guarantees EOL no sooner than 07/13/2027, with a legacy period of at least six months. There is time, but there is no slack for a rushed migration in the last quarter, which is exactly when the team is busy with something else.

What the model card says and the announcement doesn't

The announcement talks about capability, speed and price. The model card talks about operations, and that is where the migration risk lives. Reading both cards side by side, four differences change the design.

Region on bedrock-mantle: 5.6 Terra runs in-Region on the bedrock-mantle endpoint in us-east-1, us-east-2, us-west-2 and both GovCloud Regions. 6.1 Terra, at launch, only in us-east-1. If you point the OpenAI SDK at bedrock-mantle.us-west-2.api.aws, you don't migrate by changing a string.

Source Region on bedrock-runtime: the 6.1 Terra table does not list sa-east-1 or me-central-1 as a source for the global profile, and it does not include the in. geographic profile that 5.6 Terra offers for Mumbai and Hyderabad. If you call from São Paulo, this decides where the workload's client has to run.

Invoke API: the 6.1 Terra API table marks InvokeModel as supported on bedrock-runtime; the 5.6 Terra table marks it as unsupported. That helps teams with legacy Invoke code, but it doesn't change the Guardrails and application inference profile restrictions, which remain tied to Converse.

Service tiers: Standard only. No Priority, Flex or Reserved. If your capacity plan depended on reserving throughput, it does not exist for this model today.

None of this is in the What's New post. An announcement is a trigger; the operational contract lives in the documentation.

The journey, in six decisions

Inventory how the model is called today: Before touching code, pull four facts per application from invocation logs and CloudTrail: endpoint (bedrock-runtime or bedrock-mantle), API (Converse, Responses, Chat Completions), source Region and input size distribution. CloudTrail's additionalEventData.inferenceRegion field, logged in the source Region, shows where each request was actually processed. Count how many requests exceed 272 thousand tokens: they move to a different price band.
Prepare IAM and SCPs before the first request: The model card requires bedrock:InvokeModel on the account's default project (arn:aws:bedrock:{region}:{account-id}:project/default) in addition to the inference profile. Missing it produces an AccessDeniedException that looks like a credentials bug. For a geographic profile, the SCP must allow every destination Region in the profile; for a global profile, it must allow aws:RequestedRegion equal to unspecified, which many landing zones block by default.
Evaluate offline with real traffic: Replay 300 to 500 logged, already sanitized runs against us.openai.gpt-6.1-terra. Compare three signals per task: valid JSON rate against the schema with strict: true, correctness of tool choice and arguments, and output tokens per task. OpenAI's benchmark is a hypothesis; your workload is the proof.
Reposition the cache: On Responses, mark the end of the stable prefix with prompt_cache_breakpoint and use prompt_cache_options.mode: explicit so the automatic breakpoint doesn't consume the four writes per request. The minimum prefix is 1,024 tokens and the TTL is 30 minutes. Track cached_tokens against cache_write_tokens: writes cost 1.25x uncached input.
Canary with a written promotion criterion: Split traffic by flag (5%, 25%, 100%), with the model ID coming from configuration, not code. Write down beforehand what promotes and what rolls back: schema failures no higher than baseline, equal or lower p95 latency, cost per run and InvocationThrottles within budget. A criterion decided after the result is rationalization.
Retire the old path on purpose: Keep the 5.6 Terra profile as rollback for 30 days after reaching 100%. Then remove it from the IAM policy. A forgotten permission becomes an escape hatch: someone falls back to the old model in a hotfix and nobody notices until the invoice.

Converse or Responses: the decision that blocks the migration

On bedrock-runtime, the 6.1 Terra model card ties features to different APIs, and they don't add up. Guardrails only work through Converse. Application inference profiles, which is how you attribute cost per team with tags, are also Converse only. Explicit caching, on the other hand, only exists on Responses. On Converse you get implicit caching, which the documentation itself calls best effort.

Converse has two more restrictions: it doesn't accept document blocks, and it rejects an image inside toolResult with a ValidationException. An agent that returns a tool screenshot to the model breaks on the first call.

The cost of this choice can be calculated. In the reference scenario, the 30-thousand-token prefix costs US$ 0.033 uncached and US$ 0.00165 as a cache read, at US$ 1.10 and US$ 0.055 per million. Every miss costs 20 times more on the prefix. If implicit caching hits 60% and explicit hits 95%, the gap across 200 thousand runs a month is close to US$ 2.2 thousand.

Use Converse when AWS Guardrails and cost allocation by application profile are audit requirements, and measure the real hit rate through cacheReadInputTokens before accepting the loss. Use Responses when the prefix is large and stable and content filtering already happens in another layer you control. Running half the fleet on each API without deciding is the worst case: two audit trails, two cost models.

Canary migration backed by evidence

The diagram answers one question: where control of the model switch lives. The router reads the model ID from a flag and splits traffic between the two US CRIS profiles. Dashed arrows are asynchronous: invocation logs arrive later, and promotion is a human decision based on the harness, not an automatic trigger. IAM, SCPs and the profile's destination Regions are simplified away.

Application

Agent service ECS / Lambda (compute)
Model router AppConfig flag (compute)

Amazon Bedrock runtime

GPT-5.6 Terra us.openai.gpt-5.6-terra (ai)
GPT-6.1 Terra us.openai.gpt-6.1-terra (ai)

Evidence

Invocation logs S3 + CloudWatch (storage)
Eval harness replay + diff (ci)

Flows

agent -> router: asks for model
router -> old: 95% Converse
router -> new: 5% canary
old -> logs: logs
new -> logs: logs
logs -> eval: replays
eval -> router: promotes or rolls back

Before and after, at US CRIS list price

-50%: Input per million tokens. US$ 2.20 on 5.6 Terra to US$ 1.10 on 6.1 Terra, context up to 272 thousand tokens
-58%: Output per million tokens. US$ 13.20 to US$ 5.50; cache reads drop 75%, from US$ 0.22 to US$ 0.055
-57%: Cost per agent run. From ~US$ 0.068 to ~US$ 0.029 in the 40 thousand input (30 thousand cached) and 3 thousand output scenario, if output volume doesn't grow

Quota, long context and the cost that shows up later

The price cut is the easy part. Quota is what sinks the migration. On bedrock-runtime, limits are measured in tokens per minute with a 10x burndown on output: each output token consumes 10 from quota. Tokens read from cache don't count toward the input quota. In the reference scenario, one run consumes 10 thousand uncached input tokens plus 30 thousand output-equivalent tokens. Three quarters of the quota goes to output.

This creates a silent failure mode. If the new model answers with 20% more output tokens, because it reasons more or writes more, cost still drops substantially and no FinOps alarm fires. Quota, however, rises 15% per run, and the first signal is a ThrottlingException at the morning peak. That's why the canary watches OutputTokenCount and InvocationThrottles in CloudWatch, not just the invoice.

The second late cost is long context. 6.1 Terra accepts a 1,050,000-token context (OpenAI's page states 922 thousand maximum input), but above 272 thousand input tokens the whole request moves to another band: input at US$ 2.20 and output at US$ 8.25 per million on US CRIS. An agent that accumulates history without summarizing crosses that line around turn twenty and doubles the input cost of every call from then on. Set an explicit history ceiling, around 250 thousand tokens, and summarize above it. Having a 1-million-token context doesn't oblige anyone to use 1 million.

Each model's operational contract

Criterion	GPT-5.6 Terra	GPT-6.1 Terra	GPT-6.1 Sol
Input / output US CRIS (US$ per million)	2.20 / 13.20	1.10 / 5.50	2.20 / 11.00
Cache read (US$ per million)	0.22	0.055	0.11
In-Region on `bedrock-mantle`	us-east-1, us-east-2, us-west-2, GovCloud	us-east-1	us-east-1
`InvokeModel` on `bedrock-runtime`	No	Yes	Yes
Guardrails on `bedrock-runtime`	Converse only	Converse only	Chat Completions, InvokeModel and Converse
Tiers beyond Standard	None	None	Ultrafast (6x Standard)
Lifecycle	EOL no sooner than 07/13/2027	EOL no sooner than 10/26/2027	OpenAI terms, at least 6 months' notice

Risks I would handle before the canary: São Paulo as source: the 6.1 Terra table doesn't list sa-east-1 as a global profile source; if you call from there, move the client or wait. GovCloud and bedrock-mantle outside us-east-1: no path at launch. Longer output: with a 10x burndown, 20% more output becomes 15% more quota per run in the reference scenario, with no cost alarm. The 272-thousand-token line: cross it and the whole request pays the long-context band. Vendor benchmark: the 15 points and "three times faster" come from OpenAI's Agents' Last Exam, not from your workload.

Anti-patterns in this migration

Changing the model ID and calling it a migration: without an inventory of endpoint, API and source Region, the first failure comes from bedrock-mantle in us-west-2 or a client in sa-east-1, not from the model.
Promoting on the vendor benchmark: Agents' Last Exam doesn't know your schema or your tools; promotion without replaying real traffic is a bet.
Measuring only the invoice: cost drops even if output grows, and a quota with 10x burndown runs out before any FinOps alarm fires.
Switching to the global profile to dodge the Region table: it fixes availability and creates a data residency problem, and it requires aws:RequestedRegion unspecified in the SCP.

Curator's note: I would start this migration in November, not in June 2027. The first week would go entirely to the inventory and to the Converse versus Responses decision, because that decision defines auditing, caching and cost attribution for the next two years. After 16 years operating financial platforms, the lesson that keeps repeating in dependency swaps is this: the cost of a migration is not swapping the component, it's discovering late the contract that changed with it. With language models that doubles, because behavior changes too and no unit test catches it.

References

Verdict

Migrate, and start now. Use GPT-6.1 Terra when your workload runs on bedrock-runtime with a us. or global. profile from a source Region listed in the model card, when replaying real traffic shows equal or better schema and tool-use results, and when quota can absorb the new output under a 10x burndown. Stay on 5.6 Terra for now if you depend on bedrock-mantle outside us-east-1, on GovCloud, or on sa-east-1 as a source with residency requirements; the 07/13/2027 EOL leaves room to wait for the Region table to grow. Either way, decide Converse or Responses before the canary: that is the choice you will maintain, audit and pay for over the next two years.

Rating: Migrar com canário / Migrate with a cana

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Amazon Quick on a private network: PrivateLink's two planes and where they leak

Fernando Azevedo — Fri, 23 Oct 2026 15:48:23 +0000

When someone from compliance asks whether Amazon Quick "goes over the internet", the honest answer has two parts, and almost every design I have reviewed answers only one. User traffic into Quick uses PrivateLink interface endpoints. Quick's traffic out to your on-premises database, your private MCP server or your internal API uses something else: ENIs that Quick itself injects into your VPC. DNS, endpoint policy, IP/VPC restriction and IAM sit on top of those two planes, and each layer covers less than its name suggests. The useful question is not "is it private?", it's "which byte still leaves for the internet, and who decided that was fine?".

Why this matters now

While Quick was just BI, the VPC connection existed so Quick Sight could read PostgreSQL, SQL Server or Oracle behind a Direct Connect. The problem was well understood and the blast radius fit inside a dashboard. What changed is who uses that same VPC connection. It now carries Quick Agents calling MCP servers hosted on your network, Quick Flows and Quick Automate invoking action connectors, and Quick Research and Apps in Quick pulling from private sources. The ENI that used to run a SELECT for a chart now carries an agent's tool call.

In financial environments this matters in two places. Regulation: CMN Resolution 4.893 and PCI-DSS expect you to show the data path and control who enters the perimeter. The AWS post lists HIPAA, FedRAMP and PCI-DSS as the usual triggers. Audit: "traffic stays on the AWS backbone" only works as evidence if you can say which domains still leave through NAT, and some do.

The 21 Oct 2026 post is a good walkthrough. What it doesn't do, and isn't meant to, is say where each control stops working. That's what this piece covers: the documented limits of each layer, and the cost of keeping all of it working for years, not the cost of building it once.

The two planes of private Quick access, and what stays public

Ingress (user to Quick) over interface endpoints. Egress (Quick to your data) over VPC connection ENIs. Dashed lines are traffic no endpoint covers.

🏢 On-premises: rede corporativa

DNS corporativo forward de .quicksight.aws.amazon.com (network)
Oracle / SQL Server porta do banco liberada só para o SG do Quick (data)
Servidor MCP privado API interna por IP (compute)

📡 Rede: Direct Connect ou Site-to-Site VPN

Direct Connect rota para o CIDR da VPC (network)

🟧 AWS: VPC do cliente (2 AZs no mínimo)

Route 53 Resolver inbound 2+ ENIs (network)
VPCE quicksight-website policy só com aws:PrincipalAccount / aws:OrgId (security)
VPCE quicksight (API) policy por ação; 13 APIs fora (security)
ENIs da VPC connection SG QuickSight-VPC, VPC ID imutável (network)
VPCE secretsmanager / redshift / glue S3 por gateway (security)

🤖 AWS: Amazon Quick (Enterprise)

Quick Sight, Agents, Flows, Automate Identity Center, RLS/CLS (ai)

🌐 Externo: saída pública que continua obrigatória

*.cloudfront.net, *.execute-api, *.kinesisvideo sign-in do Identity Center (external)

Flows

user -> dns: resolves the site name
dns -> resolver: forwards the private zone
user -> dx: HTTPS 443
dx -> vpce-web: console and dashboards
dx -> vpce-api: API calls
vpce-web -> quick: PrivateLink (ingress)
vpce-api -> quick: PrivateLink (ingress)
quick -> eni: assumes execution role, uses ENIs
eni -> dx: data and tool-call egress
dx -> db: SQL query
dx -> mcp: action connector / MCP
eni -> sm: credentials without leaving the VPC
user -> public: assets, metrics, sign-in via NAT/proxy

Ingress plane: interface endpoints, DNS and what the policy accepts

Ingress uses two interface endpoints with different jobs: com.amazonaws.<region>.quicksight-website for the site and com.amazonaws.<region>.quicksight for the API (with a -fips variant). If administrators sign in as IAM users, add console and signin too. Place each one in at least two AZs, with a security group that accepts only 443 from the corporate CIDR.

DNS is where the first thing breaks. The API endpoint, with private DNS on, takes over quicksight.<region>.amazonaws.com by itself. The website endpoint doesn't: its private name is not the public URL. You create a private hosted zone for <region>.quicksight.aws.amazon.com with an alias A record pointing at the endpoint. But a private hosted zone only answers associated VPCs, so corporate DNS has to forward that zone to a Route 53 Resolver inbound endpoint. Without the forward, the browser resolves the public IP, access still works, and nobody notices PrivateLink is being bypassed. Test with nslookup from inside the corporate network and expect an IP from the VPC CIDR.

The website endpoint policy is almost decorative. Principal, Action and Resource must be "*", and the condition only accepts aws:PrincipalAccount or aws:OrgId. It stops your network from becoming a doorway into another account's Quick, which helps against exfiltration, but it doesn't control what the user does. The API endpoint accepts action-level restriction, and that's where being specific pays off.

Egress plane: the VPC connection is an injected ENI, not PrivateLink

When Quick needs to fetch data, it doesn't go through the interface endpoint. To create a VPC connection (Enterprise only), you supply a VPC ID, at least two subnets in different AZs, at least one security group, and an execution role that trusts quicksight.amazonaws.com. Quick assumes that role and creates ENIs in your subnets with ec2:CreateNetworkInterface. From then on your network decides: route table, NACL, security group and the Direct Connect route.

Three details turn into incidents if you don't know them. The VPC ID can't change after creation: reorganizing the network means recreating the connection and repointing every data source. The PARTIALLY AVAILABLE status means one ENI is down and the connection still works. It's exactly the state nobody watches until the second AZ fails. Connections created before 27 Apr 2023 use a non-stateful security group and need an inbound rule that lets return traffic in on every port (0–65535) from the source's SG. If yours dates from then, recreate it.

For on-premises database host names, enter the Resolver inbound IPs in the DNS resolver endpoints field. Without them Quick resolves through public DNS and can't find the host. Credentials go to Secrets Manager through an interface endpoint in the same VPC as the ENIs, so that even the password lookup doesn't leave through NAT.

One limitation changes the design: the Quick Index knowledge base VPC connection currently supports only Amazon Redshift. Confluence, SharePoint, Box, Google Drive, OneDrive and S3 can't use it.

The ENI security group is now the agent's network allow-list: The same VPC connection serves Quick Sight and Quick Agents calling private MCP. In practice, the egress of the QuickSight-VPC SG and the routes on those subnets define everything an agent can reach on your network. If the SG opens the whole on-premises CIDR because "BI needed it", the agent inherits that reach. Split subnets and security groups by purpose (analytical data on one side, action tools on the other), even if it costs a second VPC connection.

Enforcing endpoint-only access: where the restriction stops

Quick's IP/VPC restriction (update-ip-restriction --enabled --vpc-endpoint-id-restriction-rule-map vpce-...=Name) blocks anything that doesn't come from the approved endpoint. It has four properties you need to know before turning it on in production.

It covers web, embedded and mobile, but not the public API. Anyone with credentials can call any operation from any IP. For the API the control is IAM: an aws:SourceVpce or aws:SourceIp condition in a deny policy, ideally in an SCP. Here's the catch: 13 APIs don't work through the interface endpoint (the 8 action connector ones, such as CreateActionConnector, and 5 Flows ones, such as ListFlows). A blanket aws:SourceVpce deny breaks the automation that manages connectors. You write the exception with NotAction for those operations and run that automation from a controlled path with public egress.

UpdateIpRestriction replaces the whole map. A script that sends only the new rule deletes the others. Run describe-ip-restriction first, or better, keep the map versioned and always apply the full state.

A rule takes up to 10 minutes to apply and is global, enforced in every Region and managed only from the account's Region. The limit is 100 rules combined, with IP and VPC rules OR'ed together.

Lockout is real. Without a rule covering your own access, the console won't let you save. But anyone on a source outside the list loses access the moment you turn it on. Publicly embedded dashboards then work only through the endpoint.

What stays public, and what keeping the rest private costs

The documentation lists the domains the browser reaches, and several have no endpoint: *.cloudfront.net (CSS and JS), *.execute-api.<region>.amazonaws.com (client metrics), *.kinesisvideo.amazonaws.com (automation streaming), <region>.signin.aws and, with IAM Identity Center, browser sign-in, which requires the public sign-in and Identity Center endpoints. No public IP is needed, but NAT or a proxy is. Your compliance report should say "the data and API path is private; static assets, telemetry and authentication leave through an inspected proxy". Writing "zero internet" claims something the documentation itself contradicts.

Now the recurring cost, in us-east-1. An interface endpoint costs US$ 0.01 per hour per ENI, plus US$ 0.01 per GB for the first 1 PB. Website and API in 2 AZs are 4 ENIs, about US$ 29/month. Add console and signin, another US$ 29. Secrets Manager, Redshift and Glue in 2 AZs, roughly US$ 44 more. The most expensive piece is the one nobody budgets: a Resolver inbound endpoint costs US$ 0.125 per ENI per hour, with at least two, so about US$ 182/month, plus US$ 0.40 per million queries. If you already have a shared Resolver inbound in a central network VPC, reuse it: it's the biggest line item and there's nothing Quick-specific about it.

Anti-patterns I find in this design

Website endpoint without on-premises DNS forwarding: PrivateLink exists, the browser resolves the public IP and access works from outside. It's a silent failure that only shows up in nslookup or VPC Flow Logs.
Treating IP/VPC restriction as an API control: it doesn't cover the public API. Without aws:SourceVpce in IAM or an SCP, a leaked credential calls everything from anywhere.
Blanket aws:SourceVpce deny with no exception: the 13 action connector and Flows APIs don't go through the endpoint, and the automation that manages connectors stops working without an obvious error.
A script that calls UpdateIpRestriction with only the new rule: the map is replaced whole, the other rules disappear and an entire branch office loses access within 10 minutes.
One VPC connection for BI and agents with the whole on-premises CIDR open: the agent inherits BI's network reach, and the blast radius of a prompt injection becomes the corporate network.
Keeping a VPC connection from before 27 Apr 2023: the non-stateful SG requires return traffic open on 0–65535, a rule no auditor accepts without questions.

Well-Architected lens

security: Layered defense with clear roles: website endpoint policy scoped by aws:OrgId, API endpoint policy by action, IP/VPC restriction for web and mobile, an SCP with aws:SourceVpce and NotAction for the 13 APIs outside the endpoint, RLS and CLS on the data. Restriction map changes land in CloudTrail; alarm on UpdateIpRestriction.
reliability: Two AZs on every endpoint and on the VPC connection, an alarm whenever status leaves AVAILABLE (including PARTIALLY AVAILABLE), and Direct Connect with a backup VPN, because the VPC connection has no other path to on-premises.

What I would do: I'd start with the report I'll hand the auditor, and only then create the first endpoint: which domains leave through the proxy, which controls cover the API and which cover the browser. Then I'd build two security groups and two VPC connections, one for analytical data and one for agent tools, because I've seen a "temporary" BI security group last four years. The restriction map would go into code from day one, applied as full state. The lesson I carry from financial platforms: the expensive control isn't the one you configure, it's the one you think is active and isn't, and in Quick that control is DNS forwarding.

Verdict

Use the full design (website and API endpoints, VPC connection, Resolver inbound, IP/VPC restriction and an SCP with aws:SourceVpce) when you have regulated on-premises data, agents calling private tools, and an auditor who will ask for the path of every byte. If Quick only reads Redshift and S3 in the same account and sign-in goes through Identity Center, a VPC connection with service endpoints covers most of the risk, and IP restriction on the corporate egress covers the browser for well under US$ 300/month. Either way, document what stays public. That sentence is what separates a defensible design from a promise the documentation contradicts.

Rating: Recomendado com ressalvas / Recommended

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

After login: per-operation authorization, revocation and recovery for agent tools

Fernando Azevedo — Wed, 21 Oct 2026 16:01:25 +0000

A valid login is only the starting point

An agent may continue a conversation after a user loses a permission. The operational question is whether the next tool call can still change a resource. This article proposes an architecture exercise, not a report of an AWS deployment or a completed cloud experiment. The AWS scenarios below are test proposals; the local recovery lab has a narrower purpose.

The starting points are Carlos Biagolini's articles on AgentCore Identity and a serverless MCP server with IAM authentication. They motivate a useful follow-up: once identity is established, which component enforces the right to perform each operation, and what happens if execution is interrupted?

Separate three decisions

Authentication establishes the caller's identity. Authorization decides whether that caller may perform a particular operation on a particular resource and tenant. Approval records consent to a concrete proposed change. A valid token, a visible tool or an old approval does not answer all three questions.

Imagine a remediation tool shared by two tenants. A caller may have permission to invoke the endpoint and still lack permission to change the selected resource. Similarly, approval of an earlier proposal should not authorize a modified proposal. These checks must remain explicit even when the model produces a convincing explanation.

A route is a boundary, not the whole policy

API Gateway HTTP APIs support IAM, JWT authorizers and Lambda authorizers. This exercise compares IAM-based invocation with application-defined Lambda authorization. With AWS_IAM, a signed request and invocation permission control entry. A Lambda authorizer applies application logic: format 1.0 requires an IAM policy; format 2.0 also supports a simple Boolean response when enabled.

If several MCP tools share POST /mcp, the tool name and arguments live in the body. Permission to invoke that route does not establish permission for every operation carried by it. In this design, the executor enforces fine-grained authorization. Another trusted layer could do so if it understands the operation, cannot be bypassed and must run before the effect. Separate routes help entry controls but do not remove resource and tenant checks.

Do not accept a model-supplied tenant as proof of authority. In a JWT flow, use validated claims and their authorized relationship to the resource. In an IAM/SigV4 flow, use the authenticated principal and a service-controlled authorization mapping. Carry that trusted context to the execution boundary. A Lambda role's ability to access DynamoDB does not automatically prove that every caller may read every item the function can access.

Revocation has an observable window

IAM documentation explains that temporary-credential permissions are evaluated for AWS requests, while policy updates may take minutes to propagate. Per-request evaluation therefore does not imply instantaneous revocation.

A cached Lambda-authorizer result introduces another window. Include the cache key, route separation and TTL in the test design. Denying future work does not reverse a completed effect. Sensitive operations also need an explicit policy for in-flight work and a check near the effect boundary.

Approval and completion are different facts

Bind approval to reviewer identity, proposal digest, scope, expiry and expected resource state. After a worker restart, a pending record does not prove whether an effect happened. Look for durable target evidence before replaying. If evidence is insufficient, preserve uncertainty and request review instead of guessing.

The public v0.4.0 lab explores that recovery boundary with SQLite and a synthetic target. Its reproduction guide requires Node.js 22.13+ and creates local files. No AWS account is needed. CLI approval is simulated; the lab does not implement authenticated approval, IAM or revocation. Atomic local target-and-receipt writes do not establish exactly-once effects across a remote API.

In the documented exercise, a deliberate crash occurs after the synthetic target and receipt are committed but before the execution ledger acknowledges completion. Recovery uses that receipt to reconcile the operation. Keeping the original proposal matters: generating a new proposal after the target changes produces a different digest. This is a concrete way to discuss recovery without implying a cloud guarantee the exercise cannot demonstrate.

Proposed test matrix

Valid principal requests another tenant: Deny before protected data access. Evidence: Synthetic identity and decision.
Shared MCP endpoint receives an unauthorized tool: Deny at the operation boundary. Evidence: Tool name, scope and decision.
Permission changes during a session: Measure when new calls stop succeeding. Evidence: Change time and decision timeline.
Authorizer result is cached: Test before and after TTL across routes and tenants. Evidence: Cache settings and observations.
Proposal changes after approval: Reject the changed digest. Evidence: Original and changed synthetic digests.
Resource changes before execution: Reject the stale precondition. Evidence: Expected and observed version.
Process stops after effect, before acknowledgement: Reconcile using evidence. Evidence: Ledger, target version and receipt.

For every scenario, distinguish expected from observed results. These AWS authorization tests are proposed, not completed. Use synthetic inputs and avoid storing tokens, secrets or customer data.

Trade-offs and the next experiment

Fresh authorization checks add work and latency. Caching can reduce both while extending the stale-decision window. Choose the design using operation impact, revocation policy and measurements. A low-risk read and a destructive remediation may justify different controls.

Before adding another agent, ask the team to demonstrate which identity drives the decision, which component enforces it and how interrupted work is recovered. A future isolated experiment can add authenticated approval and revocation, with a cost ceiling and cleanup procedure defined before deployment.

What would you test first: cross-tenant access, revocation during a session, or recovery after the effect? A useful answer includes the evidence needed to prove the outcome.

References

Authorization boundaries

Proposed design; identity and policy must reach the effect boundary.

Trusted application boundary

Authenticated caller (user)
Route entry control (edge)
Tool contract (compute)
Operation + resource + tenant (security)
Proposal approval (security)
Constrained executor (compute)
Durable evidence (storage)

Flows

caller -> entry
entry -> contract
contract -> policy
policy -> approval
approval -> executor
executor -> receipt

C4

A user requests remediation through an agent; the model proposes, the application enforces boundaries.

Containers

API / MCP endpoint (API Gateway + application): Authenticates the caller and controls entry.
Tool executor (Application service): Checks operation, resource, tenant and approval before the effect.
Evidence store (SQLite in local lab): Preserves synthetic proposal, state and receipt for recovery.

Components

Policy check: Uses trusted context independent of model arguments.
Recovery reconciler: Reconciles using evidence; preserves uncertainty without proof.

AWS Well-Architected

security: Deny out-of-scope operations before effects; do not trust model-supplied tenant identity.
reliability: Recover using receipts and preconditions without promising remote exactly-once effects.
operational-excellence: Record expected and observed results with versions and synthetic evidence.
cost: Run the local lab first; define a ceiling and cleanup before an AWS experiment.

Checklist before automation

Bind identity, digest, scope and expiry to approval.: Document the evidence, owner and known limitation.
Test the wrong tenant before data access.: Document the evidence, owner and known limitation.
Measure propagation and caching during revocation.: Document the evidence, owner and known limitation.
Simulate a crash after the effect and inspect the receipt.: Document the evidence, owner and known limitation.

Anti-patterns

Treating authentication or old approval as universal authorization.
Repeating an effect merely because the ledger still says pending.
Promising instant revocation or remote exactly-once execution without measurement.

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Retro: EMR Serverless on IPv6-only and the subnet left without DNS64

Fernando Azevedo — Tue, 20 Oct 2026 15:42:40 +0000

The dangerous question about IPv6 on EMR Serverless is not "do my jobs run on IPv6?", it is "what does each subnet in the application know how to do when a job asks for an address that only exists in IPv4?". The 10/16/2026 announcement is short: Spark and Hive can now run on dual-stack or IPv6-only subnets, at no additional cost, in every Region and release version of the service. The documentation is more honest than the announcement: the addressing mode is derived from the subnets, and getting IPv6 traffic to your data is your responsibility. This piece is a pre-emptive retro, an incident that did not happen, built from the documentation and the failure modes it lists itself. I would rather write the postmortem before than after.

The scenario: /22 subnets that no longer fit three applications

The starting point is what AWS itself uses as a planning example. Each EMR Serverless worker consumes an IP address in the subnet where it launches, through an ENI. An application with 4 vCPU workers and a 4,000 vCPU ceiling reaches 1,000 workers, meaning 1,000 ENIs. A /22 subnet has 1,024 addresses and 1,019 usable ones, because AWS reserves five. With three applications sharing the same subnets and a 12,000 vCPU account quota, the documentation's recommendation becomes 3,000 free addresses per subnet, in at least two Availability Zones.

In a bank's data platform this usually hits a wall: the VPC IPv4 block was negotiated with corporate networking years ago, does not overlap with on-premises and does not grow without an approval round. That is exactly the case IPv6 solves. A VPC IPv6 subnet is a /64, and the address conversation is over.

In the scenario, the team goes straight to IPv6-only on three subnets, one per AZ, created by an IaC module. Two were built by hand during the pilot and got DNS64. The third came from the module, which did not know DNS64 existed. No integration test caught it, because the test job ran twice and landed on the two good subnets both times.

The numbers that drive the decision

1.000: ENIs per application. 4 vCPU workers with a 4,000 vCPU ceiling: one IP per worker
1.019: Usable IPs in a /22. Barely fits one application; three need 3,000 per subnet
US$ 0,045/GB: NAT gateway processing. Pricing page example (US East Ohio): every byte through NAT64 pays

Timeline of the simulated incident

D-7: subnet swap: The team stops the application, runs update-application with the three IPv6-only subnets in --network-configuration and starts it again. The mode is evaluated at every start, so the change only applies after the restart. Two test jobs pass.
02:00: the nightly batch starts: About a third of the jobs fail with UnknownHostException when calling endpoints that only publish an A record. Each job runs in a single subnet picked from the set, and the ones that landed on the subnet without DNS64 cannot resolve anything that is not IPv6.
02:40: retries hide the pattern: The orchestrator's retry policy resubmits the jobs. Some pass because they land on another subnet. The failure looks intermittent, and intermittent becomes "flaky network" in the on-call's first hypothesis.
04:10: correlation by subnet: Someone opens each job run detail, which shows the subnet used, and crosses it with success and failure. 100% of the failures sit on the same subnet. modify-subnet-attribute --enable-dns64 fixes name resolution.
05:00: the second failure: An ingestion job with an old JDBC driver fails with UnresolvedAddressException. It carried -Djava.net.preferIPv4Stack=true in spark.driver.extraJavaOptions, a workaround from years ago. The documentation forbids forcing the IPv4 stack in any mode.
D+5: the bill arrives: Cost Explorer shows a jump in NAT gateway data processing. S3 reads, which used to leave through the gateway endpoint, now resolve through DNS64 to 64:ff9b::/96 and cross NAT64.

Root cause: EMR Serverless derives the addressing mode from the subnets and does not validate whether the network is ready for it. Each subnet's readiness (IPv6 CIDR, auto-assign, DNS64, a 64:ff9b::/96 route to the NAT gateway, a ::/0 route to the egress-only gateway) was not encoded in the IaC module, and one-subnet-per-job selection turned a deterministic defect into an apparently random failure.

Why the failure looked random: the addressing mode rules

It is worth understanding the rules before blaming the network. An IPv4-only subnet runs as it always did. A dual-stack subnet runs IPv4 until you turn on spark.emr-serverless.networking.ipv6.enabled=true in the application's spark-defaults (for Hive, hive.emr-serverless.networking.ipv6.enabled in hive-site). An IPv6-only subnet runs IPv6 with no property at all.

When the set mixes types, the whole application runs in the one mode every subnet supports. IPv4-only with IPv6-only is rejected at validation. IPv6-only with dual-stack runs everything IPv6-only, including the dual-stack subnets, which must then meet the same prerequisites. IPv4-only with dual-stack runs everything on IPv4.

Three details produce incidents:

The mode applies at start: a running application keeps the mode it started with. You fix the subnet and nothing changes until the next stop/start.
One job, one subnet: a job runs in a single subnet picked from the list. One bad subnet out of three fails a third of the time, and retries mask it.
Dual-stack does not prefer IPv6 by default: even with the opt-in on, the engine uses IPv4 until you set spark.emr-serverless.driverEnv.SPARK_PREFER_IPV6 and spark.executorEnv.SPARK_PREFER_IPV6. Setting those on an IPv4-only subnet kills the job.

The lesson: the addressing mode is derived state, not configured state. The API does not let you declare the mode; it lets you declare subnets and infers the rest. Derived state needs a test that proves the derivation on every subnet, not on average.

Where each byte of a job on an IPv6-only subnet leaves

The path depends on the destination's DNS record: AAAA leaves directly, A becomes 64:ff9b::/96 and pays NAT64.

🟧 AWS: subnet IPv6-only (uma por AZ)

Spark driver SPARK_PREFER_IPV6 automático (compute)
Executors 1 ENI IPv6 por worker (compute)

📡 VPC: resolução e rotas

Route 53 Resolver DNS64 na subnet (network)
NAT gateway rota 64:ff9b::/96 (NAT64) (network)
Egress-only IGW rota ::/0 (network)
Interface endpoints Glue, STS, KMS, Logs (security)

🌐 Destinos

S3 dual-stack s3.dualstack. (storage)
Endpoint só IPv4 JDBC legado, API pública (external)
Endpoint IPv6 AAAA publicado (external)

Flows

drv -> r53: DNS query
exe -> r53: DNS query
r53 -> nat: A record only: synthesizes 64:ff9b::
nat -> v4: translates to IPv4, billed per GB
exe -> eigw: destination with AAAA
eigw -> s3ds: data reads and writes
eigw -> v6: egress without ingress
drv -> vpce: catalog, credentials, keys

Network remediation: turning prerequisites into invariants

The DNS64 fix took one command. The remediation that prevents recurrence took more work, and it is the one that matters.

Subnet module with a contract: the IaC module now requires, for every subnet tagged as an EMR Serverless target, assign_ipv6_address_on_creation = true, enable_dns64 = true, a 64:ff9b::/96 route to the NAT gateway and a ::/0 route to the egress-only internet gateway. Auto-assign only applies to resources created after the change, so turning it on late does not fix an ENI that already exists.

Validation before apply: a pipeline check (policy as code, or a test that reads describe-subnets and describe-route-tables) fails the deploy if any subnet in the set breaks the contract. Validation that runs after the incident is a postmortem, not a control.

Per-subnet canary: since the job picks the subnet and you do not, I create a canary application with a single subnet, one per AZ. A minimal Spark job resolves and connects to the endpoints the batch depends on (catalog, STS, KMS, the data bucket, the source database) and fails loudly. It runs before the batch window and after any network change.

Clean JVM: a sweep across every sparkSubmitParameters and runtimeConfiguration looking for preferIPv4Stack. A 2019 workaround cannot decide the network stack in 2026.

None of this is sophisticated. Its cost is not writing it, it is keeping the contract when someone creates the fourth subnet eighteen months from now without reading this document.

The cost the announcement does not mention: NAT64 and S3

"No additional cost" is true for the feature. It is not true for the path the traffic now takes.

On an IPv6-only subnet, every destination that only publishes an A record gets a synthesized address in 64:ff9b::/96, and that route points to the NAT gateway. The NAT gateway bills per hour and per GB processed; the VPC pricing page uses US$ 0.045 per GB as its example in US East (Ohio). A platform reading 8 TB a night from S3 and sending it through NAT64 pays around US$ 360 a night in processing alone, roughly US$ 10,800 over 30 nights. The S3 gateway endpoint on the workload subnet does not catch this traffic, because the destination is no longer an IPv4 prefix.

S3 only serves IPv6 through its dual-stack endpoints (s3.dualstack.<region>.amazonaws.com), and the client has to ask for them explicitly. Do not trust it before measuring: compare the NAT gateway's BytesOutToDestination before and after the switch.

Other services need the same care. In the AWS IPv6 support table, Glue has dual-stack and a private endpoint with IPv6, but not the public endpoint; Lake Formation also only has IPv6 through its private endpoint. On IPv6-only, that means interface endpoints for catalog and permissions, or paid NAT64 on every call.

And security: the S3 documentation warns that policies with IPv4-only aws:SourceIp make clients incorrectly lose or gain access once they start using IPv6. Review every bucket policy before the switch, not after.

The three modes side by side

Criterion	IPv4-only	Dual-stack with opt-in	IPv6-only
IPv4 consumption	1 per worker	1 per worker, still consumed	Zero
Activation	-	`ipv6.enabled` on the application + `SPARK_PREFER_IPV6` on the job	Automatic from the subnets
IPv4-only destinations	Direct	Direct over IPv4	Only with DNS64 + NAT64, billed per GB
Incident risk	Exhausting the subnet at peak	Low: failure falls back to IPv4	High if any subnet or dependency is not ready
When to use	Address headroom and no IPv6 mandate	Transition and dependency validation	Compliance mandate or real exhaustion, with dependencies mapped

Well-Architected reading

security: An IPv6 address is globally unique: egress only through an egress-only gateway, never a ::/0 route to the internet gateway on worker subnets. EMR Serverless already refuses security groups with ports open to 0.0.0.0/0 or ::/0. Review aws:SourceIp in bucket policies and prefer aws:SourceVpce.
reliability: Spread subnets across at least two AZs, as the documentation recommends, but treat each one as its own failure unit: a canary per subnet and an alarm per subnet. Automatic retries without per-subnet correlation turn a deterministic defect into noise.

Anti-patterns this retro tries to prevent

Validating with two jobs: with three subnets and one picked per job, two green jobs prove nothing about the third.
Going straight to IPv6-only without a dependency inventory: every driver, SDK and endpoint the job touches must support IPv6 or go through NAT64; EMR Serverless does not change your code or third-party endpoints.
Turning on SPARK_PREFER_IPV6 as a global default: on an IPv4-only subnet the preference kills the job.
Fixing the subnet without restarting the application: the mode was fixed at start, and on-call concludes the fix did not work.

What I would do: I would start with dual-stack with the opt-in, on a new application, and keep the IPv6 preference on only for jobs whose dependencies I have already measured. IPv6-only waits for a mandate or real exhaustion, and only after two weeks of green per-subnet canaries and a NAT gateway bytes chart I can explain line by line. The lesson I took from other network migrations is that the defect is almost never in the new service, it is in the subnet someone created with the old module. That is why the contract lives in IaC, not in the wiki.

Lessons and recommendation

Adopt IPv6 on EMR Serverless when the IPv4 space in shared subnets already limits your vCPU ceiling or when there is a formal IPv6 requirement. Use dual-stack with the opt-in while there is any IPv4-only dependency that matters in volume; use IPv6-only when every dependency publishes AAAA or sits behind an interface endpoint, and NAT64 is limited to control calls. Either way, encode DNS64, routes and auto-assign in the subnet module, run a per-subnet canary and measure the NAT gateway before declaring the migration done. The feature is free; an unprepared network bills per GB.

Rating: Adote com dual-stack primeiro / Adopt wi

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

CloudFront dual-stack VPC origins: IPv6 to private origins, with caveats

Fernando Azevedo — Fri, 16 Oct 2026 17:37:34 +0000

For almost two years, anyone who adopted CloudFront VPC origins and had an IPv6 target lived with a bad choice: keep the ALB in a private subnet and stay on IPv4 between edge and origin, or expose the origin to the internet to get end-to-end IPv6. The 10/15/2026 announcement removes that dilemma. The question that matters now is not 'can I get IPv6 to a private origin?', it's 'what in my VPC, my security group and my change process breaks when I turn it on?'. That is the question this analysis answers.

The launch in numbers

US$ 0: Additional cost. Dual-stack VPC origins carry no extra charge; the ENI addresses can be private.
34: Commercial Regions. Every Region with VPC origins, including sa-east-1; four have one excluded AZ.
55/60: Prefix list weight. The CloudFront managed list weighs 55 against the default 60 rules per security group.

What actually changed

VPC origins, launched in November 2024, places an ALB, NLB or EC2 instance in a private subnet and makes CloudFront the single point of entry. The connection arrives through a service-managed ENI created inside your subnet, and access is controlled by a security group AWS creates with the naming pattern CloudFront-VPCOrigins-Service-SG. Until yesterday that ENI spoke IPv4 only.

In September 2025 CloudFront gained IPv6 for custom origins, with three options: IPv4 only (default), IPv6 only and dual-stack. That announcement excluded S3 and VPC origins. What shipped now closes half of that gap: VPC origins accept dual-stack. The other half stays open, because the current CloudFront IPv6 documentation still lists IPv6 only as "excluding Amazon S3 and VPC origins".

In practice, the change is one field: when you create the VPC origin, the console asks for Origin type, and the default is still IPv4. For dual-stack to work, the private subnet needs at least one free IPv6 address in addition to the IPv4 one, because that is where the service ENI is born. Without it, ENI creation fails.

What does not change: gRPC is still out, Lambda@Edge origin request and origin response triggers are still out, a Gateway Load Balancer cannot be an origin, and neither can an NLB with a TLS listener. If you ruled out VPC origins for one of those reasons, you have no new reason to revisit.

End-to-end IPv6 path with a private origin

Every hop where IPv6 can silently fall back to IPv4, and the control that governs that hop.

🌐 DNS e borda

Route 53 alias A + alias AAAA (network)
CloudFront edge IPv6 de viewer ligado (edge)
Origin-facing escolhe IPv4 ou IPv6 (edge)

🟧 AWS: VPC privada (sa-east-1)

ENI do serviço 1 IPv4 + 1 IPv6 livres (network)
ALB interno IpAddressType: dualstack (compute)
Target group IPv4 ou IPv6 (compute)

🔐 Controles

CloudFront-VPCOrigins-Service-SG fonte da regra de entrada (security)
NACL de saída efêmeras 1024–65535 IPv4 e IPv6 (security)

Flows

viewer -> r53: AAAA lookup
viewer -> edge: HTTPS over IPv6
edge -> ofs: cache miss
ofs -> eni: private connection, family picked by the service
eni -> alb: allowed by service SG
alb -> tg: new connection, target group family
svcsg -> alb: inbound rule
nacl -> alb: return path

Where it shines

The trade-off is gone: IPv6 to the origin no longer requires a public subnet, an internet-facing ALB and an IP list to maintain.
Zero license cost: no additional charge, and the ENI IPv6 addresses do not need to be public.
Same security model: the service SG still restricts the origin to your distributions, now across both families.
Brazil included: sa-east-1 is on the list, so a regulated workload that must stay in-country is not left out.

Dual-stack is not guaranteed IPv6

This is the point the announcement does not make explicit, and it matters to whoever has to answer the audit. The CloudFront documentation describes dual-stack mode this way: the service automatically picks IPv4 or IPv6 for the origin connection to prioritize performance and availability. In other words, you don't decide the family of each connection. CloudFront does.

For most sites that is exactly the right behavior. For a team that has to prove to a regulator or a contract that traffic runs over IPv6 all the way to the origin, it is an evidence problem. Turning on dual-stack produces no proof at all; measurement does.

The same holds on the viewer side. The documentation itself warns that CloudFront may answer a viewer over IPv4 when its data says the experience will be better, and that the IPv6 share should remain a minority because not every access network supports the protocol. Two automatic decisions, one at each end, and neither is yours.

What I would do: treat "end-to-end IPv6" as a measured SLO, not a setting. At the edge, the c-ip field in standard logs shows the viewer family. At the origin, ALB access logs show the address that opened the connection, which for VPC origins is the service ENI. The daily ratio of IPv6 connections to total is the number that goes into the report. If the requirement is IPv6-only, VPC origins still doesn't meet it: IPv6 only remains excluded.

Network prerequisites that fail silently

The CloudFront side is one field. The real work is in the VPC, and much of it fails without a clear message.

IPv6 in the VPC and subnets: the ALB only accepts IpAddressType: dualstack if the VPC and its subnets have an associated IPv6 CIDR block, the route tables route IPv6, and the SG and NACL allow IPv6. In an older account built IPv4-only, that is a project, not a click.

An address for the ENI: the subnet where the VPC origin lives needs a free IPv6 address. A /28 that runs out of IPv4 is already a known problem; with dual-stack the same math repeats in the other family.

Origin security group: the documentation gives two options. Managed prefix lists, and for dual-stack that means both, com.amazonaws.global.cloudfront.origin-facing and com.amazonaws.global.ipv6.cloudfront.origin-facing. The VPC documentation assigns the CloudFront list a weight of 55, against the default of 60 rules per security group, which leaves 5 rules for everything else. The other option is to reference CloudFront-VPCOrigins-Service-SG: it takes one rule and restricts the origin to your distributions, not to any CloudFront distribution. That is the one I use.

Outbound NACL: inbound NACL rules are not evaluated for traffic from CloudFront, but outbound rules are, on the return path, on ephemeral ports 1024–65535. A NACL keeps IPv6 rules separate from IPv4 ones. If you only allowed 0.0.0.0/0, you will watch the IPv6 handshake die on the way back.

Limits that still apply: IPv6 only is still unavailable for VPC origins. gRPC and Lambda@Edge origin triggers are not supported. A Gateway Load Balancer and an NLB with a TLS listener cannot be origins, and an NLB must have a security group attached. And IPv6 ends at the ALB: the connection to targets uses the target group's family, so a dual-stack ALB with an IPv4 target group is IPv6 to the ALB, not to the application.

Migrating a VPC origin already in production

Here is the cost the announcement does not show. Per the documented procedure, updating a VPC origin requires first making sure it is not the default origin of any cache behavior, then disassociating it from every distribution that uses it, editing, waiting for the status to return to Deployed (up to 15 minutes), and only then associating it again. In production, that is a scheduled outage window, multiplied by the number of distributions sharing the origin.

I would not edit the existing VPC origin. The safer path is to create a second, dual-stack VPC origin pointing at the same ALB ARN, and swap origins through CloudFront continuous deployment: a staging distribution gets the new origin, you test with a slice of traffic, and you promote the configuration when the signals are clean. Rollback is promoting the previous configuration back, without waiting 15 minutes for a VPC origin to propagate.

The signals I would watch on staging: the distribution's 5xxErrorRate and OriginLatency in CloudWatch, the ALB's TargetConnectionErrorCount and HTTPCode_ELB_5XX_Count, and the share of IPv6 connections in the ALB access logs. If that share stays at zero, the cause is almost always an SG or NACL missing the IPv6 rule, and CloudFront falls back to IPv4 without complaining, which is good for availability and terrible for the target.

If you share VPC origins across accounts through AWS RAM, add coordination with the owner of every consuming distribution.

How to adopt without an outage window

Prepare the VPC: Associate an IPv6 CIDR with the VPC and private subnets, adjust route tables and confirm each subnet has a free IPv6 address for the ENI.
Move the ALB to dual-stack: Run aws elbv2 set-ip-address-type --ip-address-type dualstack on the internal ALB; decide separately whether the target group also moves to IPv6.
Lock down SG and NACL: ALB inbound from CloudFront-VPCOrigins-Service-SG; NACL outbound on ephemeral 1024–65535 for both IPv4 and IPv6.
Create a new VPC origin: Origin type dual-stack, same ALB ARN. Don't edit the existing one; wait for Deployed status.
Test on staging and promote: Staging distribution with the new origin, partial traffic, 5xx signals and IPv6 share; promote only when both are clean.
Finish the edge: Enable viewer IPv6 and add a second AAAA alias in Route 53 for each alternate domain name; review signed URLs using a custom IpAddress policy.

Anti-patterns

Editing the VPC origin in production: the procedure requires disassociating from every distribution, and the wait of up to 15 minutes becomes an outage.
Claiming compliance because the field says dual-stack: CloudFront picks the family per connection; without log measurement, there is no evidence.
Allowing the prefix lists when the origin only serves your own CloudFront: it consumes 55 of the default 60 rules and accepts any distribution, not just yours.
Forgetting the target group: a dual-stack ALB with IPv4 targets delivers IPv6 to the load balancer, not to the application.

Curator's note: In a financial environment I would turn this on first for a low-risk distribution, with staging and an IPv6 share target written down before the change. I have seen an IP family migration pass functional testing and fail the first audit report, because the fallback to IPv4 is silent by design. The lesson: in networking, whatever degrades without an error needs its own metric, otherwise you find out from the wrong person. And I would not edit an existing VPC origin under any circumstances; creating another one and swapping via continuous deployment costs an afternoon and saves a 2 AM window.

Verdict

Adopt it when you have an end-to-end IPv6 requirement, contractual, regulatory or platform-driven, and the origin already is or needs to be in a private subnet: there is no additional cost and the security model does not get worse. Wait if the requirement is IPv6-only, because IPv6 only is still out for VPC origins, or if the VPC has no IPv6 CIDR yet, because then the project is a network project, not a CDN one. Either way, create a new VPC origin and swap via staging; do not edit the one serving traffic.

Rating: 4/5

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Well-Architected Agent: architecture review becomes a weekly process

Fernando Azevedo — Tue, 13 Oct 2026 18:24:40 +0000

For years, the Well-Architected review was an event: a week-long workshop, a spreadsheet of high and medium risks, and then six months with nobody opening the report. On 10/01/2026 AWS put AWS Well-Architected Agent into preview, which replaces the event with a cycle: scheduled scans, up to 30 recommendations per run, weekly refresh. The question that interests me is not "is the agent right?", it's "who triages 30 findings a week, and what happens to what it cannot see?". After 16 years operating financial platforms on AWS, I've learned that a findings tool without a triage owner becomes just another red dashboard everyone learns to ignore.

The numbers that shape the design

100: accounts per profile. scanning every commercial Region from us-east-1, us-east-2 or us-west-2
30: recommendations per run. with a 7-day cooldown between scheduled runs of the same profile
5/dia: IaC reviews per profile. zip up to 25 MB; S3 folder up to 100 MB, file up to 1 MB

The signal: an aggregator with an opinion

Strip the AI layer and what AWS shipped is an aggregator with an opinion. The related-services documentation is explicit: the agent ingests Trusted Advisor findings as baseline signals, uses Compute Optimizer rightsizing signals for EC2, Lambda and RDS, consumes Security Hub CSPM, Resilience Hub assessment data (with its RTO/RPO) and Cost Optimization Hub. On top of that it adds three things: prioritization against goals you write, cross-pillar trade-off analysis, and ready remediation (console, updated IaC, aws CLI commands and Systems Manager runbooks).

Recommendations come at three levels. Resource: one instance, one function, one database, with dollar impact where applicable. Application: consolidated findings across related resources, a format the documentation itself marks as beta. Architecture: generated from CloudFormation, Terraform or CDK templates sent as a zip or S3 URI, before deployment.

Two operational details change result quality. Without Cost Explorer enabled, the agent estimates cost from public pricing, which ignores Savings Plans, Reserved Instances and contractual discounts; in an account with an EDP, the "dollar impact" can be inflated enough to invert prioritization. And access requires a Business+ Support plan or higher: Business+ gets 2 profiles and 7 applications per profile; Enterprise On-Ramp, Enterprise and Unified Operations get 10 and 30. Developer and Business are excluded.

What changes for architects

Goals became configuration: up to 10 goals per profile, plain text, one pillar each, and they order the queue. A badly written goal mis-prioritizes without raising any error.
Continuous review has fixed throughput: 30 recommendations per run and a 7-day cooldown mean a triage ceiling per profile. The bottleneck becomes the team, not the tool.
Shift-left with a quota: 5 IaC reviews per day per profile cannot back a gate on every pull request. They fit on merge to the release branch.
Criticality is declared, not inferred: application context accepts MISSION_CRITICAL, BUSINESS_CRITICAL, NON_CRITICAL and TEST_DEVELOPMENT. Whoever skips it gets recommendations tuned for the average case.
Silence is not compliance: the unsupported type list includes AWS::WAFv2::WebACL, AWS::Organizations::Policy and AWS::IAM::RolePolicy. No finding there proves nothing.

The blind spot that matters in a bank

The quotas page carries a long table of CloudFormation resource types the agent does not analyze. Most are peripheral (IoT Analytics, DeviceFarm, Evidently). But a few are exactly the controls an auditor checks first in an environment under BACEN 4.893 or PCI-DSS:

AWS::WAFv2::WebACL, WebACLAssociation, IPSet and RuleGroup: the entire WAF edge.
AWS::Organizations::Policy: SCPs, that is, the organization's data perimeter.
AWS::IAM::RolePolicy, UserPolicy and GroupPolicy: inline policies, where the "Action": "*" nobody remembers writing usually lives.
AWS::OpenSearchService::Domain, AWS::Shield::Protection and Security Hub's own configuration.

The failure mode here is silent. The report arrives clean on the security pillar, someone takes the screenshot to the risk committee, and the absence of WAF findings is read as a well-configured WAF. It isn't: it's an unanalyzed WAF.

The mitigation is cheap and has to be explicit. Keep those controls covered by AWS Config conformance packs and Security Hub CSPM directly, outside the agent, and write in the governance document which types the agent does not cover and which tool covers each one. The list is marked preview and should shrink, so review it every quarter instead of assuming it is stable.

Where the agent reads, where it writes

Role chain in a dedicated account, input signals and remediation paths. Only the IaC path goes back through the pipeline.

🔐 AWS: Conta de tooling dedicada

Agent profile us-east-1 · 10 metas (ai)
Execution role trust wellarchitected.amazonaws.com (security)
CloudTrail API wellarchitected:* (security)

🟧 AWS: Contas de workload (até 100)

Access role WellArchitectedAgentResourceScanning (security)
Recursos todas as regiões comerciais (compute)

📡 AWS: Sinais de entrada

Trusted Advisor Security Hub CSPM (data)
Compute Optimizer Cost Optimization Hub (data)
Cost Explorer sem ele: preço público (data)

📤 Saída: Remediação

IaC atualizado PR no repositório (ci)
AWS CLI / SSM runbook só em sandbox (external)

Flows

profile -> exec: assumes
exec -> access: sts:AssumeRole, ExternalId = profile ARN
access -> res: config read
ta -> profile: baseline findings
co -> profile: rightsizing and cost
ce -> profile: actual cost per resource
profile -> ct: audit
profile -> owner: 30 recommendations/week
owner -> iac: accepts and versions
owner -> cli: tests outside prod

IAM: the design is good, the default less so

The access model deserves credit. Instead of a service-linked role, the agent uses customer-managed roles in a chain: an execution role in the profile account, trusting wellarchitected.amazonaws.com and holding only sts:AssumeRole, and an access role in each workload account, with the read-only WellArchitectedAgentResourceScanning managed policy. The profile ARN is used as the external ID, which closes the confused deputy, and there is an iam:PassRole check at creation. The documentation states neither role allows modifying resources. Onboarding is opt-in per account.

That said, three adjustments I'd make before turning it on in production:

Dedicated account: AWS itself recommends hosting the profile in an account with no workloads. In a Control Tower environment, that is the tooling or audit account, never the management account.

Trust by exact ARN: use the execution role ARN, with the service-role/ path the console creates, not the account root. Roll out access roles via StackSets with the same name in every account (the common-name pattern the profile accepts) so that revoking is one operation, not a hundred.

Processing residency: the profile can only live in us-east-1, us-east-2 or us-west-2, even though it scans sa-east-1. Configuration metadata and tags leave the Region for analysis. If your tagging policy carries customer names or CPF numbers in tags (I've seen it), that becomes a conversation with the DPO under LGPD before it becomes one with the auditor.

Four remediation paths, four maintenance costs

Criterion	Console	Updated IaC	AWS CLI	SSM runbook
Creates drift	Yes	No	Yes	Yes, unless the runbook updates the code
Review trail	CloudTrail only	Pull request with approval	CloudTrail only	SSM execution history
Where I use it	Understanding the finding	Production	Sandbox	Repeatable action with no IaC state, e.g. orphaned snapshot

How I'd position it today

The agent doesn't replace a human Well-Architected review, it changes what the human review has to cover. The mechanical work (oversized instance, bucket without versioning, single-AZ RDS tagged MISSION_CRITICAL) leaves the workshop agenda. What remains for people is what the agent admittedly cannot see: SCPs, WAF, inline policies, and architecture decisions that show up in no configuration at all, like why a settlement flow is synchronous.

In practice, three design decisions:

One profile per regulatory domain, not per team: with 10 profiles on Enterprise, separate what has distinct SLOs and regimes (payments, data, internal platform). On Business+, with 2 profiles and 7 applications, the choice is harder: regulated production in one, everything else in the other.

Goals versioned in a repository: plain-text goals are a ranking parameter. Treat them as code: a file in the platform repository, applied through the wellarchitected namespace API, with review. A goal like "reduce cost" is weak; "reduce non-production compute cost by 20% without touching the core's 15-minute RTO" gives the agent a trade-off to respect.

A queue with an owner and an SLA: 30 recommendations a week is about 120 a month per profile. Without an owner who marks items complete or suppresses them with a reason, the backlog grows linearly and the tool loses credibility in two months. I'd turn on cost and resilience first, where the cost of error is low, and security only later, with the unsupported list beside it.

Anti-patterns you can already predict

Running the CLI command in production: it fixes the finding and creates drift; the next terraform apply reverts the fix and nobody notices until the next scan.
Reading a clean security pillar as attestation: with WAF, SCPs and inline policies out of scope, a clean report is partial coverage, not compliance.
Prioritizing by dollars without Cost Explorer: the agent falls back to public pricing and ignores Savings Plans and EDP; the cost queue ends up sorted by a number that isn't yours.
Profile in the management account: it concentrates the role that reaches 100 accounts in the place with the largest blast radius in the organization.
IaC gate on every PR: 5 reviews per day per profile run out before lunch; the gate either fails open or blocks the team.

The goal is the new prompt: The goals text defines the order of the remediation queue across 100 accounts. That is production configuration with effects on money and risk, written in natural language with no validation. Whoever edits goals needs the same change control as whoever edits SCPs.

Curator's note: I'd turn the agent on this week in a dedicated tooling account, cost and resilience only, with Cost Explorer enabled before the first run. Access roles go out through StackSets, trust pinned to the exact execution role ARN, and goals live in a versioned file. The lesson I carry from years of Trusted Advisor and Security Hub: the cost of a findings tool is not the license, it's the weekly triage someone has to sustain for years. If there is no named owner for the queue, I'd rather not turn it on.

Verdict

Adopt now if you already have a Business+ plan or higher, Control Tower with a tooling account, and a platform team able to triage about 120 items a month per profile: the agent removes the mechanical work from reviews and delivers IaC remediation that fits your pipeline. Wait if your main need is security evidence for regulatory audit: while WAF, SCPs and inline policies are out of scope, it complements AWS Config and Security Hub CSPM, it does not replace them. Either way, production remediation goes only through the IaC path.

Rating: Adotar com escopo / Adopt with scope

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Sharing a GPU cluster on HyperPod: three designs and when each one pays off

Fernando Azevedo — Sat, 10 Oct 2026 16:19:29 +0000

The question that usually opens the shared GPU cluster discussion is "how do I split the machines across teams?". The question that decides whether the design survives is a different one: when the research team's training job gets evicted at 2 AM to make room for an inference endpoint, who finds out, how many GPU-hours were thrown away, and which cost center absorbs them? The reference architecture AWS published on 10/08/2026 for SageMaker HyperPod on EKS answers a good part of that. Here I put it side by side with the two alternatives I see most often and say under which conditions each one is worth its maintenance cost.

The real choice: three ways to split GPUs

Option A, one shared HyperPod EKS cluster with Task Governance: the design in the post. Each team gets a hyperpod-ns-<team> namespace, its own SageMaker domain for Studio users, and a permission set in IAM Identity Center for CLI users. Fairness across teams belongs to Task Governance, built on Kueue: guaranteed quota per team, lending of idle capacity, and priority classes.

Option B, one cluster per team, ideally in separate accounts: the boundary is the AWS account. There is no noisy neighbor because there is no neighbor. In exchange, each team carries its own capacity floor, and GPUs idle in one cluster cannot rescue a full queue in another.

Option C, a shared cluster with namespaces and ResourceQuota only: this is what shows up when the platform starts as a single-team cluster and other teams trickle in. It works until the first week of contention, when the quota becomes a hard ceiling and arrival order decides who gets to train.

The difference between the three is not the cost of standing up the cluster. It is what each one demands in continuous operation: reviewing quotas, debugging Kueue admission, maintaining the UID/GID webhook, reconciling chargeback. That is the cost that stays for years, and it is the one I compare on.

From login to GPU: where each boundary in design A is enforced

Identity decides the namespace, the namespace decides the queue, the queue decides whether the pod runs on its own or borrowed quota, and the namespace becomes the chargeback key.

🔐 IAM Identity Center: identidade

Permission set AWSReservedSSO_TeamA_* (security)
SageMaker domain TeamA-domain + TeamA-role (security)

🟧 AWS: HyperPod EKS control plane

EKS access entries escopo: hyperpod-ns-team-a (security)
Admission webhook UID/GID via sessionName (compute)
DynamoDB mapa POSIX (data)

🤖 Task Governance: Kueue

LocalQueue hyperpod-ns-team-a-localqueue (messaging)
ClusterQueue cota garantida + borrow limit (messaging)
Idle resource sharing IdleResourceSharing: Enabled (messaging)

🟧 AWS: pool de GPU compartilhado

Nós HyperPod health checks + auto-recovery (compute)
FSx for Lustre /fsx/TeamA (storage)

📤 Saída: custo e visibilidade

Kubecost custo por namespace (data)

Flows

idp -> ps: federates groups
user -> ps: aws sso login
user -> dom: Studio
ps -> ae: kubectl
dom -> ae: execution role
ae -> wh: pod created
wh -> ddb: resolves UID/GID
wh -> lq: queue-name label
lq -> cq: admission
cq -> idle: over quota: borrows
cq -> gpu: admits pod
gpu -> fsx: checkpoint
gpu -> cost: usage per namespace

Isolation: a namespace is an organizational boundary, not a security one

The post itself states it plainly: a namespace is an isolation boundary, not a hard security boundary, and the design targets several teams within one company, not tenants who do not trust each other. I agree, and I add what that means in a regulated environment.

Design A stacks four controls: EKS access entries scoped per namespace (one for the domain role, another for the AWSReservedSSO_<permission-set>_<id> role), RBAC that returns Forbidden on cross-namespace calls, a default-deny NetworkPolicy per namespace, and S3 restricted to the team's prefix. All four run on the same kernel, the same NVIDIA drivers and the same control plane.

Silent failure number one: a NetworkPolicy without a CNI that enforces it is just YAML. If the Amazon VPC CNI does not have network policy enabled, kubectl apply succeeds, the object exists, and not a single packet is blocked. Test with a team B pod trying to open a connection to a team A service before you call the boundary done.

My rule is this. If the teams share the same regulatory regime and the same data classification, a namespace is enough. If one team trains on customer data under LGPD and another on synthetic data, or if a BACEN 4.893 auditor is going to ask for demonstrable segregation, the right boundary is the account, and the design goes back to B.

Fairness: quota, borrowing, and who gets evicted

Task Governance splits two policies. Compute prioritization decides how idle capacity is handed out (first-come first-serve or fair-share) and how tasks are queued (arrival order or task ranking with priority classes). Compute allocation sets, per team, the guaranteed quota, a fair-share weight from 0 to 100, and the lending posture: Lend and Borrow, Lend, or Don't Lend. The borrow limit can be a percentage, up to 10,000% of quota, or absolute per instance type.

The documentation example shows the mechanics: team A with a quota of 6 using 2, team B with a quota of 5 using 4, a 4-unit job lands on team B and 3 units are borrowed from A. What the documentation also says, and what defines operations, is that a task running on borrowed quota can be preempted by a higher-priority task when capacity runs out.

Silent failure number two: a job without kueue.x-k8s.io/priority-class gets the lowest priority. An inference endpoint deployed without the label becomes the first eviction candidate in the cluster.

Preemption has a price in GPU-hours. A 16-GPU training run on borrowed capacity with a checkpoint every 30 minutes loses up to 8 GPU-hours per eviction, 4 on average. If the cluster evicts that job twice a day, that is roughly 240 GPU-hours a month thrown away without tripping a single alert. That is why I treat borrowed capacity like spot: only jobs with checkpoints on FSx and idempotent resume run on it.

Chargeback: who pays for idle and borrowed GPUs

The post uses Kubecost grouped by namespace, and since the namespace is already the team, no extra workload tagging is needed. That works for the question "how much did team A consume". On its own it does not answer two questions that show up at the first month-end close.

The borrowed GPU: when team B runs on team A's quota, the pod lives in B's namespace. Charging B is correct, but team A will look at the report and see it paid for a guaranteed quota someone else used. Decide upfront whether the guaranteed quota is a fixed price (team A pays for the reservation, used or not) or usage-based. Both rules are defensible; changing the rule mid-quarter is not.

The idle GPU: capacity not allocated to any quota needs an owner in the report. Split Cost Allocation Data for EKS, which started covering GPU, Trainium and Inferentia on 09/02/2025, splits cost per pod and reports unused capacity separately, with a 9:1 GPU versus CPU and memory weighting in the documentation example. That example is an EC2 instance. HyperPod nodes are ml.* instances billed through SageMaker, so confirm on your bill whether that data shows up before you swap Kubecost for CUR. I would keep Kubecost as the chargeback source and use CUR to reconcile the total.

The three designs side by side

Criterion	A: shared HyperPod + Task Governance	B: one cluster per team	C: namespaces + ResourceQuota
Isolation boundary	Namespace + access entry + NetworkPolicy	AWS account	Namespace + RBAC
One team's idle GPU	Lent, with limits and preemption	Stays idle	Stays idle (quota is a hard cap)
Contention	Fair-share by 0–100 weight + task ranking	None across teams	Arrival order
Chargeback	Per namespace, needs a borrowing rule	Account bill, trivial	Per namespace
Parts to maintain	Quota policies, UID/GID webhook, Kubecost, dashboards	N clusters, N upgrades, N pipelines	Few, until the first dispute
When it fails badly	Job without priority-class or checkpoint	Low utilization becomes fixed cost	Large team monopolizes the cluster

Decision matrix

A: shared HyperPod + Task Governance

Pros

One team's idle GPU becomes another's capacity, with IdleResourceSharing covering even unallocated capacity
Explicit priority: production inference preempts experiments
One driver and EKS upgrade for every team

Cons

A namespace will not pass a strong-segregation audit
The UID/GID webhook and its DynamoDB map become yours to run
Preemption wastes GPU-hours when jobs do not checkpoint

Verdict: Default for teams in one company under one regime

B: one cluster per team

Pros

Segregation provable by account, IAM and bill
A bad upgrade's blast radius stays within one team

Cons

Each team pays its own GPU floor, even when idle
Platform effort multiplied by N

Verdict: When data or regulation differ across teams

C: namespaces + ResourceQuota

Pros

Nothing new to learn

Cons

No borrowing, no cross-team priority, no fair-share
GPU disputes become meetings, not policy

Verdict: Only as a step until the second team arrives

Guaranteed quota is not guaranteed capacity: The documentation says to keep the sum of quotas below cluster capacity: 20 instances, quotas summing to under 20. But HyperPod pulls unhealthy nodes out of rotation, and idle resource sharing already excludes NotReady or unschedulable nodes from its math. Team quotas do not shrink with them. In a 16-node cluster with 2 nodes recovering, quotas summing to 16 promise capacity that does not exist at that moment. I keep at least one node per instance type outside any quota.

What I would do: I would start with design A, but with three rules written down before the first job: any manifest without kueue.x-k8s.io/priority-class is rejected by an admission policy, every job running on borrowed quota writes checkpoints to FSx at a known interval, and the billing rule for borrowed quota is signed off by the cost center owners. I would also make the UID/GID webhook's failurePolicy an explicit decision: Fail so the cluster stops scheduling when DynamoDB does not answer, Ignore only if you accept pods with the wrong POSIX identity writing to /fsx. I have seen shared platforms die not from a technical limit, but because the first unexplained eviction made a team ask for its own cluster. The lesson: in a shared cluster, predictability is worth more than utilization.

Verdict

Use Option A (shared HyperPod with Task Governance) when the teams belong to the same company, handle data with the same classification, and someone owns the quota policies and Kueue observability (kubectl describe workload has to become routine, not archaeology). Use Option B (cluster per account) when one team operates under LGPD with customer data and another does not, or when the audit requires segregation by account; accept paying for idle GPUs as the price of isolation. Treat Option C as a transitional state: as soon as a second team competes for GPUs, move to A. The gain in design A is not having a single cluster, it is lending idle GPUs without negotiating in a meeting, and it only pays off if evictions are predictable and chargeback is agreed upfront.

Rating: A, com regras de admissão e checkpoint /

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Critical or non-critical: the failure contract of ECS daemons

Fernando Azevedo — Thu, 08 Oct 2026 18:00:58 +0000

Every container cluster carries a layer of software nobody asked for and everybody needs: the log shipper, the EDR agent, the metrics exporter, the mesh proxy. For years that layer was treated as a packaging detail, one more sidecar in the task definition, a DaemonSet on Kubernetes. The launch of non-critical daemons in Amazon ECS Managed Daemons, on September 3, 2026, exposes the question that was hiding underneath the packaging: when that agent dies, what should happen to the machine hosting it? The answer is now a parameter (critical), and parameters get versioned, reviewed, and gotten wrong.

The problem: cross-cutting agents are coupling disguised as convenience

The sidecar-per-task model solves distribution and fails at economics and governance. An instance running 40 tasks runs 40 copies of the log shipper; at 128 MiB each, that is 5 GiB of RAM spent duplicating one function. Worse than the waste is the drift: the agent version becomes the property of each application team, and the security team finds out during an audit that three services still run a seven-month-old image.

ECS already had the DAEMON scheduling strategy for the EC2 launch type, but that is placement only, one daemon per instance, with no guarantee that it is up before the first application task serves traffic. For telemetry that is a hole in the graph. For a runtime security agent, it is a window in which the instance processes transactions without coverage.

Managed Daemons, GA on April 1, 2026 for ECS Managed Instances capacity providers, closed that hole by inverting the relationship: the instance only transitions the application task to RUNNING after the daemon is running. The price of that guarantee was rigid coupling, any daemon failure drained and replaced the instance. Correct for a compliance agent. Absurd for a metrics exporter. The September launch does not add a new capability; it undoes a coupling that should never have been the only option.

Anatomy: three parts, two guarantees, one switch

The pattern has three distinct resources. First, a daemon task definition: its own resource type with its own ARN (daemon-task-definition/...), separate from your application task definition. Second, a daemon, created with CreateDaemon, binding that definition to a cluster and to one or more ECS Managed Instances capacity providers. Third, the EC2 instances provisioned by those capacity providers, on each of which ECS keeps exactly one daemon task.

The first guarantee is ordering: ECS provisions the instance, starts the daemon task, and only then promotes the application task to RUNNING. Independent measurements published in April 2026 showed the daemon reaching RUNNING 18 to 21 seconds ahead of the application, with the instance held in REGISTERING during that window. That ordering applies to every daemon, critical or not. This is the point I most often see misread: marking a daemon non-critical does not cost you the guarantee that the agent comes up before traffic; it only costs you the guarantee that it stays up.

The second guarantee is remediation, and that is what the critical switch governs. Note how ECS remediates: it does not restart the container, it drains and replaces the whole instance. That is deliberately blunt, an agent that died may have left eBPF probes detached, network rules half-applied, buffers lost. Replacing the instance is the only cheap way back to a known state.

Instance lifecycle and the criticality branch

The startup ordering (left) is identical in both modes. The critical parameter only decides what happens after the daemon fails, and the non-critical branch is only safe if the observability arm is wired up.

🟦 AWS: Provisioning & startup order

Managed Instances capacity provider (compute)
EC2 instance state: REGISTERING (compute)
Daemon task exactly 1 per instance (compute)

🔀 Failure contract, critical parameter

critical = true | false per-daemon, default true (security)

🟥 critical=true, fail closed

Registration blocked instance never goes ACTIVE (security)
DRAINING + replace ~5 min per wave, 25% default (compute)

🟩 critical=false, fail open

Instance stays ACTIVE running tasks untouched (compute)
Placement continues registration never blocked (compute)
Coverage gap silent unless observed (security)

🟧 Observability, the other half of the pattern

EventBridge Daemon service action (messaging)
Service action log critical + non-critical (data)
CloudWatch alarm + bake time rollback (data)
Coverage SLO % instances with healthy daemon (data)

Flows

svc -> cp: 1. pending task
cp -> ec2: 2. provisions EC2
ec2 -> daemon: 3. daemon first (~18-21s)
daemon -> gate: 4. failed / stopped / unhealthy
gate -> block: fails to start
gate -> drain: fails after ACTIVE
gate -> active: critical=false
active -> place: new tasks land
place -> gap: no agent coverage
gate -> eb: event in both modes
gate -> log: service action log
eb -> alarm: rule → SNS/Lambda
alarm -> drain: circuit breaker rollback
eb -> slo: feeds the coverage error budget
gap -> slo: only visible if measured

What actually differs between the two modes

Criterion	Critical (default)	Non-critical (critical=false)
Daemon fails to start	Instance never completes registration; no application task is placed on it	Registration is never blocked; application tasks launch immediately
Daemon stops or goes unhealthy later	ECS drains and replaces the whole instance (auto-repair)	Instance stays ACTIVE; existing tasks are untouched
Placement of new tasks	Stops on the affected instance	Continues normally, agent or no agent
Startup ordering	Daemon before the application	Daemon before the application, identical
Signal emitted on failure	EventBridge event + service action log	EventBridge event + service action log, identical
Residual risk you take on	Fleet churn: an unstable agent turns into cascading instance replacement	Blind spot: coverage degrades silently if nobody consumes the event

When to use each mode: the rule I apply

The rule fits in one line: mark as critical the agent whose absence changes what the system is allowed to do; mark as non-critical the agent whose absence only changes what you can see. A runtime EDR, a network policy enforcer, an mTLS sidecar, or a file-integrity agent required by PCI DSS fall in the first group, without them the instance should not be eligible to process a transaction. An OTel metrics collector, a node exporter, a continuous profiler fall in the second.

The nuance lives in the log router. In financial environments the log often is the audit trail: "what you can see" becomes "what you can prove", and Fluent Bit stops being auxiliary. Do not classify by tool category; classify by the regulatory obligation that stream underwrites. If the agent's local buffer survives the outage window without dropping events, non-critical is defensible; if it drops them, it is critical.

The design corollary is what pays off most in practice: because criticality is per daemon, and a capacity provider accepts multiple daemons, do not package agents with different contracts into the same task definition. A task definition bundling the EDR and the metrics collector forces the stricter contract on both, and turns a collector memory leak into fleet replacement. Two daemons, two release cycles, two owners, two criticalities.

"Non-critical" is a control-plane term, not a business term. The parameter describes what ECS does to the instance, not how much that agent matters to you.

What you pay for classifying wrong, in both directions

Too critical. A bad agent revision makes the task flap, and every flap becomes an instance replacement. On a 60-instance fleet with the drain percentage at its default of 25%, that is 15 simultaneous replacements per wave; at the measured ~4m50s per wave, a one-line agent change costs roughly 20 minutes of fleet rotation. The direct bill impact is modest and computable: the Managed Instances management fee for a c6a.2xlarge is US$0.037/hour per instance (Oregon), so 60 instances add up to ~US$1,620/month in fee alone, plus EC2 and the overlap during draining. The expensive bill is elsewhere: application task rescheduling, drained connections, cold caches, cold JIT.

Two defaults conspire here. Bake time defaults to 0 minutes, meaning that once the last wave finishes, the window in which the circuit breaker watches alarms to roll back is nil. A regression that only shows up under load sails right through.

Too non-critical. Fail-open emits no pain: the instance stays ACTIVE, the application SLO does not move, and agent coverage decays in silence. Add the documented operational trap: daemon configuration settings are not persisted between updates, and critical defaults to true. An UpdateDaemon fired by a pipeline without an explicit --no-critical re-arms fleet-wide churn that nobody asked for. It is exactly the kind of change that only surfaces at the next incident.

Anti-patterns this launch makes easy

Marking everything non-critical to "stop the churn", you did not remove the failure, you just muted the alarm it used to pull.
Flipping to non-critical without first wiring the EventBridge rule for daemon events to a destination that wakes somebody up. Fail-open with no event consumer is a compliance blind spot that looks like stability.
Packaging the security agent and the telemetry agent in the same daemon task definition, forcing a single failure contract onto software with opposite risk profiles.
Managing daemons via ad-hoc CLI calls: since UpdateDaemon does not persist settings and critical reverts to the true default, every partial call is a silent contract change.
Keeping bake time at 0 while configuring alarms, paying the cognitive cost of the circuit breaker without buying the observation window that makes it useful.
Treating the critical daemon as a substitute for application health checking. It repairs agent coverage, not business health; a healthy agent on a disk-full instance is still "healthy".

A Well-Architected reading of the decision

security: Fail-closed remains the correct default for enforcement agents: a critical daemon keeps the instance from registering without coverage. Govern who may loosen it with IAM on ecs:CreateDaemon/ecs:UpdateDaemon, scoped by daemon ARN and conditioned on tags, so the security team owns the security daemon and the platform team owns the rest.
reliability: Non-critical removes a real correlated failure mode: an agent regression hitting every instance at once used to produce simultaneous fleet replacement. In exchange, agent recovery stops being automatic, plan explicit repair.

Reference configuration I would take to production: Two daemons per capacity provider: sec-runtime with critical: true and obs-otel with critical: false. Drain percentage at 10 (not 25) so a bad wave hits 10% of the fleet, and bake time between 15 and 30 minutes with at least two CloudWatch alarms wired to the circuit breaker, one on application errors, one on the agent's own health. One EventBridge rule capturing daemon service action and daemon deployment state change events, targeting SNS for the critical daemon and a queue/Lambda feeding the coverage SLO for the non-critical one. Every daemon declared in full in IaC, never by partial CLI call, with drift detection, because UpdateDaemon reverts whatever you omit. And check the instance profile: it needs AmazonECSInstanceRolePolicyForManagedInstances; the legacy AmazonEC2ContainerServiceforEC2Role policy stops the daemon from starting.

Curator's note: I have watched a memory leak in an observability agent turn into node rotation during a settlement window, and the post-mortem did not name the leak as root cause, it named the fact that nobody had ever written down which agents were allowed to die. Since then I require an explicit criticality field in the ADR for any agent before it lands on the platform, with the owner and the regulatory obligation it underwrites next to it. This launch finally gives me a place in the control plane to materialize that decision instead of leaving it implicit in a runbook. What I would not do is use critical: false as a painkiller: if an agent fails often enough to hurt, the agent is the problem, and marking it non-critical merely trades a noisy incident for a silent gap. My default stays critical, and every exception needs a coverage SLO to get approved.

Verdict

Adopt it, and treat critical as an architectural decision, not a tuning flag. Keep the critical default for enforcement agents (EDR, network policy, mTLS, file integrity) and for log routers that underwrite an audit trail. Move to non-critical only metrics, profiling and telemetry whose temporary loss is not a loss of evidence, and only after the EventBridge rule has a consumer and the coverage SLO is published. Split agents with different contracts into different daemons, drop the drain percentage to 10, raise bake time to 15–30 minutes, and declare everything in IaC, because UpdateDaemon silently reverts omissions to critical: true. If your cluster is not on Managed Instances yet, this launch does not change your roadmap; if it is, it closes the last reasonable excuse for keeping an agent sidecar in every task definition.

Rating: Adopt, with the switch written down

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

AMIs with a contract: anatomy of EC2's new instance-type block

Fernando Azevedo — Thu, 08 Oct 2026 18:00:24 +0000

Um AMI sempre foi um contrato verbal. O artefato sabe exatamente de que hardware ele precisa, driver ENA, NVMe, arquitetura do binário, extensão de CPU que a JVM compilou, e nada disso estava escrito em lugar nenhum que o RunInstances pudesse ler. A partir de 4 de setembro de 2026 dá para escrever: o dono do AMI declara SupportedInstanceTypes, UnsupportedInstanceTypes ou os dois, e o EC2 bloqueia o lançamento fora da lista. Sem custo adicional, em todas as regiões. A leitura fácil é "menos launch quebrado". A leitura que interessa é outra, o ponto de enforcement mudou de lugar, e com ele mudou quem paga a conta da manutenção.

The problem: the contract existed, it just wasn't executable

Every platform team that publishes golden AMIs has lived this scene. The hardened image was built for x86_64, with a statically compiled EDR agent and a kernel that only loads its network driver on a Nitro host. Someone creates an Auto Scaling group with attribute-based instance type selection asking for 4 vCPUs and 16 GiB, leaves Spot diversification wide, which is the correct recommendation for filling capacity, and EC2 picks a Graviton family because it matches the attributes and is the cheapest right now.

The instance comes up. That detail is what makes the failure mode hurt. It doesn't die at RunInstances; it dies later, with no console, no network, nothing but a health check running out. The ASG waits the default 300-second grace period, marks the instance unhealthy, terminates it and tries again, and since the allocation decision hasn't changed, it picks the same pool. You get a launch/terminate loop billed at a 60-second minimum per instance created, an alarm firing at 2 AM, and a graph that looks exactly like a capacity shortage.

Every defense available lived on the caller side: an ec2:InstanceType condition in an IAM policy, an SCP at the Organization, ExcludedInstanceTypes inside a fleet's InstanceRequirements. All correct, all sharing one structural defect, they describe what that principal may launch, not what that image can survive. Share the AMI with 40 accounts and you replicate the rule 40 times, in 40 places that drift apart.

The contract travels with the artifact, and fails at the gate, not at the health check

Reference design: the pipeline derives the list from a capability query, writes the specification onto the AMI, and RunInstances becomes the single evaluation point for every account that received the image.

🏗️ Pipeline de imagem, build time

EC2 Image Builder pipeline do golden AMI (ci)
DescribeInstanceTypes filtro por arquitetura + hypervisor (compute)
ReplaceImageInstanceTypeSpecification lista explícita, gerada (ci)

🟧 AWS, plano de controle do EC2

AMI InstanceTypeSpecification (storage)
CopyImage / share especificação preservada (storage)
RunInstances avalia tipo × contrato (security)

🚀 Consumidores, launch time

Auto Scaling group ABIS + Spot diversificado (compute)
Launch template versão do AMI fixada (compute)
Instância lançada tipo dentro do contrato (compute)
InvalidParameterCombination lançamento bloqueado (external)

🔍 Sinais, o que você tem que instrumentar

CloudTrail errorCode em RunInstances (data)
Scaling activity falha registrada no ASG (messaging)
CloudWatch metric filter por imageId (data)

Flows

plat -> builder: publishes the golden AMI
builder -> capq: asks what the hardware does
capq -> setspec: compatible types, enumerated
setspec -> ami: writes both lists
ami -> copy: cross-region / cross-account
ami -> gate: contract read at launch
asg -> lt: scales out
lt -> gate: RunInstances
gate -> ok: type allowed
gate -> blocked: type off the list
blocked -> ct: auditable error
blocked -> sa: capacity not delivered
ct -> alarm: metric filter
sa -> alarm

Anatomy: two lists, one evaluation order, zero state

The attribute is InstanceTypeSpecification and it carries SupportedInstanceTypes and UnsupportedInstanceTypes. The logic is short enough to fit in an ADR: with no specification, everything passes (that's the default, and that's why nothing broke on the 4th); with only the unsupported list, everything passes except what matches; with the supported list set, the type must match the supported list and must not match the unsupported one. Denial always wins.

You write it with aws ec2 replace-image-instance-type-specification --image-id ami-... --instance-type-specification '{"SupportedInstanceTypes": ["t3.*","a2.*"], "UnsupportedInstanceTypes": ["t3.micro"]}'. Omitting the parameter removes the restriction. You read it back through describe-images, which returns the InstanceTypeSpecification field with the patterns as stored. A blocked launch returns InvalidParameterCombination from RunInstances, with a message pointing at DescribeImages as the place to look.

Four properties change the design, and none of them are in the announcement:

Replace, not Modify: the operation swaps the whole specification. There is no add or remove, it's read-modify-write with no conditional version. Two pipelines patching the same image in parallel produce last-writer-wins silently, and the loser only finds out when a legitimate launch gets blocked.

Only the owner writes. An account that received the shared AMI consumes the contract and does not negotiate it.

CopyImage preserves the specification. That's what makes the pattern worth it in a multi-Region topology: you declare once and the sa-east-1 replica is born with the contract.

Marketplace is mutually exclusive. You can't set a specification on an AMI listed in AWS Marketplace, and you can't list an AMI that already has one. If you're an ISV shipping an appliance, this feature does not exist for you.

Three ways to constrain instance type, and what each one actually covers

Criterion	IAM / SCP `ec2:InstanceType` condition	`ExcludedInstanceTypes` in ABIS	`InstanceTypeSpecification` on the AMI
Where the rule lives	In the principal's policy, per account or OU	In the fleet or ASG configuration	On the artifact, once, in the owning account
Applies to launches from another account	No, that policy is theirs, not yours	No, the consumer configures the fleet	Yes: EC2 evaluates it regardless of caller
Survives CopyImage and sharing	Irrelevant, it doesn't follow the image	Irrelevant, it doesn't follow the image	Yes, preserved on copy
When the failure shows up	`UnauthorizedOperation` at launch	Never, the type simply isn't chosen	`InvalidParameterCombination` at launch
Maintenance cost	High: replicate and audit across N accounts	Medium: one edit per fleet that exists	Low per place, high per hardware generation

Where the glob betrays you: a string is not an architecture predicate

Both lists accept *. The docs give the examples: t3.* matches the whole t3 family, g5.* matches g5, *.12xlarge matches any 12xlarge in any family, *xlarge matches any type xlarge or larger. It looks expressive. This is where I'd stop and ask the question that matters: are you describing hardware, or are you describing names?

What the AMI actually requires is a capability: arm64 versus x86_64, Nitro versus Xen hypervisor, presence of a GPU, presence of local NVMe. What the specification evaluates is a pattern over the instance type's string. The two coincide today because AWS names Graviton families with a g and local-storage families with a d. They coincide by convention, and a naming convention is not a contract: c6gn, c7gd, m7i-flex, u7in-32tb and mac2-m2pro already show a scheme full of exceptions, suffixes and infixes that no simple glob covers. A *g.* written to exclude Graviton is a heuristic, and a hand-written heuristic, in a file nobody revisits, is debt that accrues interest.

The second trap is temporal, and it's the expensive one. An explicit allow-list blocks every future generation by construction. New families reach new Regions constantly; the day the next generation shows up in your Spot pool, the fleet you deliberately left wide, because wide diversification is the correct practice for Spot capacity, will skip exactly the cheapest, most available family. You get no error: the ASG simply doesn't choose it, price creeps up, interruption rate creeps up, and nobody connects that to a JSON written eight months ago. It's the same tension behind OnDemandMaxPricePercentageOverLowestPrice, which ships with a 20% default headroom precisely so it doesn't narrow your pool without telling you.

Generate the list, don't write the list: The only way to keep the contract honest is to derive it from a capability query in the same build that produced the image: aws ec2 describe-instance-types --filters Name=processor-info.supported-architectures,Values=x86_64 Name=hypervisor,Values=nitro --query 'InstanceTypes[].InstanceType', feeding the result straight into replace-image-instance-type-specification. For fleets that already think in attributes, GetInstanceTypesFromInstanceRequirements plays the same role and lets you preview the set before writing it. The list ends up explicit, verbose and ugly, which is exactly what you want, because it is regenerated on every build, and your lag behind the AWS catalog becomes your pipeline cadence instead of the memory of whoever wrote the glob.

The hard block on incident day, and the signal nobody instruments

The documentation is explicit: EC2 enforces the specification as a hard block, and launch templates and Auto Scaling groups that reference the AMI can fail if the configured instance type isn't allowed. That isn't an implementation detail, it's the pattern's central trade. You converted a slow, expensive failure (zombie instance, health check, loop) into a fast, cheap one (InvalidParameterCombination at RunInstances). Fast and cheap wins on 95% of days. On the other 5%, you're in the middle of a zonal capacity event, price-capacity-optimized has finally found a pool with room, and your contract says no.

In financial systems that has a name: you traded correctness risk for availability risk, and you did it on purpose. When the AMI genuinely won't boot on that hardware, the trade is trivially good, the instance was never going to serve traffic. When the list is merely stale, you invented an unavailability mode that didn't exist before, in exactly the window where elasticity matters. That's why a generated allow-list and a minimal deny-list aren't aesthetic preferences: one errs toward letting through, the other errs toward blocking.

The hole is observability. The specification doesn't affect running instances, emits no metric of its own, and shows up on no dashboard by default. The auditable signal is the RunInstances event in CloudTrail carrying an errorCode, build a metric filter grouped by requestParameters.imageId and alarm on any block, because in steady state the expected rate is zero. A recurring block isn't the control working; it's a launch template pointing at an AMI whose contract changed underneath it. And add the read side to the runbook: describe-images with its InstanceTypeSpecification field is the first thing to check when a launch fails on a parameter, the error message itself sends you there, and whoever is on call at 2 AM needs to know that before opening a support case.

How to get this wrong

Applying the contract to an already-shared AMI without an inventory of consumers: you can't see the launch templates in the other 40 accounts, and the block only surfaces on their next scale-out, which is probably a traffic spike, not a deploy.
Hand-writing the list: a human-curated glob is a snapshot of the instance catalog on the day it was written, and the catalog changes every week.
Two pipelines writing the same AMI's specification: ReplaceImageInstanceTypeSpecification replaces everything and has no conditional write; the second one erases the first with no error.
Using this as a cost control: blocking *.24xlarge to contain spend is FinOps policy disguised as compatibility, that belongs in an SCP with ec2:InstanceType or in OnDemandMaxPricePercentageOverLowestPrice, where exceptions have an owner and a process.
Mistaking this for pipeline validation: the contract fails at launch, not at build. It does not replace a test that boots the image on each allowed family before promoting the AMI to production.

Reading the pattern through the pillars

operational-excellence: The contract becomes generated code in the same pipeline as the AMI, versioned with it. Add describe-images to the runbook: the error message points there, and it's the first diagnostic step.
security: Constraining the list is neither isolation nor least privilege, only the owner writes it and the rule follows the image, but it does not replace an IAM condition or an SCP for governing who launches what.
reliability: A real gain by killing the launch/terminate loop from incompatible AMIs; a real risk of failing to scale during a capacity event if the list is stale. An alarm with an expected rate of zero is what separates the two.
cost: The feature costs nothing and cuts the waste of instances that booted only to die at the health check, but a narrow allow-list pushes the fleet into pricier pools while emitting no signal at all.

What I'd actually do: I'd turn this on first for the AMIs that carry a genuine physical limit, an arm64 image, an image whose driver only exists on Nitro, an inference image that assumes a GPU, and leave general-purpose AMIs alone, where the permissive default is already right. On every image I restrict, I'd generate the list from describe-instance-types inside the build and fail the pipeline if it comes back empty, because an empty list from a broken query blocks everything silently. And I'd put a zero-expected-rate alarm on the RunInstances errorCode in CloudTrail the same day, not next sprint. The lesson that cost me, on financial-grade platforms: any control that blocks something and doesn't count how often it blocked eventually becomes a root cause nobody can name. The block is the easy part; the counter is what saves you at 2 AM.

References

Verdict

Adopt it, narrowly scoped, with the list generated. Use InstanceTypeSpecification when three conditions hold together: the AMI has a real hardware incompatibility rather than a cost preference, you don't control every account that launches the image, and the pipeline that builds the AMI can derive the list from a capability query on every build. Missing any one of them, an ec2:InstanceType condition in an SCP remains the right tool, it errs toward people who know how to ask for an exception. Don't use this as a FinOps control, don't hand-write globs, and don't enable it on a shared AMI before running describe-images against the launch templates already pointing at it. The feature costs nothing and the permissive default guarantees nothing breaks until you act; the price is entirely maintenance, and it comes due every time AWS ships a new family. If your team has nowhere to hang that maintenance, the pattern is still correct and the answer is still no.

Rating: Adopt with scope

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

DEV Community: Fernando Azevedo

One listing, many legal entities: AWS Marketplace subsidiaries in practice

The situation: the catalog was never the problem

The model: two layers and nothing shared by default

Who controls what: product in the primary, contract and money in the subsidiary

Primary account

Subsidiary account

Buyer

Flows

Where the risk lives: revoking is not switching off

Authorization as code: UpdateInformation replaces everything

Playbook for tomorrow

Three ways to sell through more than one entity

Anti-patterns I would expect in the first quarter

Questions that will come up

Can the subsidiary create a CPPO for a local channel with the authorized product?

Can I migrate agreements from the duplicate listing to the primary's product?

Can the primary create offers or complete KYC on the subsidiary's behalf?

Are Marketplace fees consolidated in the primary?

What does it cost and where is it available?

Verdict

References

AWS Builder Loft São Paulo: a new home for building together

What I took home from that evening

A place designed for continuity

Why this matters for Brazil

What I would like to bring to the table

From an encounter to a next step

Thank you for being part of this beginning

Background and ways to stay involved

Your agent crashed after the write. Should you retry?

What was verified

Start with the question the log cannot answer

Where evidence must survive

Local CLI

SQLite · separate commits

Flows

Prepare a run another person can reproduce

Experiment 1: the effect exists; the acknowledgement was lost

Recovering the record does not require repeating the effect

Observed results from both experiments

Experiment 2: the state looks right; ownership is unknown

Three commits explain the failure windows

C4

Containers

Components

What changes when the tool calls AWS

Decision: uncertainty is an operational outcome

Checklist for reviewing your own tool

A Well-Architected reading

Traps the exercise makes visible

Diagnostic replay and mutation retry are different operations

References

The question to bring to your review

Migrating agents from GPT-5.6 Terra to GPT-6.1 Terra on Bedrock

The starting point

What the model card says and the announcement doesn't

The journey, in six decisions

Converse or Responses: the decision that blocks the migration

Canary migration backed by evidence

Application

Amazon Bedrock runtime

Evidence

Flows

Before and after, at US CRIS list price

Quota, long context and the cost that shows up later

Each model's operational contract

Anti-patterns in this migration

References

Verdict

Amazon Quick on a private network: PrivateLink's two planes and where they leak

Why this matters now

The two planes of private Quick access, and what stays public

🏢 On-premises: rede corporativa

📡 Rede: Direct Connect ou Site-to-Site VPN

🟧 AWS: VPC do cliente (2 AZs no mínimo)

🤖 AWS: Amazon Quick (Enterprise)

🌐 Externo: saída pública que continua obrigatória

Flows

Ingress plane: interface endpoints, DNS and what the policy accepts

Authorization as code: `UpdateInformation` replaces everything