DEV Community: TOMOAKI ishihara

Automating CodeCommit PR Reviews with Amazon Bedrock: 986 Reviews in 10 Weeks, $0.18 Each, and What We Could Not Measure

TOMOAKI ishihara — Thu, 29 Oct 2026 13:43:21 +0000

TL;DR

A team of 5 ran 986 reviews over about 10 weeks on a single CDK infrastructure repository. Each review took about 60 seconds on average, and the actual Bedrock bill totaled $180.60 (about $0.18 per review)
Review count, cost, and duration are now visible as numbers, but we still can't measure whether the findings were correct, or how lead time changed before and after adoption

Introduction
Architecture Overview
How the PR Review Works
Measuring the Effect
Visualization
Thinking About Cost
Results in Production
Design Decisions
Limitations and Future Improvements
Conclusion

Introduction

CodeCommit is a very convenient service that lets you keep your whole development environment inside AWS. However, compared with GitHub, its surrounding ecosystem is limited. An OSS tool such as Qodo AI's PR-Agent has a provider for CodeCommit, but its features are limited and it assumes you use it from the CLI.

So I built a mechanism that reviews CodeCommit pull requests (PRs) with Amazon Bedrock and posts the result as comments. I did not use the OSS PR-Agent. Instead, I built it from scratch with EventBridge and Lambda so that it runs automatically whenever a PR is created or updated. Many of the articles I referred to ended at "post the comment", but once you think about operating it, you also need to make the following visible:

How much does it cost per month?
Whose PRs are they, and how many tokens does each model use?
Are findings actually being raised, and if so, which severity is most common?
Has the time until a PR is merged changed?

This article also explains how to collect and report these numbers.

Architecture Overview

The overall architecture looks like this.

Full architecture diagram (including CDK test CI, usage dashboard, and Slack/Teams notifications)

This is the review flow after a PR is made.

EventBridge receives PR creation and update events, and a Lambda function (Python) builds the diff and passes it to Bedrock. The result is posted to CodeCommit as a PR comment. The posted comments look like the following (only the beginning is shown). Code and documents are posted as separate comments.

Code review

Doc review

The system keeps three kinds of data needed for visualization.

Layer	What you learn	Source data	Storage
① Review content	Findings (by severity), LGTM, agreement between models, duration, errors	Bedrock structured output	CloudWatch custom metrics
② Cost and usage	Token counts and estimated cost by repository, PR author, and model	Bedrock invocation logs	CloudWatch Logs
③ Development flow	Time from PR creation to merge	PR status change events	JSON in S3

How the PR Review Works

Building the diff from blobs

CodeCommit's GetDifferences returns the list of changed files and their blob IDs, but not the diff text. So I fetch the before and after blobs with GetBlob and assemble a unified diff with Python's difflib.

diff_lines = list(difflib.unified_diff(
    before_content.splitlines(),
    after_content.splitlines(),
    fromfile=f"a/{path}",
    tofile=f"b/{path}",
    n=5,          # lines of context around each change
    lineterm="",
))

Here is how each change type is handled. Only the before/after diff is passed to the model, so unchanged lines consume no tokens.

Change type	What is passed to the model
Modified (M)	Unified diff (with 5 lines of context before and after)
Added (A)	The whole file
Deleted (D)	Path only (noted as "deleted")
Binary / empty	No body (noted as such)

I also set an upper limit on the number of diff lines handled in one review. When the limit is reached, the rest is cut off per file, and the PR comment states that it was truncated.

Routing by path

Changed files are classified by path into code / document / skip. Code and documents are reviewed separately, and paths that match skip are excluded from the review. The patterns are set with the Lambda environment variables DOCUMENT_PATH_PATTERNS and SKIP_PATH_PATTERNS.

Skipping a review

I also added a feature to skip the review of a whole PR when the title contains [skip-review]. Excluding by path is also possible, but writing it in the title skips the review immediately. In this repository, documentation is generated as HTML with MkDocs, so regenerating it doesn't trigger a review of a huge number of files.

Combining multiple models with Agents-as-Tools

You can specify multiple models through parameters. When you specify multiple models for review, you can also specify a separate model that summarizes their results. For example, you can use two models, Nova Pro and Claude Sonnet 4.6, for review, and Claude Haiku 4.5 to summarize the results.

In the per-environment parameter file, set reviewMode to multi and write the model IDs in multiModelConfig. To run with a single model, use reviewMode: "single" and singleModelConfig: { modelId }.

reviewMode: "multi",
multiModelConfig: {
  modelIds: ["apac.amazon.nova-pro-v1:0", "jp.anthropic.claude-sonnet-4-6"],  // reviewers
  summaryModelId: "jp.anthropic.claude-haiku-4-5-20251001-v1:0",              // merges the results
},

The reviewer models are exposed as @tools and called by the summary agent. At first, I passed the diff to each reviewer tool as an argument. However, since the one calling the tool is the summary agent (an LLM), it had to generate the diff character by character as an argument. The diff is large, so this generation took a long time and caused timeouts. Now the tools take no arguments. The prompt containing the diff is held in a variable outside the tool, and the tool references it directly. Because the diff no longer passes through the LLM, this problem doesn't occur.

def _make_reviewer(mid: str):
    @tool
    def reviewer_tool() -> str:  # no arguments: the diff never passes through the LLM
        if mid not in reviewer_results:  # each reviewer runs only once
            sub_agent = Agent(model=_build_bedrock_model(mid, repository_name, pr_author))
            reviewer_results[mid] = str(sub_agent(prompt_text))  # prompt_text holds the diff
        return reviewer_results[mid]
    return reviewer_tool

Comment limit: 10,240 bytes

CodeCommit comments have a limit of 10,240 bytes. Code reviews and document reviews are posted as separate comments, and if a comment still exceeds the limit, the end is truncated with a note that content was omitted.

Avoiding duplicate posts for the same commit

An HTML comment containing the commit ID is appended to the end of each posted comment. Before reviewing, the Lambda reads the existing comments and skips the review if it finds the same marker and commit ID.

suffix = f"\n\n<!-- commit:{commit_id} -->"  # appended to every review comment

# before reviewing: look through the existing comments on the PR
if DUPLICATE_MARKER in content and source_commit_id in content:
    return True  # already reviewed this commit, skip

Measuring the Effect

This time, I designed and implemented the system with the goal not only of building the review mechanism but also of reporting the adoption effect inside the company. So I decided which numbers to collect up front, in parallel with implementing the review.

That said, not everything was in place from the start. I added the metrics I needed little by little while operating the system.

① Receive review content as structured data, not free text

I make the summary agent always call a tool named report_review_result at the end.

@tool
def report_review_result(
    summary_markdown: str,   # the PR comment body
    findings: list[dict],    # [{"severity": "critical|warning|info", "count": 2, "detected_by": ["nova", "sonnet"]}]
    lgtm: bool,
) -> str:
    """Report the final review result."""

You could extract severity from the comment's Markdown with a regular expression, but even a small change in the body's format would break it. If you receive it as tool arguments, the number of findings and LGTM can be used directly as numbers.

The received values are sent to the PRAgent/Review namespace with PutMetricData.

put_metric("FindingCount", count, dimensions={
    "Repository": repo, "ReviewGroup": group,
    "Severity": severity, "Environment": env,
})

These are the main metrics sent.

Metric	Description
FindingCount	Number of findings (by repository, review type, severity, and environment)
ReviewCount / ReviewLatencyMs	Number of reviews and duration
ReviewWithinSLACount / ReviewExceededSLACount	Whether the review finished within the target time
InputTokenCount / OutputTokenCount	Token counts
MultiModelAgreementCount / MultiModelUniqueCount	Findings detected by 2 or more models / findings detected by only 1 model
ToolUseFallbackCount / KbFallbackCount / ErrorCount	Fallbacks and errors
ChangedFileCount / ChunkCount / SkipCount	PR size, number of chunks, and number of skips

Sending can be switched on and off with the deploy parameter ENABLE_METRICS. If sending fails, the review itself does not fail; the error is only logged.

Agreement between models

With two models, you can tell whether both raised the same finding or only one did. A finding with two or more entries in detected_by counts as agreement, and one with a single entry counts as uniquely detected. A high agreement rate is a reason to cut down to one model and lower the cost, and a low rate suggests that using two is worthwhile. Note, however, that "findings both models agree on are correct" is not guaranteed (see the limitations below).

② Cost and usage: add metadata to the invocation log

If you enable Bedrock's model invocation logging, the token count of each call is stored in the log. However, as it is, you can't tell which repository and whose PR a call belongs to.

Bedrock's Converse has requestMetadata, and the values you put there are written to the invocation log. With Strands' BedrockModel, you can pass it through additional_args.

BedrockModel(
    model_id=model_id,
    additional_args={"requestMetadata": {
        "repository": repo, "prAuthor": author, "environment": env,
    }},
)

The reporting Lambda aggregates this log with Logs Insights.

fields @timestamp
| filter ispresent(requestMetadata.repository)
| stats sum(input.inputTokenCount) as inTok, sum(output.outputTokenCount) as outTok
  by requestMetadata.repository, modelId

There is an upper limit on the number of fields you can specify in stats ... by in Logs Insights, so "repository × model" and "PR author × repository" are split into separate queries.

Cost is estimated by multiplying the aggregated token counts by the configured unit prices. The unit prices are given in modelPricing in the parameter file (USD per 1,000 tokens for input and output). This is only an estimate, so check the exact amount in Cost Explorer or similar.

③ Record the time until merge

A separate Lambda receives the PR status change event (pullRequestStatusChanged) and records the lead time.

Note that this event alone cannot tell you whether the PR was merged or closed without merging. So the Lambda calls GetPullRequest and checks pullRequestTargets[].mergeMetadata.isMerged. PRs that were closed without being merged are not recorded.

Records are saved to S3 as JSON, partitioned by date.

reports/pr_lead_time/YYYY/MM/DD/<repository>_pr-<id>.json

{
  "recordType": "pr_merge_lead_time",
  "repositoryName": "sample-repo",
  "pullRequestId": "123",
  "author": "alice",
  "mergeOption": "SQUASH_MERGE",
  "createdAt": "2026-01-01T00:00:00Z",
  "mergedAt": "2026-01-01T03:30:00Z",
  "leadTimeSeconds": 12600,
  "leadTimeHours": 3.5
}

Visualization

The collected numbers can be viewed in the following three ways.

CloudWatch dashboard

The metrics from ① are shown as graphs such as the number of reviews per repository, findings by severity, agreement and unique detections between models, and token usage per model. Repositories come and go, so the widgets are built with SEARCH expressions and no repository name is hard-coded.

CloudWatch dashboard (excerpt showing findings by severity and token usage per model)

Alarms are set on review errors, SLA breaches, messages in the DLQ, and so on.

Daily, weekly, and monthly reports

A report-generating Lambda is started on a schedule, runs Logs Insights, and saves JSON like the following to S3.

reports/daily/YYYY/MM/DD/report.json
reports/weekly/YYYY/Www/report.json
reports/bedrock/...        # Bedrock usage by model and PR author
reports/monthly/...

The content is sent as a notification through SNS.

Monthly report

Daily report

{
  "reportType": "daily",
  "environment": "dev",
  "periodStart": "2026-09-29T00:00:00Z",
  "periodEnd": "2026-09-30T00:00:00Z",
  "generatedAt": "2026-09-30T00:05:40Z",
  "summary": {
    "reviewCount": 40,
    "lgtmCount": 0,
    "totalInputTokens": 1073657,
    "totalOutputTokens": 167242,
    "avgLatencySeconds": 72.403,
    "totalReviewedFiles": 188,
    "totalReviewedLines": 12995
  },
  :
}

Usage dashboard (optional)

For those who want to see the JSON in S3 on a web page, I also built a static site on CloudFront + S3, and an API Gateway (HTTP API) protected by Cognito JWT authentication. The API reads the reports in S3 through a Lambda function and shows them on two screens: an overall view and a per-PR-author view.

Authentication uses Authorization Code + PKCE, and self sign-up is disabled
The static site is locked down in S3 with OAC and is reachable only through CloudFront

This can be enabled as an option.

Usage dashboard: overall (top part excerpted)

Usage dashboard: individual (top part excerpted; review lead time not yet supported)

Thinking About Cost

If you have a few hundred PRs a month, the main cost is Bedrock token charges. The rough formula is as follows.

Monthly cost ≈ number of PRs × reviews per PR × (input tokens × input price + output tokens × output price) × number of models

The extra cost of collecting metrics is determined by the number of custom metrics (one per combination of dimensions) and the number of PutMetricData calls. The count grows with the combinations of repository, severity, and environment, so check it if you have many repositories. Check the latest CloudWatch pricing page for unit prices.

Results in Production

These are the results of a development team (5 people) running the system for about 10 weeks on a single CDK infrastructure repository. The numbers come from the report JSON in S3 and the billed amounts in Cost Explorer. This environment also includes a few reviews of other repositories, but most are for the repository under evaluation. The setup used a single model (reviewMode: "single"), Claude Sonnet 4.6 (cross-region inference). This is not an operating record of multiple models.

Month	Reviews	Findings (Critical / Warning / Info)	LGTM	Avg. duration	Actual Bedrock bill
July (from the 23rd)	73	112 / 319 / 272	0	68 s	$20.94
August	362	406 / 1,236 / 1,027	6	61 s	$54.65
September	551	530 / 1,861 / 1,653	15	58 s	$105.00
Total	986	1,048 / 3,416 / 2,952	21	about 60 s	$180.60

On average, one review covered 12 files and about 300 lines, with 1.1 Critical and 3.5 Warning findings. No findings (LGTM) occurred in 21 of 986 reviews (about 2%), so almost every review raised something
Duration averaged about 1 minute. The weekly p95 was in the 2–3 minute range, and the slowest week was about 5 minutes
By author, 2 of the 5 people accounted for about 80% of Bedrock calls. The difference in how often people open PRs shows up directly

Estimate vs. actual

One review cost about $0.18. From July to September, the number of reviews grew about 7.5 times and the bill about 5 times. The per-review cost was $0.29 in July, $0.15 in August, and $0.19 in September.

Estimating from token counts with the formula above (29.58 million input and 4.33 million output tokens, about $154 at Sonnet's public list price) came out about 15% lower than the actual bill. It seems better to check the exact amount in Cost Explorer.

Design Decisions

Decision	Choice	Reason and trade-off
Receiving results	tool_use (structured output)	Robust against changes in the body format. However, it counts the severity self-reported by the model as is, so correctness isn't guaranteed
Sending metrics	Call `PutMetricData` directly	About 10–15 calls per review is too little to justify EMF (estimated at about $0.015/month for 1,500 calls). Consider EMF if the volume grows
Detecting merges	Check with `GetPullRequest`	The event alone can't distinguish a merge from a rejection. The trade-off is one extra API call per PR
Adding data to invocation logs	`requestMetadata`	Uses the values measured by Bedrock instead of re-counting tokens in the app
Dashboard authentication	Serve the static site's HTML and JS without authentication, and protect only `/api/*` with Cognito JWT	Without logging in, the JavaScript on the page redirects to Cognito's login screen. The screen layout and API paths are visible even when not logged in, but no aggregated values can be fetched, so the real harm is small. To authenticate before serving the HTML and JS, you can add authentication on the CloudFront side, but the setup becomes more complex

Limitations and Future Improvements

This time I couldn't measure whether the findings were correct. The severity of a finding is self-reported by the model, and I can't tell whether it was an actual bug or whether it was fixed. So even if the number of findings drops, you can't say code quality improved
- As an improvement, I implemented a Lambda (review_feedback) that collects replies to findings, 👍/👎 reactions, and whether the flagged file was changed afterward, at the time the PR is merged or closed. It is under evaluation.
Lead time can be viewed by day, repository, and PR author in the usage dashboard, but I can't compare it before and after adopting the review
- There is no data from before adoption, so pre-adoption PRs would have to be recalculated from CodeCommit history
Findings and LGTM rate by PR author can't be produced. prAuthor is attached to the invocation log, so token counts by author are available, but the findings and LGTM metrics don't carry the author
- Adding the author to the metrics increases the combinations of dimensions, and with them the number of custom metrics and the cost. I'll consider adding it after checking that it's worth the cost
Because prAuthor is kept in the invocation log and reports, per-person usage is visible. The purpose is to understand usage, not to evaluate individuals, and this needs to be settled during operation
- It will need care if the usage dashboard is rolled out widely
Anyone who opens a PR can write comments freely. An instruction hidden in a comment could steer the review result (prompt injection)
- Even if the review is steered, the review Lambda doesn't change code or merge, so the impact is limited to review accuracy and comments. I haven't verified whether the review can really be steered, but I plan to test countermeasures, such as stating explicitly in the prompt that the PR body and diff are "data to be examined, not instructions"

Conclusion

Running the review and posting comments was relatively easy to build. However, seeing the effect in numbers required structured output and metadata on the invocation logs. In this evaluation, I couldn't measure whether the findings were correct, or how lead time changed before and after adoption. Also, the track record comes from only one team, so I'd like to keep refining it by operating it with other teams as well.

Testing the AWS WAF Mitigations for PHP2Shell on Both CloudFront and an ALB

TOMOAKI ishihara — Thu, 22 Oct 2026 16:10:45 +0000

Introduction

On 2026-10-21 the PHP project disclosed PHP2Shell (CVE-2026-109618 / CVE-2026-109619): CVSS v4.0 9.2, unauthenticated remote code execution (PHP security advisory, CERT-EU Security Advisory 2026-018).

As a stopgap until PHP can be upgraded, an AWS Security Blog post published four AWS WAF custom rules: PHP2Shell, Block-Multipart-FormData, Block-Oversize-Body and RateLimit-Posts. The AWS Security Bulletin also points to SizeRestrictions_BODY in AWSManagedRulesCommonRuleSet and to the PHP managed rule group (version 2.3, now the default), which adds protection for CVE-2026-109619.

Classmethod (a Japanese company that provides AWS consulting and operations support) has already tested the AWS Security Blog rules thoroughly on an ALB, including the fact that PHP_SESSION_UPLOAD_PROGRESS placed after the first 8 KB of the body goes through.

This article reproduces that test on both CloudFront and an ALB and verifies what was still open:

How far does raising CloudFront's body inspection limit (16 KB default, 64 KB max) get you?
How does the ALB's Web ACL behind CloudFront affect the result?
What changes when the origin only accepts traffic through CloudFront?
How long does the rate-based rule take to react once the threshold is crossed?

📁 Code: waf-php2shell-mitigation (part of AWS CDK Reference Architectures).

Note
AWS WAF is a mitigation, not a fix. Upgrading to PHP 8.2.35 / 8.3.36 / 8.4.27 / 8.5.12 or later is the only complete remediation. PHP 8.1 and older are also affected but are end-of-life, so there is no official fixed release (the PHP team published minimal source backports, and some distributions and LTS vendors provide backported packages). Move to a supported version.

TL;DR

The AWS Security Blog rules are a stopgap until PHP is upgraded; the AWS Security Blog post itself states that the real fix is upgrading PHP. As a stopgap, the rules behaved as specified on both CloudFront and the ALB.

AWS WAF, however, inspects a request body only from the start up to a size limit: 8 KB on an ALB (fixed), 16 KB by default on CloudFront, up to 64 KB (AWS WAF documentation). On the ALB, a string ending at byte 8,192 was blocked; shifting it by one byte let it through. The Block-Oversize-Body rule from the same post (a size limit meant for CVE-2026-109618), when set to BLOCK, also stops requests whose bodies go past that limit.

How to configure that rule depends on your setup:

If you use CloudFront, raise the inspection limit to 64 KB and block bodies larger than 64 KB with Block-Oversize-Body. The whole body is then inspected, and legitimate requests up to 64 KB still pass.
With CloudFront + ALB + WAF, set the ALB's "block bodies over 8 KB" rule to COUNT and let the ALB accept traffic from CloudFront only. Left in BLOCK, the ALB stops legitimate requests that CloudFront already allowed. If the ALB Web ACL includes AWSManagedRulesCommonRuleSet, override its SizeRestrictions_BODY (blocks bodies over 8 KB) to COUNT for the same reason.
If you expose only an ALB, blocking bodies over 8 KB is the baseline. According to the AWS Security Blog post, both CVEs sit on the multipart/form-data processing path, so an app that takes no file uploads and only accepts JSON over 8 KB could block Block-Multipart-FormData instead of using the size limit (not verified in this article). An app that accepts file uploads over 8 KB cannot be fully protected by WAF, so prioritize the PHP upgrade.

If your app changes session.name or session.upload_progress.name from the defaults, rewrite the rule strings to the actual names. Matching is case-sensitive, so this also applies when only the letter case differs.

Architecture

Client ─▶ CloudFront + Web ACL (us-east-1, 64 KB body inspection)
           ─▶ ALB + Web ACL (ap-northeast-1, fixed 8 KB)
              ─▶ Lambda container (nginx + PHP-FPM 8.4.27 + Lambda Web Adapter)
Client ─────────▶ ALB direct (comparison only, allow-listed IP)

A WAF decision does not depend on the PHP version behind it, so the origin runs a patched PHP 8.4.27 as a Lambda container image. On the PHP side, it applies these server-side mitigations from the AWS Security Blog post:

; php.ini
session.upload_progress.enabled = 0
memory_limit = -1        ; the real ceiling is the Lambda memory size
post_max_size = 8M

; PHP-FPM pool
pm = static
pm.max_children = 1
pm.max_requests = 1      ; replace the worker after every request

Telling which WAF and which rule acted, from HTTP headers

With WAF on both CloudFront and the ALB, a bare 403 does not tell you which layer answered. Every rule therefore names itself:

const ruleAction = (ruleName: string, action: RuleAction): wafv2.CfnWebACL.RuleActionProperty =>
  action === 'BLOCK'
    ? { block: { customResponse: { responseCode: 403,
        responseHeaders: [{ name: 'x-php2shell-blocked-by', value: `${label}:${ruleName}` }] } } }
    : { count: { customRequestHandling: {
        insertHeaders: [{ name: `php2shell-${label}-${ruleName.toLowerCase()}`, value: 'counted' }] } } };

BLOCK returns x-php2shell-blocked-by: cloudfront:PHP2Shell; COUNT inserts x-amzn-waf-php2shell-alb-oversize: counted into the origin request, and the PHP app echoes it back. Headers inserted by the CloudFront Web ACL travel through the ALB to the origin, and the BLOCK response header keeps its own name (no x-amzn-waf- prefix).

Note that clients can also send headers starting with x-amzn-waf-. If your application changes its behavior based on these headers in production, add a rule that blocks x-amzn-waf- headers arriving from outside, as the AWS WAF documentation describes.

Result 1: rule conditions (reproducing the ALB test on both layers)

Request	via CloudFront	ALB direct
POST, cookie `PHPSESSID` + `PHP_SESSION_UPLOAD_PROGRESS` in body	403	403
POST, both strings in body	403	403
POST, PROGRESS only / cookie only	200	200
PUT, both strings in body	200	200
multipart PROGRESS field + cookie	403	403
lowercase cookie key `phpsessid`	200	200
lowercase `php_session_upload_progress` in body	200	200
JSON or text/plain containing both strings (false positive)	403	403

The rule uses TextTransformation: NONE, so it only matches PHP's default names. If your app renames the session cookie or the upload progress field, change the rule strings too.

Result 2: string position vs inspection limit

Cookie PHPSESSID, with PHP_SESSION_UPLOAD_PROGRESS ending at byte N (oversize rule in COUNT):

N	ALB direct	CloudFront `KB_16`	CloudFront `KB_64`
8,192	403	403	403
8,193	200	403	403
16,385	200	200	403
65,536	200	200	403
65,537	200	200	200

WAF logs agree: requestBodySizeInspectedByWAF is 8192 on the ALB and 65536 on CloudFront KB_64, and larger bodies carry oversizeFields: ["REQUEST_BODY"].

Result 3: oversize rule in BLOCK (8,192 bytes)

Everything from 8,193 bytes is blocked regardless of position. So is a legitimate ~19 KB JSON POST with no rule string at all. An API that accepts JSON over 8 KB cannot use this as is.

Result 4: with WAF on both CloudFront and the ALB, the ALB blocks legitimate requests

CloudFront can inspect the first 64 KB of the body, so I raised CloudFront's Block-Oversize-Body threshold to 65,536 bytes. CloudFront then passes legitimate requests up to 64 KB and blocks bodies too large to inspect in full.

The ALB's Web ACL is where this goes wrong. Putting CloudFront in front of an ALB that already had WAF, or adding the same rules to both layers as defense in depth, is common. Even when the origin only accepts CloudFront, requests through CloudFront are evaluated again by the ALB Web ACL.

ALB `Block-Oversize-Body` (8,192 bytes)	Legit 19 KB JSON	PROGRESS at byte 65,537
BLOCK	403 `alb:Oversize`	403 `cloudfront:Oversize`
COUNT	200	403 `cloudfront:Oversize`

(Both through CloudFront; CloudFront blocks above 65,536 bytes.)

At first I thought raising CloudFront's limit would be enough. With the ALB rule in BLOCK, though, the ALB's 8 KB rule stops the 19 KB JSON that CloudFront inspected and allowed. To benefit from CloudFront's setting, the ALB size rule has to be COUNT.

Result 5: restricting ALB access to CloudFront

COUNT on the ALB opens a different problem: a request sent straight to the ALB skips CloudFront, and the ALB inspects only 8 KB. A request with PROGRESS ending at byte 8,193, sent directly, was 403 with the ALB rule in BLOCK and 200 with it in COUNT.

If you follow the AWS best practice and restrict the ALB to traffic from CloudFront, the Result 4 fix (COUNT on the ALB) is not a problem. Setting it to COUNT while the ALB is still directly reachable, however, is dangerous.

As the documentation describes, I combined a security group and a custom header. The security group admits only the CloudFront origin-facing managed prefix list (com.amazonaws.global.cloudfront.origin-facing). That list is shared by every CloudFront distribution, so on its own it also lets other distributions in. I therefore set an X-Origin-Verify origin custom header on CloudFront, and the ALB listener rule forwards only requests whose header value matches (default action 403).

const cloudFrontPrefixList = ec2.PrefixList.fromLookup(this, 'CloudFrontOriginFacing', {
  prefixListName: 'com.amazonaws.global.cloudfront.origin-facing',
});
albSecurityGroup.addIngressRule(ec2.Peer.prefixList(cloudFrontPrefixList.prefixListId), ec2.Port.tcp(80));

listener.addAction('FromCloudFront', {
  priority: 10,
  conditions: [elbv2.ListenerCondition.httpHeader('X-Origin-Verify', [originVerifyValue])],
  action: elbv2.ListenerAction.forward([targetGroup]),
});

This approach relies on keeping the header name and value secret. For this verification CloudFront talks to the ALB over HTTP (port 80); the documentation recommends HTTPS and rotating the header regularly. Another option is to keep the ALB off the internet and connect an internal ALB to CloudFront as a VPC origin.

For Results 1–4, I had added an exception that let my own IP address reach the ALB directly, for comparison. After removing that exception and redeploying, a direct curl to the ALB got no response at all, not even a 403, and timed out: the security group drops the traffic before it reaches the ALB. Through CloudFront, requests still returned 200, and requests with the PHP2Shell pattern were still blocked with 403.

Other observations

Check	Result
`RateLimit-Posts` (100 per 60 s, COUNT) at ~2.4 req/s	first match on request #138, 57 s in
1,100,000-byte body, ALB → Lambda	413 (1 MB Lambda target limit), not in the ALB WAF log
WCU	CloudFront 50 / ALB 45 (`AWSManagedRulesCommonRuleSet` alone is 700)
`pm.max_requests=1`	FPM worker PID changes on every request
Lambda container cold start	3.2 s first, 0.2 s after; ~48 MB memory

The rate-based rule first matched on request #138 (in COUNT; in BLOCK, blocking would start there). The threshold is 100 per 60 seconds, so about 16 seconds and 38 requests went through after it was crossed. If you set the threshold to exactly the number you can tolerate, requests that arrive while detection lags go through on top of it. The overshoot is roughly "request rate × lag": here, 2.4 req/s × ~16 s ≈ 38. Faster attacks overshoot more, so set the threshold below what you can tolerate. The AWS WAF documentation also says rate-based rules are not meant for precise rate limiting, and that the detection lag is usually under 30 seconds but can stretch to minutes.

Cost

About $40/month if left running (ALB ~$18, two Web ACLs $10, eight rules $8, plus requests). With KB_64, only requests whose bodies exceed 16 KB pay an extra $0.30 per million per additional 16 KB. The verification itself, deploy to destroy within a few hours, cost under $1.

Wrap-up

The AWS Security Blog rules behave as specified on both CloudFront and the ALB, but only for strings within the inspected part of the body. Even with CloudFront's limit raised to 64 KB, an 8 KB rule left on the ALB blocks legitimate requests, and a directly reachable ALB lets requests through with everything past 8 KB uninspected. Configure WAF for the whole request path rather than for CloudFront and the ALB separately.

That said, every measure verified here is a mitigation. Upgrade PHP first.

References

AWS CDK 100 Drill Exercises #013: CloudFront S3 Static Website — Serving a Private Bucket over HTTPS with OAC and Cross-Region WAF

TOMOAKI ishihara — Sat, 29 Aug 2026 07:27:51 +0000

Introduction

This is the 13th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

Last time (#012), we covered publishing a site using S3 static website hosting alone. That setup had two clear limitations: no HTTPS support, and the bucket itself sitting open to Principal * (albeit IP-restricted).

This time we rebuild the same static site behind Amazon CloudFront, with a completely private S3 bucket. The bucket can never be read directly from the outside again, traffic is served as HTTPS over TLS 1.3, and you can optionally add defense-in-depth with WAFv2.

What you'll learn in this article

How to serve from CloudFront without ever making the S3 bucket public, using Origin Access Control (OAC)
Applying security headers like CSP and HSTS via a ResponseHeadersPolicy, and hiding the Server header
A design that rewrites 403/404 to index.html for SPA client-side routing, while showing genuine 5xx failures with their real status codes intact
How to handle the constraint that CLOUDFRONT-scoped WAFv2 is pinned to us-east-1 using crossRegionReferences
A two-stage IP allow-list design: "before managed rule evaluation" and "after managed rule evaluation"

📁 Code repository: GitHub

Architecture Overview

Trait	Benefit
Private bucket + OAC	Unlike S3 website hosting, the bucket itself is never publicly reachable. Only CloudFront's OAC ID can read it
Cross-region WAF stack	The Web ACL and its related resources live in a stack pinned to us-east-1, with only the ARN passed to the main stack via `crossRegionReferences: true`
Error mapping that doesn't hide real failures	`403`/`404` go to the SPA's `index.html`; `500`-`504` keep their original status codes and show a dedicated error page
Single-flag toggles	`enableWaf` and `geoRestrictionCountries` can each be turned on/off with one optional parameter

Data Flow

Viewer
  │  HTTPS (min TLS 1.3), optional geo-restriction
  ▼
CloudFront Distribution
  ├─ WAFv2 Web ACL (optional, us-east-1): managed rule groups + IP allow-list → default action: block
  ├─ ResponseHeadersPolicy: applies CSP + security headers to every response
  ├─ Default behavior "/*"
  │     └─ S3 origin (OAC) ───────────────────► WebsiteBucket (private)
  │
  └─ Error handling
        ├─ 403 / 404  → /index.html, HTTP 200, TTL 5 min
        └─ 500-504    → /error.html, original status code preserved, TTL 1 min

Implementation Highlights

1. Private bucket + OAC — the polar opposite of "who can read this"

In the previous S3 website hosting version, the bucket was (conditionally) open to Principal: *. This time it's the exact opposite: the bucket itself is completely private.

const websiteBucket = createAccountRegionalBucket({
  scope: this, id: 'WebsiteBucket',
  // no websiteIndexDocument or similar settings at all
});

const distribution = new cloudfront.Distribution(this, 'WebsiteDistribution', {
  defaultBehavior: {
    origin: cloudfrontOrigins.S3BucketOrigin.withOriginAccessControl(websiteBucket),
    viewerProtocolPolicy: cloudfront.ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
  },
});

S3BucketOrigin.withOriginAccessControl() handles both creating the OAC and adding a bucket policy scoped to the distribution's ARN in a single line. Nobody can read the bucket on its own — the only relationship that grants read access is a resource-level one with "this specific CloudFront distribution."

2. Bulk security headers via `ResponseHeadersPolicy`

CloudFront's ResponseHeadersPolicy can inject response headers regardless of how the origin is implemented. On top of CSP, HSTS, and X-Frame-Options, we overwrite the Server header with an empty string to hide server information.

securityHeadersBehavior: {
  contentSecurityPolicy: {
    contentSecurityPolicy: "default-src 'self'; script-src 'self'; ...",
    override: true,
  },
  strictTransportSecurity: {
    accessControlMaxAge: cdk.Duration.days(365),
    includeSubdomains: true,
    preload: true,
    override: true,
  },
  // ...
},
customHeadersBehavior: {
  customHeaders: [{ header: 'server', value: '', override: true }],
},

Even though this setup only serves static HTML, the CSP explicitly blocking anything beyond self-hosted scripts/styles — and not allowing unsafe-inline — sets a useful design precedent for whenever a JS framework gets layered on top later.

3. Gentle with the SPA, honest about real failures

CloudFront's custom error responses rewrite both 403 and 404 to index.html + 200. This ensures that when a path that doesn't exist as an S3 object (an SPA client-side route) is accessed directly, control is correctly handed off to the app's own routing.

Genuine origin/edge failures like 500-504, on the other hand, keep their original status code and only swap in error.html for the body.

errorResponses: [
  { httpStatus: 403, responseHttpStatus: 200, responsePagePath: '/index.html', ttl: cdk.Duration.minutes(5) },
  { httpStatus: 404, responseHttpStatus: 200, responsePagePath: '/index.html', ttl: cdk.Duration.minutes(5) },
  // 5xx are genuine failures, not SPA routing concerns, so keep the status code as-is
  // and only replace the body, so origin error details aren't leaked
  ...[500, 502, 503, 504].map((httpStatus) => ({
    httpStatus, responseHttpStatus: httpStatus, responsePagePath: '/error.html', ttl: cdk.Duration.minutes(1),
  })),
],

You'll often see implementations that just turn everything into a 200 to paper over errors, but doing that buries genuine failures under a 200 from a monitoring perspective. Here, 403/404 are treated as SPA-specific, while 5xx stays honest — the two roles are kept separate.

4. CLOUDFRONT-scoped WAFv2 is pinned to us-east-1 — wire it with `crossRegionReferences`

When you specify scope: 'CLOUDFRONT' for a WAFv2 Web ACL, that Web ACL can only be created in us-east-1, regardless of which region the distribution itself lives in. Try creating it elsewhere and you get rejected with The scope is not valid.

In this reference implementation, all WAF-related resources are factored out into a separate stack (CloudfrontWafStack) that always deploys to us-east-1. Only its ARN is passed to the main stack, using crossRegionReferences: true set on both stacks.

const wafStack = new CloudfrontWafStack(this, pascalCase(`${props.project}Waf`), {
  env: { account: props.params.accountId, region: 'us-east-1' },
  crossRegionReferences: true,
  enableWaf: props.params.enableWaf,
  allowedIpsAfterRules: props.allowedIps,
});

const mainStack = new CloudfrontS3StaticWebsiteStack(this, pascalCase(`${props.project}Main`), {
  webAclArn: wafStack.webAclArn,
  crossRegionReferences: true,
});
mainStack.addStackDependency(wafStack);

Even when enableWaf: false (or unset), CloudfrontWafStack itself is still deployed — it just doesn't create a Web ACL, and webAclArn becomes an empty string, so the distribution deploys without a webAclId. By designing this as "don't create the Web ACL" rather than "remove the whole stack when WAF isn't needed," toggling enableWaf on and off later never breaks the cross-region reference wiring itself.

5. Allow-listing before or after the managed rules

The Web ACL's defaultAction is block. That means nothing gets through unless there's an explicit Allow rule. This stack has a two-stage IP allow list.

AllowSpecificIPsBeforeRules (priority 1, created only when configured): bypasses evaluation of the AWS managed rule groups (Common/KnownBadInputs/AdminProtection/IpReputation/AnonymousIp) entirely
AllowSpecificIPsAfterRules (priority 100, always created): evaluated after the managed rule groups. When no IPs are specified, it defaults to allowing the entire IPv4/IPv6 address space (the managed rules still apply)

addresses: props.allowedIpsAfterRules
  ? props.allowedIpsAfterRules.map(ip => `${ip}/32`)
  : ['0.0.0.0/1', '128.0.0.0/1'], // WAF rejects /0, so the full IPv4 range is split into two /1s

In the default wiring, the deploy operator's IP is passed as allowedIpsAfterRules. That means even allow-listed IPs first get inspected for known attack patterns by rules like Core/KnownBadInputs before being let through. Use allowedIpsBeforeRules only for the exceptional case where you want to bypass managed-rule false positives during active testing.

6. WAF logs go directly to S3, with sensitive headers masked

To deliver WAF logs directly to S3, the bucket name must start with aws-waf-logs- (otherwise creating the CfnLoggingConfiguration itself is rejected). On top of that, authorization and cookie headers are masked before being logged.

redactedFields: [
  { singleHeader: { Name: 'authorization' } },
  { singleHeader: { Name: 'cookie' } },
],

Deploy & Verify

export PROJECT=your-project
export ENV=dev

npm run bootstrap   # first time only
# Also bootstrap us-east-1 once — the WAF stack deploys there and crossRegionReferences
# needs that region bootstrapped as well:
#   npx cdk bootstrap aws://<account-id>/us-east-1 --profile $PROJECT-$ENV

npm run stage:deploy:all

curl https://<output WebsiteDistributionDomainName>/

The reference dev config ships with enableWaf: true and geoRestrictionCountries: ['JP'], so a straight deploy gives you a WAF-protected, Japan-only distribution. Adjust parameters/dev-params.ts to change either.

Access from an IP not in the WAF's allow list, and WAF returns a 403 before the request ever reaches CloudFront's own error page.

Cost Estimate

💰 Rough monthly estimate (Tokyo region, low traffic)

Service	Usage	Rough monthly cost
CloudFront	Low request volume, minimal data transfer	~$1-2
AWS WAF	Web ACL ($5) + 6 rules ($1 each), low request volume	~$11
S3 (2 buckets)	A few MB of content + access/WAF logs	Under $0.10

Total (WAF enabled): ~$12-14/month
Total (enableWaf: false): ~$1-2/month

WAF is billed per Web ACL, per rule, and per evaluated request, making it the dominant cost factor at low traffic volumes. For pure demo use where IP/geo restriction isn't needed, enableWaf: false significantly lowers the cost.

Summary

What we learned from this pattern:

Origin Access Control lets you serve from CloudFront without ever making the S3 bucket public. The bucket alone is unreadable by anyone — the polar opposite of last time's setup
ResponseHeadersPolicy lets you apply security headers in bulk, regardless of the origin's implementation. Even for a static site, designing the CSP to explicitly deny anything beyond self-hosted resources holds up well for future expansion
Don't conflate error responses for SPA routing rewrites with visibility into genuine failures. 403/404 become 200; 5xx keeps its status code — both are "show a friendly page," but they mean very different things
CLOUDFRONT-scoped WAFv2 is pinned to us-east-1. This constraint applies regardless of which region the distribution itself lives in, and it never goes away. Separating the feature on/off toggle from the region-constraint wiring via crossRegionReferences: true makes future changes more resilient
defaultAction: block pairs naturally with a design where even allow-listed IPs still get inspected for attack patterns once. Allowing before managed-rule evaluation should stay the exception, not the norm

References

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS CDK 100 Drill Exercises #012: S3 Static Web Site — Getting a "Public" Bucket Policy Past Block Public Access

TOMOAKI ishihara — Sat, 29 Aug 2026 07:27:43 +0000

Introduction

This is the 12th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

This time we build the simplest way to publish a static site on AWS — S3 static website hosting — with no CDN, no WAF, and no custom domain. It looks like a plain, unremarkable pattern, but implementing it surfaces one genuinely interesting question.

"A bucket policy that allows s3:GetObject to Principal: * should count as 'public' — so why does deploying it against a bucket with BlockPublicAccess.BLOCK_ALL enabled succeed?"

The short answer: S3's Block Public Access doesn't decide public vs. private purely based on whether Principal is *. This time we verify that behavior with actual CDK code.

What you'll learn in this article

The basics of S3 static website hosting (websiteIndexDocument/websiteErrorDocument)
How the aws:SourceIp condition exempts a "public" policy from S3's Block Public Access evaluation
Why enforceSSL can't be enabled, because the S3 website endpoint doesn't support HTTPS
How to auto-detect the deploy operator's own global IP so the site is viewable immediately after deployment
The limits of this pattern — what you can and can't do without CloudFront

📁 Code repository: GitHub

Architecture Overview

Trait	Benefit
No CloudFront, no WAF, no ACM certificate	The fastest, cheapest way to get a static site online. A solid baseline before moving on to `cloudfront-s3-static-website`, covered next time
An IP allow list that coexists with Block Public Access	Lets you actually verify that a `Principal: *` policy scoped with an `aws:SourceIp` condition is exempted from S3's "public" determination
Automatic detection of the operator's IP	Detects the deploying machine's own IP via `curl`, so the site is viewable right after deployment with no manual configuration

Data Flow

Browser
  │  HTTP (the S3 website endpoint doesn't support HTTPS)
  ▼
S3 bucket website endpoint (<bucket>.s3-website-<region>.amazonaws.com)
  │  Evaluated by bucket policy: source IP must match allowedIps/allowedIpv6s
  ▼
index.html / error.html

Every request is anonymous. The website endpoint doesn't support SigV4 authentication, so the only access control available is the bucket policy's aws:SourceIp condition.

Implementation Highlights

1. `websiteIndexDocument` alone doesn't finish the job — the tug-of-war with Block Public Access

Just enabling S3 static website hosting is only a few lines of CDK code.

new s3.Bucket(this, 'WebsiteBucket', {
  websiteIndexDocument: 'index.html',
  websiteErrorDocument: 'error.html',
});

But that alone leaves nobody able to view the site. New S3 buckets have public access blocked by default (at the account level), and this reference implementation also sets blockPublicAccess: BLOCK_ALL on the bucket explicitly (via the repo's shared bucket helper) — so every anonymous s3:GetObject gets denied. Simply turning Block Public Access off, though, risks unintentionally opening the bucket to the entire world.

Instead, the stack keeps blockPublicAccess: BLOCK_ALL in place and adds only a bucket policy scoped down by aws:SourceIp.

websiteBucket.addToResourcePolicy(new iam.PolicyStatement({
  effect: iam.Effect.ALLOW,
  principals: [new iam.AnyPrincipal()],
  actions: ['s3:GetObject'],
  resources: [websiteBucket.arnForObjects('*')],
  conditions: {
    IpAddress: {
      'aws:SourceIp': allowedIps.map(ip => `${ip}/32`),
    },
  },
}));

At first glance, this policy — which includes Principal: * — looks like it should get the PutBucketPolicy call itself rejected by Block Public Access's BlockPublicPolicy: true. In practice, though, the deployment succeeds.

Why: when S3's Block Public Access decides "is this policy public?", it looks at more than just whether Principal is * — it also inspects the Condition clause. To be judged non-public, a statement has to be scoped to a fixed value of one of a specific set of keys: aws:SourceIp, aws:SourceArn, aws:SourceVpc, aws:SourceVpce, aws:SourceOwner, aws:SourceAccount, aws:PrincipalOrgID, s3:DataAccessPointArn, s3:DataAccessPointAccount, or aws:userid. A Principal: * policy whose reachable callers are pinned down by an aws:SourceIp CIDR condition therefore isn't treated as "public." (One caveat: an aws:SourceIp range broader than /8 for IPv4 — e.g. 0.0.0.0/1 — is still considered public. The /32 host addresses this stack uses are well within bounds.)

This behavior also comes up frequently in the context of SCPs and IAM Access Analyzer, but actually deploying an IP-restricted bucket policy against a Block-Public-Access-enabled bucket in CDK and confirming it firsthand deepens the understanding considerably.

⚠️ Note: if you omit both allowedIps and allowedIpv6s, no bucket policy is added at all. blockPublicAccess: BLOCK_ALL still applies, so the result is a site that's "unreachable by anyone," not "publicly open." This stack has no "allow everyone if unspecified" fallback.

2. `enforceSSL: false` — a setting that looks wrong at first glance

Every other bucket in this repository follows a baseline policy of enforceSSL: true, rejecting HTTP connections. WebsiteBucket is the one exception, set to enforceSSL: false.

const bucket = new s3.Bucket(scope, id, {
  enforceSSL: false, // the website endpoint only supports HTTP
  websiteIndexDocument: 'index.html',
  websiteErrorDocument: 'error.html',
  // ...
});

The S3 static website hosting endpoint (*.s3-website-<region>.amazonaws.com) doesn't support HTTPS at all, to begin with. Set enforceSSL: true here, and the bucket policy would reject the very (plaintext HTTP) requests the website endpoint actually receives, breaking the site entirely.

In other words, "I want this over HTTPS" isn't something this bucket can solve on its own — it's solved by putting CloudFront in front of it. That's exactly the bridge to cloudfront-s3-static-website, covered in the next article.

3. Auto-detecting the deploy operator's IP so the site is instantly viewable

bin/s3-static-web-site.ts fetches the deploying machine's own global IP via curl and feeds it straight into the bucket policy's allow list.

const allowedIps = parseIpListEnv(process.env.ALLOWED_IPS) ?? [getMyGlobalIp()];
const allowedIpv6s = parseIpListEnv(process.env.ALLOWED_IPV6S) ?? (() => {
  const myIpv6 = getMyGlobalIpv6();
  return myIpv6 ? [myIpv6] : undefined;
})();

IPv6 detection (getMyGlobalIpv6()) returns undefined quietly when it can't be determined (e.g. in a devcontainer/CI environment with no IPv6 connectivity), while IPv4 detection (getMyGlobalIp()) throws an exception on failure. That means deploying from a fully offline machine fails unless you explicitly set the ALLOWED_IPS environment variable. The same override applies when the machine you're deploying from differs from the machine whose browser will view the site (e.g. deploying from a devcontainer but browsing from the host PC) — use ALLOWED_IPS/ALLOWED_IPV6S explicitly in that case too.

ALLOWED_IPS=203.0.113.10,203.0.113.20 \
ALLOWED_IPV6S=2001:db8::1 \
npm run stage:deploy:all

Deploy & Verify

export PROJECT=your-project
export ENV=dev

npm run bootstrap   # first time only
npm run stage:deploy:all

curl http://<output WebsiteBucketUrl>/

Access from an IP not on the allow list, and the S3 website endpoint returns an HTTP 403 (Access Denied) page.

Cost Estimate

💰 Rough monthly estimate (Tokyo region, low traffic)

Service	Usage	Rough monthly cost
S3 storage	A few MB of static content	Under $0.01
S3 requests	Low-volume GET	Under $0.01
S3 server access logs	Small volume	Under $0.01

Total: under $0.05/month

With no CloudFront, no WAF, and no compute in the picture at all, the cost is essentially just S3 storage and request charges.

Summary

What we learned from this pattern:

S3's Block Public Access evaluation doesn't decide public/private purely by whether Principal is *. Statements scoped by a specific set of condition keys that includes aws:SourceIp are exempted from that "public" determination
The S3 static website hosting endpoint is HTTP-only. Enabling enforceSSL breaks the site entirely — going HTTPS requires an additional layer such as CloudFront
The "fallback when unspecified" behavior isn't consistent across helpers. IPv6 detection quietly returns undefined on failure, while IPv4 detection throws. When using helper functions, design your error handling around this asymmetry rather than assuming both behave the same
An IP allow list is not a substitute for authentication. aws:SourceIp is only evaluated per request, so while it's useful as a restriction for demos or personal use, it's insufficient as access control tied to a specific individual

Next time, we'll evolve this same workspace into CloudFront + WAF + OAC to add HTTPS and a private origin, in cloudfront-s3-static-website.

References

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS CDK 100 Drill Exercises #011: A CodePipeline for CloudFront & S3 — and a self-reference bug

TOMOAKI ishihara — Sat, 29 Aug 2026 07:12:17 +0000

Introduction

This is the 11th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

This time, the pattern itself is a plain one: a CodePipeline that pulls a static site from CodeCommit, builds it with CodeBuild, uploads it to S3, and invalidates CloudFront. Nothing exotic. What made this one worth writing up is what happened when I actually wired the test suite up and ran it for the first time: the synthesized CloudFormation template came back with

Template is undeployable, these resources have a dependency cycle: ... -> PipelineXXXX -> PipelineXXXX

The pipeline was referencing itself.

What you'll learn in this article

A CodePipeline with no hand-written pipeline or build role — each pipeline action grants itself only the resource-scoped permissions it needs (the invoked Lambdas still carry their own execution-role policies)
Why "upload" and "delete stale files" are worth splitting into two separate pipeline stages instead of one aws s3 sync --delete
The CodePipeline continuation-token pattern for polling a long-running operation (CloudFront invalidation) without blocking a Lambda invocation
How a CDK token used inside a pipeline's own stage configuration can create a genuine CloudFormation self-reference cycle — and how to avoid it
Two more deploy-time landmines that only show up once you actually run Template.fromStack(): an S3 bucket name that isn't lowercase, and a notification rule with zero targets

📁 Code repository: GitHub

Architecture Overview

Feature	Benefit
No custom pipeline/build IAM roles	Every action grants itself exactly the resource-scoped permissions it needs when bound to the pipeline
Upload/cleanup split across stages	`S3DeployAction` uploads; a dedicated Lambda removes what's no longer in the build
Continuation-token invalidation	CloudFront invalidation status is polled via CodePipeline's own re-invocation mechanism, not a blocking wait
Opt-in approval + notifications	A single `approvalTopicArn` parameter turns on both a Manual Approval stage and pipeline-result notifications

Data Flow

CodeCommit (push to branch)
    │  EventBridge rule triggers the pipeline
    ▼
CodePipeline
  ├─ Source              : CodeCommitSourceAction → SourceOutput artifact
  ├─ Build (optional)    : CodeBuildAction (buildspec.yml) → BuildOutput artifact   ← enabled by envParams.enableBuild
  ├─ Approval (optional) : ManualApprovalAction, notifies approvalTopicArn
  ├─ Deploy              : S3DeployAction → upload the build to the target bucket
  ├─ Sync                : Lambda → delete target-bucket objects not present in the build
  └─ InvalidateCache     : Lambda → CloudFront CreateInvalidation, polled via continuation token

The Build stage is opt-in (envParams.enableBuild, off by default in the reference config). When it's disabled, the source artifact is deployed to S3 directly and the Deploy/Sync stages consume SourceOutput instead of BuildOutput.

Implementation Highlights

1. Let CDK grant IAM permissions per action — don't hand-roll a pipeline role

Every CodePipeline action construct (CodeCommitSourceAction, CodeBuildAction, S3DeployAction, LambdaInvokeAction) grants itself the exact, resource-scoped permissions it needs the moment it's bound to a stage. So the pipeline is created with no custom role at all:

const pipeline = new codepipeline.Pipeline(this, 'Pipeline', {
  pipelineName: pipelineName,
  restartExecutionOnUpdate: true,
  artifactBucket: artifactBucket,
  // no `role` — CDK creates one and grants each action's permissions as stages are added
});

This reference implementation originally had a hand-written pipeline role with artifactBucket.grantReadWrite(), repository.grantRead(), and three addToPolicy() calls including codedeploy:* and codestar-notifications:* on Resource: '*'. None of those wildcard grants were actually needed — nothing in the pipeline used CodeDeploy, and CodeStarNotifications doesn't check permissions on the pipeline's own role at all. Once I traced through what each action construct grants on bind() (CodeBuildAction scopes codebuild:BatchGetBuilds/StartBuild/StopBuild to the project ARN, S3DeployAction calls bucket.grantWrite() on the deploy target and grantRead() on the artifact bucket, LambdaInvokeAction calls lambda.grantInvoke(), and the Pipeline construct itself calls artifactBucket.grantReadWrite(this.role) internally), the entire hand-written pipeline role and its policies could just be deleted.

The two Lambdas the pipeline invokes are a separate matter: their execution roles still get explicit, resource-scoped policies in the stack (s3:ListBucket/GetObject/PutObject/DeleteObject on the target bucket for the sync Lambda; cloudfront:CreateInvalidation/GetInvalidation and sns:Publish for the invalidation Lambda), plus an artifactBucket.grantRead() so the sync Lambda can read the input artifact. LambdaInvokeAction only grants the pipeline role permission to invoke the function — it doesn't touch the function's own role.

2. Split "upload" from "cleanup" instead of one `aws s3 sync --delete`

// Deploy stage: upload/overwrite only
new codepipeline_action.S3DeployAction({
  actionName: 'S3_Deploy',
  bucket: cdk.aws_s3.Bucket.fromBucketName(this, 'TargetBucket', props.envParams.deploymentTargetBucketName),
  input: buildOutput,
});

// Sync stage: a separate Lambda removes stale objects afterward
new codepipeline_action.LambdaInvokeAction({
  actionName: 'Lambda_S3_Sync',
  lambda: s3SyncLambda,
  inputs: [buildOutput],
  userParameters: { DEST_BUCKET_NAME: props.envParams.deploymentTargetBucketName },
});

S3DeployAction is a managed action, so the common "upload the build" case needs no custom code at all. Deletion — which requires diffing the new build's file list against what's already in the bucket — is isolated in its own Lambda, testable and replaceable independently of the upload step.

3. The continuation-token pattern for CloudFront invalidation

CloudFront invalidations can take minutes to complete. Instead of a Lambda that polls in a loop until it's done (risking a timeout), the invalidation Lambda reports a continuationToken back to CodePipeline (via put_job_success_result), and CodePipeline re-invokes it until it reports success:

def lambda_handler(event, context):
    job_id = event['CodePipeline.job']['id']
    job_data = event['CodePipeline.job']['data']

    if 'continuationToken' in job_data:
        # Re-invoked: check on the invalidation we created last time
        continuation_token = json.loads(job_data['continuationToken'])
        invalidation_id = continuation_token['InvalidationId']
        status = monitor_invalidation_state(distribution_id, invalidation_id)
        if status != 'Completed':
            continue_job_later(job_id, invalidation_id)  # ask CodePipeline to call us again
        else:
            put_job_success(job_id)
    else:
        # First invocation: create the invalidation and ask to be re-invoked
        invalidation_id = create_invalidation(distribution_id)
        continue_job_later(job_id, invalidation_id)

Each invocation is short-lived; CodePipeline itself handles the "come back later and check again" scheduling. On completion (or failure) the Lambda also publishes a short status message to a dedicated SNS topic the stack always creates — that notification is independent of the opt-in approvalTopicArn feature below.

4. The bug: a pipeline that referenced itself

The InvalidateCache Lambda's userParameters originally included the pipeline's own name, read straight off the construct:

// Before — looks harmless
userParameters: {
  "PIPELINE_NAME": pipeline.pipelineName,
  "DISTRIBUTION_ID": props.envParams.cloudfrontDistributionId,
}

pipeline.pipelineName is a CDK token that resolves to { "Ref": "PipelineXXXX" }. And this userParameters blob is itself embedded inside one of the Pipeline resource's own Stages[].Actions[].Configuration properties — because the InvalidateCache action is a stage of that very pipeline. The rendered template ends up with the Pipeline resource containing a Ref to itself, which Template.fromStack()'s cyclic-dependency checker correctly rejects:

Template is undeployable, these resources have a dependency cycle: ... -> PipelineXXXX -> PipelineXXXX

I never hit this while the stack only existed in bin/, because nothing ever called Template.fromStack() on it. The moment I wrote a real snapshot test, it surfaced immediately. The fix is to compute the name as a plain string before constructing the Pipeline, and reuse that instead of the token:

// Fixed — a real string, not a Ref back to the Pipeline resource
const pipelineName = `${props.project}-${props.environment}-pipeline`;

const pipeline = new codepipeline.Pipeline(this, 'Pipeline', {
  pipelineName: pipelineName,
  // ...
});

// later, in the InvalidateCache stage:
userParameters: {
  "PIPELINE_NAME": pipelineName, // plain string, no cycle
  "DISTRIBUTION_ID": props.envParams.cloudfrontDistributionId,
}

The general lesson: any token that resolves to Ref/Fn::GetAtt on a construct is safe to use in most places, but if it ends up embedded inside that same construct's own properties (because the reference lives inside an action/stage that belongs to that construct), CloudFormation sees a resource pointing at itself. If you already have the plain value on hand before construction, prefer it.

5. Two more bugs that only a real `Template.fromStack()` run would catch

An S3 bucket name that wasn't guaranteed lowercase.

// Before
bucketName: `${props.project}-${props.environment}-artifact-bucket`,

S3 bucket names must be lowercase. A props.project of "TestProject" produced InvalidBucketNameValue at synth time. The fix, matching the convention already used elsewhere in this repo:

bucketName: `${props.project}-${props.environment}-artifact-bucket`.toLowerCase(),

A NotificationRule that could get zero targets.

The pipeline notification rule is only meaningful when an approval/notification topic is configured. The original code always created it, with an empty targets: [] when no topic was set:

// Before — deploys fine when a topic is set, fails when it isn't
new codestar_notification.NotificationRule(this, 'PipelineNotificationRule', {
  // ...
  targets: props.envParams.approvalTopicArn
    ? [sns.Topic.fromTopicArn(this, 'NotificationTopic', props.envParams.approvalTopicArn)]
    : [],
});

AWS::CodeStarNotifications::NotificationRule requires at least one target — an empty list deploys to a CloudFormation error, not a synth-time one, which makes it even easier to miss in local testing. The fix is to skip creating the resource entirely when there's no topic to notify:

if (props.envParams.approvalTopicArn) {
  new codestar_notification.NotificationRule(this, 'PipelineNotificationRule', {
    // ...
    targets: [sns.Topic.fromTopicArn(this, 'NotificationTopic', props.envParams.approvalTopicArn)],
  });
}

The same if also gates the Manual Approval stage, so a single optional parameter (approvalTopicArn) turns the entire "gate deployments and notify someone" feature on or off per environment.

Deploy & Verify

# The npm scripts read $PROJECT / $ENV and target the AWS named profile ${PROJECT}-${ENV}
export PROJECT=your-project
export ENV=dev

npm run bootstrap   # first time only
npm run deploy:all

# Push to the configured branch to trigger the pipeline, then check its status
aws codepipeline get-pipeline-state --name <project>-<env>-pipeline

Cost Estimate

💰 Rough monthly estimate (Tokyo region, ~20 pipeline executions/month)

Service	Usage	Rough monthly cost
CodePipeline (V2 pricing)	~20 runs	~$0.40
CodeBuild (`BUILD_GENERAL1_SMALL`, ~3 min/build) — only when `enableBuild`	~20 builds	~$0.30
Lambda (Sync + Invalidate)	Light invocations	~$0.05
S3 artifact bucket	A few MB	< $0.05
CloudWatch Logs (2 log groups, or 3 with the build stage; 1-week retention)	Light volume	< $0.05

Total: roughly $1/month with the build stage enabled, ~$0.55 without

Cost scales with pipeline executions, not idle time — nothing runs between deploys.

Summary

What we learned from this pattern:

Let CDK's action constructs grant their own IAM permissions. A hand-written pipeline role tends to accumulate wildcard grants that nothing in the pipeline actually uses
Split "upload" and "cleanup" across stages when a managed action (S3DeployAction) covers the common case and only the edge case (stale-object deletion) needs custom code
The continuation-token pattern lets a Lambda-backed pipeline action represent a long-running operation without blocking on it
A token that resolves to a construct's own Ref can create a self-reference if it ends up embedded inside that construct's own properties — compute plain values ahead of time when they're going to be reused inside the same construct's actions
Template.fromStack() is the fastest way to catch deploy-time-only failures (bucket naming rules, resources with mandatory non-empty properties, dependency cycles) — none of the three bugs in this article showed up until the test suite actually synthesized and validated the template
Guard optional child resources with the same condition as the feature they belong to — a NotificationRule with zero targets and a Manual Approval stage that's never wired to a topic are two symptoms of the same missing guard

References

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS CDK 100 Drill Exercises #009: CloudWatch Logs S3 Archive — Comparing 3 Log Archive Patterns in One Repository

TOMOAKI ishihara — Mon, 17 Aug 2026 17:24:58 +0000

Introduction

This is the 9th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

Leave CloudWatch Logs alone and anything past its retention period simply disappears. When you need to keep logs around for audits or incident investigation, most teams eventually land on "archive it to S3" — but there's more than one way to get there.

This time we implement three different patterns for archiving CloudWatch Logs to S3 as five independent CDK stacks, and compare the latency, cost, and operational-simplicity trade-offs with concrete code.

What you'll learn in this article

Near-real-time log archiving with Kinesis Data Firehose (via a subscription filter)
Tiered storage-class transitions with S3 lifecycle rules (Standard → IA → Glacier IR → Deep Archive)
How to bolt archiving onto an existing CloudWatch Logs log group after the fact
Scheduled batch export with the CloudWatch Logs Export Task API (and how to work around the one-concurrent-execution limit)
Decoding the CWL payload in a Lambda subscription filter and writing it to S3 in a custom format
How the required IAM trust relationships and bucket policies differ between patterns for the CloudWatch Logs service

📁 Code repository: GitHub

Architecture Overview

Pattern	Approach	Stacks
A: Firehose	Subscription filter → Kinesis Data Firehose → S3	Basic / Lifecycle / Existing (3 stacks)
B: Export Task	EventBridge Scheduler → Lambda → Export Task API → S3	Export (1 stack)
C: Direct Lambda write	Subscription filter → Lambda → S3	Lambda (1 stack)

Why three patterns

Trait	Pattern A (Firehose)	Pattern B (Export Task)	Pattern C (Lambda)
Delivery latency	Near real-time (seconds to minutes)	Scheduled (tolerates up to ~12h of delay)	Near real-time
Cost at high volume	Moderate (Firehose + S3)	Low (Lambda + S3 only)	Moderate (scales with Lambda invocation count)
Output format	Raw CWL JSON lines, GZIP-compressed	Raw CWL format	Custom JSON (fully controllable)
Operational simplicity	High (managed Firehose)	High (managed Export API)	Low (Lambda code needs maintenance)
Custom transformation	Limited (depends on Firehose's data transformation feature)	None	Fully controllable inside Lambda

In short: A if you need near-real-time delivery, B if a cost-optimized batch job is good enough, C if you want to define the output format yourself.

Data Flow

Pattern A (Stacks 1-3)
CloudWatch Log Group
  → Subscription filter (SubscriptionFilter + FirehoseDestination, role = CwlToFirehoseRole)
  → Kinesis Data Firehose (GZIP compression)
  → S3 archive bucket
  * Stack 1 (Basic) shares 5 log groups and uses dynamic partitioning on owner/logGroup
    to route each log group to its own S3 prefix. Stack 2 (Lifecycle) uses a single log
    group with a plain date prefix and no dynamic partitioning.

Pattern B (Stack 4)
EventBridge Scheduler
  → Lambda (calls logs:CreateExportTask for the previous day's logs)
  → CloudWatch Logs Export Task API
  → S3 archive bucket (bucket policy allows writes from logs.amazonaws.com)

Pattern C (Stack 5)
CloudWatch Log Group
  → Subscription filter (LambdaDestination, CDK manages invoke permissions automatically)
  → Lambda (decodes the gzip+base64 CWL payload and writes JSON to S3)
  → S3 archive bucket

Key Components

Component	Design point
S3 archive bucket	SSE-S3 encryption, versioning enabled, public access blocked, SSL enforced. Provisioned independently per stack (5 buckets total)
CwlToFirehoseRole	Assumed by `logs.amazonaws.com`. Scoped to log groups via an `aws:SourceArn` condition
FirehoseRole	Assumed by `firehose.amazonaws.com`. Grants the standard Firehose→S3 permission set (`PutObject`, `GetObject`, `ListBucket`, multipart upload actions) scoped to the archive bucket
Dynamic partitioning (Stack 1)	jq extracts `owner`/`logGroup` to route 5 shared log groups into separate S3 prefixes. `CloudWatchLogProcessor` can't be used because it strips those fields; a jq fallback is required to handle `CONTROL_MESSAGE` records
Tiered lifecycle rules (Stack 2)	Standard → IA (30 days) → Glacier IR (90 days) → Deep Archive (365 days) → expiration (7 years)
Importing an existing log group (Stack 3)	Referenced via `LogGroup.fromLogGroupName()`. No new `AWS::Logs::LogGroup` resource is created
Export Task Lambda (Stack 4)	A concurrency guard that checks for running/pending tasks before calling `CreateExportTask`
Direct Lambda write (Stack 5)	`LambdaDestination` automatically manages invoke permissions. The payload is decoded and saved as custom JSON

Implementation Highlights

1. Pattern A — Kinesis Data Firehose subscription

For CloudWatch Logs to deliver to Firehose, it needs an explicit IAM role. We split responsibilities across two roles.

// CwlToFirehoseRole: assumed by logs.amazonaws.com. Only the trust policy is defined here
const cwlToFirehoseRole = new iam.Role(this, 'CwlToFirehoseRole', {
    assumedBy: new iam.ServicePrincipal('logs.amazonaws.com', {
        conditions: {
            StringLike: {
                'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*`,
            },
        },
    }),
});

// L2 SubscriptionFilter + FirehoseDestination(role: cwlToFirehoseRole)
const filterPattern = logGroupParams.filterPattern ?? defaultLogGroupArchiveConfig.filterPattern;
new logs.SubscriptionFilter(this, 'CwlSubscriptionFilter', {
    logGroup: this.logGroup,
    destination: new logs_destinations.FirehoseDestination(this.deliveryStream, {
        role: cwlToFirehoseRole,
    }),
    filterPattern: filterPattern
        ? logs.FilterPattern.literal(filterPattern)
        : logs.FilterPattern.allEvents(),
});

Why not attach firehose:PutRecord/PutRecordBatch permissions directly on the role?
Inside bind(), aws-logs-destinations's FirehoseDestination calls deliveryStream.grantPutRecords(role) on the role you pass in (or one it generates), automatically granting the necessary permissions. If you also add your own inline policy to the role, you end up with two AWS::IAM::Policy resources granting the same permission. The trick is to define only the trust policy yourself (scoped by SourceArn) and let the L2 construct handle the permission grant.

I used to write this with the L1 CfnSubscriptionFilter plus a hand-rolled inline policy, on the assumption that "the L2 SubscriptionFilter can't accept a roleArn for Firehose." That assumption was wrong. In the current aws-cdk-lib (2.261.0 in this repository), FirehoseDestination wires the role's ARN all the way through to CfnSubscriptionFilter for you, so the L2 construct alone is sufficient.

The aws:SourceArn condition scopes who can assume this role down to "log groups within this account/region." Combined with scoping the access policy side to just this specific Firehose delivery stream, permissions are minimized in two independent layers.

1a. Per-log-group dynamic partitioning (Stack 1 – Basic)

Stack 1 creates 5 log groups that share the same Firehose delivery stream. Without partitioning, events from all five would end up mixed together in the same S3 object, so dynamic partitioning routes each log group to its own prefix.

const s3Destination = new firehose.S3Bucket(archiveBucket, {
    dataOutputPrefix:
        'AWSLogs/!{partitionKeyFromQuery:owner}/CWLogGroup/!{partitionKeyFromQuery:logGroup}/!{timestamp:yyyy/MM/dd/HH}/',
    dynamicPartitioning: { enabled: true },
    bufferingInterval: cdk.Duration.seconds(60), // dynamic partitioning requires >= 60s
    bufferingSize: cdk.Size.mebibytes(64),        // and >= 64MiB
    processors: [
        new firehose.DecompressionProcessor({
            compressionFormat: firehose.DecompressionProcessorCompressionFormat.GZIP,
        }),
        firehose.MetadataExtractionProcessor.jq16({
            owner: '(if (.owner // "") == "" then "controlmessages" else .owner end)',
            logGroup:
                '(if (.logGroup // "") == "" then "controlmessages" else (.logGroup | ltrimstr("/")) end)',
        }),
        new firehose.AppendDelimiterToRecordProcessor(),
    ],
});

I hit two snags here.

First: don't add the CloudWatchLogProcessor (message extraction). Firehose has a "message extraction" feature that pulls just the contents of message out of a decompressed CWL record — but enabling it wipes out owner/logGroup/logStream entirely, leaving only the raw contents of message. With those gone, the downstream MetadataExtractionProcessor fails with DynamicPartitioning.MetadataExtractionFailed when it tries to reference .owner/.logGroup via jq. Right after DecompressionProcessor, the record still has owner/logGroup at the top level (see Fig. 1 of Firehose's "message extraction" documentation), so jq can read them directly without a message extraction processor.

Second: handling the CONTROL_MESSAGE records CloudWatch Logs periodically sends as a health check. These records have owner/logGroup set to the empty string "", so a naive .logGroup | ltrimstr("/") returns the empty string as-is, which fails with partitionKeys values must not be null or empty. You need a jq if/then/else to supply a fallback value ("controlmessages"). It's tempting to reach for // (the alternative operator) here, but // only substitutes on null/false, not on an empty string.

The trade-off is that each record written to S3 is the entire CWL envelope (with a logEvents array that may contain multiple events), not a flat one-row-per-log-event line. Flattening (CloudWatchLogProcessor) is incompatible with metadata-based partitioning. If you need individual messages, UNNEST/json_extract the logEvents array in something like Athena.

2. Tiered S3 lifecycle (Stack 2 – Lifecycle)

For long-term archives, you can optimize cost by moving objects to progressively cheaper storage classes as access frequency drops.

archiveBucket.addLifecycleRule({
    id: `MoveToIAAfter${moveToIaAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.INFREQUENT_ACCESS, transitionAfter: cdk.Duration.days(moveToIaAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `MoveToGlacierAfter${moveToGlacierAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.GLACIER_INSTANT_RETRIEVAL, transitionAfter: cdk.Duration.days(moveToGlacierAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `MoveToDeepArchiveAfter${moveToDeepArchiveAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.DEEP_ARCHIVE, transitionAfter: cdk.Duration.days(moveToDeepArchiveAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `ExpireCurrentObjectsAfter${expireAfterDays}Days`,
    enabled: true,
    expiration: cdk.Duration.days(expireAfterDays),
});

Non-current versions (older objects overwritten while versioning is enabled) get their own IA transition and expiration rule, and incomplete multipart uploads are aborted after 7 days. Rather than bundling everything into one giant lifecycle configuration, each transition gets its own addLifecycleRule so parameters like moveToIaAfterDays can be tuned per environment independently.

3. Attaching to an existing log group (Stack 3 – Existing)

Sometimes you don't want to create a new log group — you want to bolt archiving onto a log group an application is already writing to.

const existingLogGroup = logs.LogGroup.fromLogGroupName(
    this,
    'ExistingLogGroup',
    existingLogGroupParams.logGroupName,
);

fromLogGroupName() only references the log group in CDK — it doesn't create an AWS::Logs::LogGroup resource. Since you can pass this existing log group's ARN (existingLogGroup.logGroupArn) straight into the trust policy's aws:SourceArn condition, the subscription filter and IAM role wiring can reuse the exact same code as Stacks 1 and 2.

4. Pattern B — Scheduled Export Task

The CloudWatch Logs Export Task API has a one concurrent execution per account limit. Before submitting a new task, the Lambda checks for running or pending tasks to avoid errors.

# export-task/index.py (core logic)
def lambda_handler(event, context):
    logs_client = boto3.client("logs")

    # Skip if a task is already running or pending
    running = logs_client.describe_export_tasks(statusCode="RUNNING")
    pending = logs_client.describe_export_tasks(statusCode="PENDING")
    if running["exportTasks"] or pending["exportTasks"]:
        print("An export task is already running or pending. Skipping.")
        return {"status": "skipped", "reason": "task_in_progress"}

    # Export the previous day's (UTC) logs
    yesterday = datetime.datetime.utcnow() - datetime.timedelta(days=1)
    from_time = int(yesterday.replace(hour=0, minute=0, second=0, microsecond=0).timestamp() * 1000)
    to_time = int(yesterday.replace(hour=23, minute=59, second=59, microsecond=999999).timestamp() * 1000)

    response = logs_client.create_export_task(
        taskName=f"export-{yesterday.strftime('%Y-%m-%d')}",
        logGroupName=LOG_GROUP_NAME,
        fromTime=from_time,
        to=to_time,
        destination=S3_BUCKET_NAME,
        destinationPrefix=f"{S3_PREFIX}/{yesterday.strftime('%Y/%m/%d')}",
    )
    return {"status": "created", "taskId": response["taskId"]}

The key detail is checking PENDING in addition to RUNNING. If a scheduled task is merely sitting in the queue and you submit the next one anyway, you get a LimitExceededException.

The S3 bucket side needs two resource policies allowing writes from the CloudWatch Logs service principal.

// 1. Permission to check the bucket ACL before exporting
archiveBucket.addToResourcePolicy(
    new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        principals: [new iam.ServicePrincipal('logs.amazonaws.com')],
        actions: ['s3:GetBucketAcl'],
        resources: [archiveBucket.bucketArn],
        conditions: {
            ArnLike: { 'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*` },
        },
    }),
);

// 2. Permission to write the exported objects
archiveBucket.addToResourcePolicy(
    new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        principals: [new iam.ServicePrincipal('logs.amazonaws.com')],
        actions: ['s3:PutObject'],
        resources: [archiveBucket.arnForObjects('*')],
        conditions: {
            StringEquals: { 's3:x-amz-acl': 'bucket-owner-full-control' },
            ArnLike: { 'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*` },
        },
    }),
);

The condition pinning s3:x-amz-acl to bucket-owner-full-control is explicitly required by CloudWatch Logs' official documentation. Omit it and the export task fails with a permissions error.

5. Pattern C — Subscription filter → Lambda → S3

With LambdaDestination, CDK automatically manages the resource policy (Lambda::Permission) that lets logs.amazonaws.com invoke your Lambda.

new logs.SubscriptionFilter(this, 'CwlSubscriptionFilter', {
    logGroup: this.logGroup,
    destination: new logDestinations.LambdaDestination(this.archiveFunction),
    filterPattern: filterPattern
        ? logs.FilterPattern.literal(filterPattern)
        : logs.FilterPattern.allEvents(),
});

On the Lambda side, the code decodes the CWL compressed payload and writes it to S3 in a JSON format it defines itself, per invocation.

# cwl-to-s3/index.py (core logic)
def lambda_handler(event, context):
    compressed = base64.b64decode(event["awslogs"]["data"])
    payload = json.loads(gzip.decompress(compressed).decode("utf-8"))

    if payload.get("messageType") == "CONTROL_MESSAGE":
        # Health-check message CloudWatch Logs sends, e.g. when a subscription starts
        return {"status": "skipped", "reason": "control_message"}

    now = datetime.datetime.utcnow()
    safe_stream = payload["logStream"].replace("/", "_").replace("$", "").replace("[", "").replace("]", "")
    s3_key = f"{S3_PREFIX}/{now.strftime('%Y/%m/%d/%H')}/{safe_stream}_{uuid.uuid4().hex[:8]}.json"

    record = {
        "logGroup": payload["logGroup"],
        "logStream": payload["logStream"],
        "owner": payload.get("owner"),
        "exportedAt": now.isoformat() + "Z",
        "events": payload["logEvents"],
    }
    s3_client.put_object(Bucket=S3_BUCKET_NAME, Key=s3_key, Body=json.dumps(record, ensure_ascii=False).encode("utf-8"))
    return {"status": "ok", "processed": len(payload["logEvents"]), "s3Key": s3_key}

Note the early skip for payloads with messageType == "CONTROL_MESSAGE". CloudWatch Logs sends these health-check control messages right after a subscription starts (among other times), and they have no logEvents key at all — without this check, the invocation fails with a KeyError.

Both patterns A and C deliver via a subscription filter for near-immediate delivery, but with A you can let Firehose handle the transformation, while with C you write the Lambda code yourself and get complete control over the output format in exchange.

Deploy & Verify

export PROJECT=your-project
export ENV=dev

npm run bootstrap -w workspaces/cloudwatch-logs-s3-archive   # first time only
npm run stage:deploy:all -w workspaces/cloudwatch-logs-s3-archive

To deploy individual stacks, use the per-stack stage:deploy:* scripts (each scoped to a cdk deploy stack selector like **/*Basic).

npm run stage:deploy:basic -w workspaces/cloudwatch-logs-s3-archive
npm run stage:deploy:lifecycle -w workspaces/cloudwatch-logs-s3-archive
npm run stage:deploy:export -w workspaces/cloudwatch-logs-s3-archive

# Pattern A Stack 1 (Basic): write test data into 5 log groups (/<project>/<env>/basic-*)
# (given the dynamic-partitioning buffering requirements of >= 60s / >= 64MiB, use the
#  repo's helper script rather than a one-off put-log-events call)
./write-test-logs.sh --project <project> --env <env>

# After ~60s, output appears partitioned under AWSLogs/<owner>/CWLogGroup/<logGroup>/...
aws s3 ls s3://<archive-bucket>/AWSLogs/ --recursive

# Pattern A Stack 2 (Lifecycle): single log group, no dynamic partitioning
./write-test-logs-lifecycle.sh --project <project> --env <env>
aws s3 ls s3://<archive-bucket>/ --recursive

# Pattern B: manually invoke the export Lambda
aws lambda invoke --function-name <project>-<env>-cwl-export-task --payload '{}' response.json
cat response.json

# Pattern C: write a test log entry (the subscription filter invokes the Lambda immediately)
aws logs put-log-events \
  --log-group-name /<project>/<env>/app-lambda \
  --log-stream-name test-stream \
  --log-events timestamp=$(date +%s000),message="hello lambda"
aws s3 ls s3://<archive-bucket>/subscriptions/ --recursive

Cost Estimate

💰 Rough monthly estimate (Tokyo region, 1 GB of log data per day)

Service	Applies to	Rough monthly cost
Kinesis Data Firehose	A	~$0.90 (at $0.03/GB)
Lambda	B, C	Effectively negligible at low invocation counts
S3 (Standard)	All	~$0.75 (at $0.025/GB-month)
CloudWatch Logs (ingestion)	All	~$0.76/GB (charged regardless of which archive pattern you use, and not included in the total below)
EventBridge	B	$1.00 per million events (negligible at minimal frequency)

Rough total for Pattern A's archiving cost alone: about $2/month at 1 GB/day of log volume. CloudWatch Logs ingestion itself (~$23/month at 1 GB/day) is a separate cost you pay upfront, before any archive pattern runs

Running all five stacks simultaneously at production-scale traffic for comparison testing will cost more than the above. Don't forget to clean up afterward with npm run stage:destroy:all -w workspaces/cloudwatch-logs-s3-archive.

Summary

What we learned from this pattern:

Pattern A (Firehose): Minimal code, well-suited for near-real-time archiving. Delivery from CloudWatch Logs requires FirehoseDestination (L2) plus two IAM roles with distinct responsibilities
Pattern B (Export Task): The lowest cost for batch use cases, but limited to one concurrent execution per account. A guard checking for running/pending tasks on the Lambda side is essential
Pattern C (Lambda): Full control over the output format, at the cost of ongoing Lambda code maintenance, including handling control messages
IAM trust relationships differ by service: For Firehose, FirehoseDestination grants permission to the trusted role via grantPutRecords; for Export Task, an S3 bucket policy with an s3:x-amz-acl condition is required; for LambdaDestination, CDK auto-generates the invoke permission — the same "delivery from CloudWatch Logs" concept wires up differently every time
Split tiered lifecycle rules by transition: rather than one giant rule, separate rules make the transitionAfter thresholds easy to tune independently as environment parameters
fromLogGroupName() is enough for bolting archiving onto an existing log group: pass the existing ARN into the trust policy directly and reuse the same wiring, with no new resource created
Firehose dynamic partitioning is incompatible with message extraction: CloudWatchLogProcessor strips owner/logGroup, so drop it if you're using jq to derive partition keys from those fields. You also need to account for CloudWatch Logs' CONTROL_MESSAGE records (where owner/logGroup are empty strings) with a jq if/then/else fallback (// doesn't catch empty strings)

References

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS CDK 100 Drill Exercises #010: Budget Alerts & Cost Anomaly Detection — 5 FinOps Alerting Patterns in CDK

TOMOAKI ishihara — Mon, 17 Aug 2026 15:24:33 +0000

Introduction

This is the 10th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

By the time you notice, AWS costs have often already piled up. You catch the anomaly when the invoice arrives, or you open the console and think "wait, I shouldn't have spent this much this month." AWS gives you several ways to prevent this, but they're actually quite different in nature. Do you want to be told when you cross a known threshold? Do you want to catch unusual activity even while still within budget? Or do you want weekly usage automatically shared with your team? The right mechanism depends on which of these you're actually after.

This time we implement five different patterns for AWS cost alerting as five independent CDK stacks, and compare each one's detection trigger, notification path, and setup complexity with concrete code.

What you'll learn in this article

How to express AWS Budgets threshold notifications as a data-driven { type, thresholdPercent }[] array instead of hardcoding them
The frequency: IMMEDIATE constraint required for SNS delivery in AWS Cost Anomaly Detection, and how to hand-assemble thresholdExpression as a JSON string
How AWS Budgets, Cost Anomaly Detection, and CloudWatch alarms each require SNS publish permissions in their own distinct way (different SourceAccount/SourceArn conditions per service principal)
The hard limit that AWS-managed Cost Anomaly Detection monitors are effectively capped at one or two per account — a constraint I only discovered by actually deploying, and the redesign needed to share one monitor across multiple stacks
The trap where AWS Chatbot's guardrailPolicies defaults to AdministratorAccess when left unset — and why an empty array [] doesn't prevent this (CDK synth strips empty arrays out of the template)
How to use Step Functions' new query language JSONata to format Cost Explorer API results straight into an SNS message with zero Lambda functions
A string-escaping trap that spans three layers — JavaScript → JSON → JSONata (\n happens to work, \s silently breaks)
Six reusable constructs factored out into common/constructs/cost/ for reuse across the whole repository

📁 Code repository: GitHub

Architecture Overview

Pattern	Approach	Stack
A: Budgets	Threshold-based budget alert → SNS → Email	Budget (1 stack)
B: Anomaly Detection	ML-based anomaly detection → SNS → Email	Anomaly (1 stack)
C: Unified	Budgets + Anomaly Detection consolidated into one SNS topic, optionally Slack	Unified (1 stack)
D: Billing Alarm	Classic CloudWatch `EstimatedCharges` alarm → SNS → Email	BillingAlarm (1 stack, pinned to us-east-1)
E: Cost Digest	Scheduled Step Functions → SNS → Slack/Teams	CostDigest (1 stack)

Why five patterns

Trait	A (Budgets)	B (Anomaly Detection)	C (Unified)	D (Billing Alarm)	E (Cost Digest)
Trigger	Reactive — known threshold	Reactive — deviation from historical trend (ML)	Both, reactive	Reactive — cumulative spend threshold	Proactive — scheduled (no threshold needed)
Best suited for	"Tell me when I cross $X/month"	"I want to notice unusual activity even within budget"	Consolidating everything into one channel	The oldest safety net, independent of Cost Explorer	"Share weekly spend with the team"
Notification channel	SNS + Email	SNS + Email	SNS + Email + optional Slack	SNS + Email	SNS + Email + optional Slack/Teams

In short: A to stop at a known ceiling, B to catch unexpected movement, C to consolidate both into one place, D as the last-resort insurance for when you can't even trust Cost Explorer, E for a proactive recurring report.

Data Flow

Pattern A (Stack 1)
CfnBudget (account-wide + service filter)
  → SNS topic (budgets.amazonaws.com allowed to Publish)
  → Email

Pattern B (Stack 2)
CfnAnomalyMonitor (DIMENSIONAL / SERVICE)
  → CfnAnomalySubscription (frequency: IMMEDIATE)
  → SNS topic (costalerts.amazonaws.com allowed to Publish)
  → Email

Pattern C (Stack 3)
CfnBudget + CfnAnomalySubscription (attaches to Stack 2's monitor, creates no new monitor)
  → SNS topic (both service principals allowed)
  → Email + AWS Chatbot → Slack (optional)

Pattern D (Stack 4, pinned to us-east-1)
CloudWatch alarm (AWS/Billing EstimatedCharges)
  → SNS topic (cloudwatch.amazonaws.com allowed to Publish)
  → Email

Pattern E (Stack 5)
EventBridge Scheduler (cron)
  → Step Functions (JSONata): GetCostAndUsage → PublishCostDigest
  → SNS topic (Publish from its own IAM role, no service policy needed)
  → Email + AWS Chatbot → Slack / Microsoft Teams (either or both)

Key Components

Component	Design point
`CostAlertTopic` (shared construct)	SNS topic + resource policy driven by `allowBudgetsPublish`/`allowCostAnomalyDetectionPublish`/`allowCloudWatchAlarmPublish` flags
`CostBudget` (shared construct)	A `CfnBudget` wrapper that dynamically builds `notificationsWithSubscribers` from `BudgetNotificationRule[]`
`CostAnomalyDetection` (shared construct)	`CfnAnomalyMonitor` + `CfnAnomalySubscription`, encapsulating the `thresholdExpression` JSON assembly
`BillingAlarm` (shared construct)	`AWS/Billing EstimatedCharges` metric + alarm. Assumes it's pinned to us-east-1
`SafeSlackChannelConfiguration` / `SafeMicrosoftTeamsChannelConfiguration` (shared constructs)	Safe Chatbot wrappers that fall back to `ReadOnlyAccess` when `guardrailPolicies` isn't specified
Step Functions (Stack 5)	`QueryLanguage.JSONATA`, `tracingEnabled: true`, full execution logging to CloudWatch Logs

Implementation Highlights

1. Data-driven budget notification rules

Rather than hardcoding "80%, 100%, forecasted 100%" into the stack, notification rules are stored as a plain array in the environment parameters.

// common/types/cost.ts
export interface BudgetNotificationRule {
    readonly type: 'ACTUAL' | 'FORECASTED';
    readonly thresholdPercent: number;
}

// parameters/dev-params.ts
budget: {
    amount: 10,
    notifications: [
        { type: 'FORECASTED', thresholdPercent: 100 },
        { type: 'ACTUAL', thresholdPercent: 80 },
        { type: 'ACTUAL', thresholdPercent: 100 },
        { type: 'ACTUAL', thresholdPercent: 200 }, // escalation for runaway costs
    ],
},

This lets any environment add or remove thresholds without touching the stack code. common/constructs/cost/budget.ts just .map()s over the array.

const notificationRules = props.notifications ?? defaultBudgetNotifications;
const notificationsWithSubscribers = notificationRules.map((rule) => ({
    notification: {
        notificationType: rule.type,
        comparisonOperator: 'GREATER_THAN',
        threshold: rule.thresholdPercent,
        thresholdType: 'PERCENTAGE',
    },
    subscribers,
}));

2. SNS topic policies aren't optional — and each service does it differently

Both AWS Budgets and AWS Cost Anomaly Detection publish to SNS as service principals rather than by assuming an IAM role. CloudWatch alarm actions likewise need explicit permission. Without it, notifications fail silently — the configuration looks correct, but nothing ever arrives.

// common/constructs/cost/cost-alert-topic.ts (simplified)
if (props.allowBudgetsPublish) {
    topic.addToResourcePolicy(new iam.PolicyStatement({
        principals: [new iam.ServicePrincipal('budgets.amazonaws.com')],
        actions: ['sns:Publish'],
        conditions: {
            StringEquals: { 'aws:SourceAccount': account },
            ArnLike: { 'aws:SourceArn': `arn:${partition}:budgets::${account}:*` },
        },
    }));
}
if (props.allowCostAnomalyDetectionPublish) {
    // costalerts.amazonaws.com, condition is aws:SourceAccount only (no SourceArn scoping needed)
}
if (props.allowCloudWatchAlarmPublish) {
    topic.grantPublish(new iam.ServicePrincipal('cloudwatch.amazonaws.com'));
}

All three need "permission to publish to SNS," yet the strictness of the conditions is all over the place. Budgets requires both SourceAccount and SourceArn; Cost Anomaly Detection requires only SourceAccount (even AWS's own official samples don't specify SourceArn); and CloudWatch's cloudwatch-actions.SnsAction doesn't even do the kind of automatic grant other alarm actions do — you have to call grantPublish explicitly.

Don't add a customer-managed KMS key to these topics. The troubleshooting documentation for both Budgets and Cost Anomaly Detection explicitly calls out topic encryption as a classic cause of silently-failed notifications, because the service principal would also need kms:GenerateDataKey*/kms:Decrypt granted in the key policy. Here we enforce enforceSSL: true for transport encryption only, leaving at-rest encryption at SNS's default managed encryption.

3. Cost Anomaly Detection's `thresholdExpression`

CfnAnomalySubscription.thresholdExpression is typed as a plain string in aws-cdk-lib. CDK doesn't have a typed representation of Cost Explorer's Expression grammar, so you have to assemble it yourself and JSON.stringify() it.

thresholdExpression: JSON.stringify({
    And: [
        {
            Dimensions: {
                Key: 'ANOMALY_TOTAL_IMPACT_PERCENTAGE',
                MatchOptions: ['GREATER_THAN_OR_EQUAL'],
                Values: [String(thresholdPercentage)],
            },
        },
        {
            Dimensions: {
                Key: 'ANOMALY_TOTAL_IMPACT_ABSOLUTE',
                MatchOptions: ['GREATER_THAN_OR_EQUAL'],
                Values: [String(thresholdAbsoluteUsd)],
            },
        },
    ],
}),

ANDing "percentage relative to expected spend" with "absolute dollar amount" filters out noise like "10x the usual cost, but the absolute amount is only $3."

SNS delivery requires frequency: IMMEDIATE. AWS Cost Anomaly Detection only supports SNS subscribers on IMMEDIATE subscriptions — DAILY/WEEKLY are email-only. If you also want a daily email summary, add a second CfnAnomalySubscription (frequency: DAILY, EMAIL subscriber) pointing at the same monitor.

4. Stack 3 doesn't create its own anomaly monitor — AWS's "one per account" limit

This is a trap I only hit after actually deploying, once the writing was already done. Deploy Stack 3 with Stack 2 already deployed, and it fails reliably with this error:

CREATE_FAILED | AWS::CE::AnomalyMonitor | ...
Resource handler returned message: "null" (HandlerErrorCode: AlreadyExists)

At first I thought "the monitor names are different, so why?" — but the cause wasn't the name, it was an AWS-imposed limit. The CreateAnomalyMonitor API reference itself doesn't mention this constraint; the source is a blog post from the AWS Cloud Financial Management team.

"You can create one AWS services managed monitor plus one additional AWS managed monitor (linked account, cost allocation tag, or cost category) per management account."

"AWS managed monitors for linked accounts, cost allocation tags, and cost categories can only be created in management accounts."

— Extending AWS managed monitors in AWS Cost Anomaly Detection (AWS Cloud Financial Management blog)

Since Stack 2 has already created the account's one and only SERVICE-dimension monitor, Stack 3 trying to create a second one on the same dimension always hits AlreadyExists. Switching monitorDimension to LINKED_ACCOUNT doesn't help either — that runs into the separate constraint of only being creatable in an AWS Organizations management account, which fails in most test accounts anyway.

The fix is simple: don't have Stack 3 create a new monitor — attach a second CfnAnomalySubscription to Stack 2's existing monitor instead. Attaching multiple subscriptions to one monitor is a configuration AWS officially supports.

// common/constructs/cost/anomaly-detection.ts
if (props.existingMonitorArn) {
    this.monitorArn = props.existingMonitorArn;
} else {
    this.monitor = new ce.CfnAnomalyMonitor(this, 'Monitor', { /* ... */ });
    this.monitorArn = this.monitor.attrMonitorArn;
}

this.subscription = new ce.CfnAnomalySubscription(this, 'Subscription', {
    monitorArnList: [this.monitorArn],
    // ...
});

On the Stage side, Stack 2 is instantiated before Stack 3, and anomalyStack.monitorArn is simply passed into Stack 3's props. CDK converts this into a real CloudFormation cross-stack export/import, so there's no need for a hand-written Fn::ImportValue.

const anomalyStack = new BudgetsCostAnomalyDetectionAnomalyStack(this, ..., { ... });

new BudgetsCostAnomalyDetectionUnifiedStack(this, ..., {
    ...commonStackProps,
    anomalyMonitorArn: anomalyStack.monitorArn,
    // ...
});

I'll admit it stings a bit that Stack 3 ended up depending on Stack 2, despite the whole premise being "five independent patterns." But since this is a hard constraint on AWS's side, I decided it's more honest to make the dependency explicit in both code and documentation rather than hide it.

Once the deploy went through, I noticed something else. Asking myself "what's actually different between drillexercises-dev-service-anomaly-subscription (Stack 2) and drillexercises-dev-unified-anomaly-subscription (Stack 3)," the answer turned out to be: they're watching exactly the same thing. Both are attached to the same monitor, and both read thresholdPercentage/thresholdAbsoluteUsd from the same params.anomalyDetection, so by default the thresholds are identical too. That means deploying both Stack 2 and Stack 3 results in two notifications for the same single anomaly — an email via Stack 2, plus an email (and optionally Slack) via Stack 3.

To address this, I made it possible to set a stricter threshold on Stack 3 specifically.

// lib/types/anomaly-params.ts
export interface AnomalyDetectionParams extends AnomalyThreshold {
    readonly monitorDimension?: 'SERVICE' | 'LINKED_ACCOUNT';
    // Escalation threshold for Stack 3. Falls back to the base value if unset
    readonly unifiedEscalation?: AnomalyThreshold;
}

// lib/stacks/budgets-cost-anomaly-detection-unified-stack.ts
const anomalyThreshold = anomalyParams.unifiedEscalation ?? anomalyParams;
new CostAnomalyDetection(this, 'UnifiedAnomalyDetection', {
    existingMonitorArn: props.anomalyMonitorArn,
    thresholdPercentage: anomalyThreshold.thresholdPercentage,
    thresholdAbsoluteUsd: anomalyThreshold.thresholdAbsoluteUsd,
    // ...
});

In dev-params.ts, the base threshold is "20%+ actual cost AND $5+ absolute," while Stack 3's escalation threshold is "50%+ actual cost AND $20+ absolute." This demonstrates a severity split: Stack 2 casts a wide net via email, while Stack 3 (the unified/Slack channel) only receives the more serious anomalies.

To be honest, though, this doesn't eliminate the duplication itself. An anomaly large enough to satisfy both thresholds still triggers two notifications. In real-world usage, the straightforward choice is to use only one of Stack 2 or Stack 3 for anomaly detection. Think of unifiedEscalation purely as a demonstration that "a single monitor can feed multiple severity-tiered subscriptions."

5. AWS Chatbot's guardrail policy default is `AdministratorAccess` (for both Slack and Teams)

This is the biggest "hurts if you don't know it" gotcha of this whole exercise. Both SlackChannelConfigurationProps.guardrailPolicies and CfnMicrosoftTeamsChannelConfigurationProps.guardrailPolicies apply the AWS-managed AdministratorAccess policy by default when left unspecified.

"So can I disable it by passing an empty array []?" — that's the trap. CDK's synth step strips empty list properties out of the synthesized CloudFormation template entirely, so from the API's perspective the property is simply unset, and the AdministratorAccess default kicks in anyway. I only noticed this by actually inspecting cdk synth output.

// common/constructs/cost/safe-slack-channel.ts
guardrailPolicies:
    props.guardrailPolicies && props.guardrailPolicies.length > 0
        ? props.guardrailPolicies
        : [iam.ManagedPolicy.fromAwsManagedPolicyName('ReadOnlyAccess')],

Since Microsoft Teams has no CDK L2 construct at all, SafeMicrosoftTeamsChannelConfiguration also auto-generates the minimal "notification only" IAM role (cloudwatch:Describe*/Get*/List* only) used by AWS's own sample Chatbot policy.

6. Reusable cost constructs (`common/constructs/cost/`)

The pieces above aren't specific to this workspace, so they're factored out into infrastructure/common/constructs/cost/ for reuse across the whole repository.

Construct	Wraps	Used by
`CostAlertTopic`	SNS Topic + conditional resource policy + email subscription	Stack 1, 2, 3, 4
`CostBudget`	`CfnBudget` driven by `BudgetNotificationRule[]`	Stack 1 (x2), Stack 3
`CostAnomalyDetection`	`CfnAnomalyMonitor` + `CfnAnomalySubscription` (skips monitor creation when `existingMonitorArn` is given)	Stack 2, Stack 3
`BillingAlarm`	`AWS/Billing EstimatedCharges` alarm	Stack 4
`SafeSlackChannelConfiguration`	Slack configuration with safe guardrail defaults	Stack 3, Stack 5
`SafeMicrosoftTeamsChannelConfiguration`	Teams configuration with a least-privilege role plus safe guardrails	Stack 5

Stack 5's Step Functions/JSONata logic itself isn't factored out, since the message formatting is tightly coupled to this workspace's specific content; only the Chatbot delivery portion uses the safe wrappers above.

7. The cost digest's JSONata state machine (Stack 5)

Stack 5 generalizes a "post a weekly cost summary to Teams" CloudFormation template originally hand-written for Microsoft Teams so it also supports Slack — the biggest challenge of this exercise. The state machine consists of exactly two states, both written as raw ASL via sfn.CustomState (since there's no typed CDK task for the Cost Explorer or SNS AWS-SDK integrations).

const getCostAndUsage = new sfn.CustomState(this, 'GetCostAndUsage', {
    stateJson: {
        Type: 'Task',
        Resource: 'arn:aws:states:::aws-sdk:costexplorer:getCostAndUsage',
        Arguments: { /* Granularity, Metrics, TimePeriod (JSONata), GroupBy, Filter */ },
        Assign: { AngryThreshold: angryThresholdUsd, AccountId: cdk.Aws.ACCOUNT_ID, /* ... */ },
        Output: { Start: '{% ... %}', CostSum: '{% ... %}', CostSorted: '{% ... %}' },
    },
});
const publishCostDigest = new sfn.CustomState(this, 'PublishCostDigest', {
    stateJson: {
        Type: 'Task',
        Resource: 'arn:aws:states:::sns:publish',
        Arguments: { Message: { /* title/description, JSONata templates */ }, TopicArn: topic.topicArn },
    },
});
getCostAndUsage.next(publishCostDigest);

new sfn.StateMachine(this, 'CostDigestStateMachine', {
    definitionBody: sfn.DefinitionBody.fromChainable(getCostAndUsage),
    queryLanguage: sfn.QueryLanguage.JSONATA,
    tracingEnabled: true,
    logs: { destination: logGroup, level: sfn.LogLevel.ALL, includeExecutionData: true },
});

Next/End aren't hand-written anywhere. Just chaining with .next() is enough — CDK's state graph composition attaches them automatically (Next: "PublishCostDigest" on the first state, End: true on the last).

A JS string-escaping trap worth knowing about. The description message concatenates JSONata string literals for line breaks ("\n") and a regex for stripping the AWS/Amazon prefix off service names ($replace(/^(AWS|Amazon)\s*/, "")). Write either escape as a single backslash inside a TypeScript template literal and JavaScript resolves it immediately at the string-literal level — \n silently becomes a real newline character, and \s (not a recognized JS escape) silently loses its backslash entirely, leaving a bare s that turns the regex into s* and quietly breaks the service-name cleanup. The fix for both is the same: write \\n and \\s (double backslash) in the TypeScript source, so what actually reaches JSONata — after this string round-trips through JSON as part of the synthesized ASL — is the two-character escape sequence \n/\s, not a literal control character or a mangled regex. I found this not just by reading the JSONata spec, but by actually inspecting the synthesized cdk synth output before writing tests.

The EventBridge Scheduler's StepFunctionsStartExecution target (aws-scheduler-targets) automatically generates and attaches the scheduler's own execution role — unlike the original CloudFormation template this was ported from, there's no need to hand-write the states:StartExecution IAM statement.

Also worth noting: the title/description JSONata expressions aren't one-off inline strings — they come from two locale-specific builder functions, buildCostDigestTitleExpression/buildCostDigestDescriptionExpression. Flipping params.costDigest.locale ('ja' | 'en', default 'en') is all it takes to switch to the Japanese digest message (e.g. "😱 Costs are spiking" / "😊 Costs are steady" becomes "😱 コストが跳ね上がっています" / "😊 コストは落ち着いています").

Deploy & Verify

export PROJECT=your-project
export ENV=dev

npm run bootstrap -w workspaces/budgets-cost-anomaly-detection   # first time only
npm run synth -w workspaces/budgets-cost-anomaly-detection
npm run deploy:all -w workspaces/budgets-cost-anomaly-detection

Before deploying, replace the placeholder email address in parameters/dev-params.ts with a real one (SNS/Budgets email subscriptions require confirmation). To use Slack/Teams delivery, uncomment notification.slack/notification.teams.

# Pattern A: check the budget configuration (real billing data updates only a few times a day, so no instant trigger)
aws budgets describe-budgets --account-id <account-id>

# Pattern B: check the anomaly monitor/subscription (baseline establishment takes 24h+)
aws ce get-anomaly-monitors
aws ce get-anomaly-subscriptions

# Pattern C: check the topic's subscriptions
aws sns list-subscriptions-by-topic --topic-arn <finops-alert-topic-arn>

# Pattern D: check the alarm and its data (this stack only exists in us-east-1)
aws cloudwatch describe-alarms --alarm-names <project>-<env>-estimated-charges --region us-east-1

# Pattern E: run on demand without waiting for the schedule
aws stepfunctions start-execution --state-machine-arn <cost-digest-state-machine-arn>

Cost Estimate

💰 Rough monthly estimate (Tokyo region, Stack 4 only in us-east-1)

Service	Applies to	Rough monthly cost
AWS Budgets	A, C	First 2 budgets free; ~$0.02/day per budget after that
AWS Cost Anomaly Detection	B, C	No additional charge
Amazon SNS	All	First 1M requests within the free tier
AWS Chatbot	C, E (optional)	No additional charge
CloudWatch alarm (billing)	D	~$0.10/alarm/month
Step Functions (Standard)	E	First 4,000 state transitions/month free; negligible at once daily
EventBridge Scheduler	E	First 14M invocations/month free

Rough total: effectively $0-2/month at typical alert volumes.

Summary

What we learned from this pattern:

Pattern A (Budgets): Best for a known, fixed cost ceiling. Notifications fail silently if the SNS topic policy isn't set explicitly
Pattern B (Anomaly Detection): Best for catching unexpected spend even within budget. SNS delivery requires IMMEDIATE frequency, and thresholdExpression is a hand-assembled JSON string
Pattern C (Unified): Consolidating into one alert channel is more operationally realistic than managing a separate topic per signal — but since AWS caps AWS-managed Cost Anomaly Detection monitors at effectively one or two per account, Stack 3 doesn't create its own monitor and instead attaches an extra subscription to Stack 2's monitor via a real cross-stack reference. unifiedEscalation lets you split by severity by raising Stack 3's threshold, but as long as both stacks are deployed, the duplicate notification itself never goes away — so in real usage, pick either Stack 2 or Stack 3 for anomaly detection, not both
Pattern D (Billing Alarm): The simplest safety net, but has two gotchas CDK can't express: the manual "Receive Billing Alerts" opt-in, and being pinned to us-east-1
Pattern E (Cost Digest): A proactive, scheduled digest complements the other four patterns well. sfn.CustomState lets you write raw ASL/JSONata and skip Lambda entirely, but watch the string escaping at the JS → JSON → JSONata boundaries
AWS Chatbot's AdministratorAccess guardrail default applies to both Slack and Teams, and an empty array doesn't disable it — always pin an explicit, least-privilege guardrail
Wiring that repeats across stacks is worth factoring out into a construct like common/constructs/cost/ once, rather than copy-pasting

References

AWS Budgets User Guide
Creating an Amazon SNS topic for budget notifications
AWS Cost Anomaly Detection
Extending AWS managed monitors in AWS Cost Anomaly Detection (the source for the "one SERVICE monitor per account" limit behind Stack 3's design)
Create a billing alarm to monitor your estimated AWS charges
Setting up AWS Chatbot with Microsoft Teams
Variables and JSONata in AWS Step Functions
Amazon EventBridge Scheduler

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS CDK 100 Drill Exercises #009: CloudWatch Logs S3 Archive — Comparing 3 Log Archive Patterns in One Repository

TOMOAKI ishihara — Mon, 17 Aug 2026 15:24:21 +0000

Introduction

This is the 9th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

What you'll learn in this article

Near-real-time log archiving with Kinesis Data Firehose (via a subscription filter)
Tiered storage-class transitions with S3 lifecycle rules (Standard → IA → Glacier IR → Deep Archive)
How to bolt archiving onto an existing CloudWatch Logs log group after the fact
Scheduled batch export with the CloudWatch Logs Export Task API (and how to work around the one-concurrent-execution limit)
Decoding the CWL payload in a Lambda subscription filter and writing it to S3 in a custom format
How the required IAM trust relationships and bucket policies differ between patterns for the CloudWatch Logs service

📁 Code repository: GitHub

Architecture Overview

Pattern	Approach	Stacks
A: Firehose	Subscription filter → Kinesis Data Firehose → S3	Basic / Lifecycle / Existing (3 stacks)
B: Export Task	EventBridge Scheduler → Lambda → Export Task API → S3	Export (1 stack)
C: Direct Lambda write	Subscription filter → Lambda → S3	Lambda (1 stack)

Why three patterns

Trait	Pattern A (Firehose)	Pattern B (Export Task)	Pattern C (Lambda)
Delivery latency	Near real-time (seconds to minutes)	Scheduled (tolerates up to ~12h of delay)	Near real-time
Cost at high volume	Moderate (Firehose + S3)	Low (Lambda + S3 only)	Moderate (scales with Lambda invocation count)
Output format	Raw CWL JSON lines, GZIP-compressed	Raw CWL format	Custom JSON (fully controllable)
Operational simplicity	High (managed Firehose)	High (managed Export API)	Low (Lambda code needs maintenance)
Custom transformation	Limited (depends on Firehose's data transformation feature)	None	Fully controllable inside Lambda

In short: A if you need near-real-time delivery, B if a cost-optimized batch job is good enough, C if you want to define the output format yourself.

Data Flow

Pattern A (Stacks 1-3)
CloudWatch Log Group
  → Subscription filter (SubscriptionFilter + FirehoseDestination, role = CwlToFirehoseRole)
  → Kinesis Data Firehose (GZIP compression)
  → S3 archive bucket
  * Stack 1 (Basic) shares 5 log groups and uses dynamic partitioning on owner/logGroup
    to route each log group to its own S3 prefix. Stack 2 (Lifecycle) uses a single log
    group with a plain date prefix and no dynamic partitioning.

Pattern B (Stack 4)
EventBridge Scheduler
  → Lambda (calls logs:CreateExportTask for the previous day's logs)
  → CloudWatch Logs Export Task API
  → S3 archive bucket (bucket policy allows writes from logs.amazonaws.com)

Pattern C (Stack 5)
CloudWatch Log Group
  → Subscription filter (LambdaDestination, CDK manages invoke permissions automatically)
  → Lambda (decodes the gzip+base64 CWL payload and writes JSON to S3)
  → S3 archive bucket

Key Components

Component	Design point
S3 archive bucket	SSE-S3 encryption, versioning enabled, public access blocked, SSL enforced. Provisioned independently per stack (5 buckets total)
CwlToFirehoseRole	Assumed by `logs.amazonaws.com`. Scoped to log groups via an `aws:SourceArn` condition
FirehoseRole	Assumed by `firehose.amazonaws.com`. Grants the standard Firehose→S3 permission set (`PutObject`, `GetObject`, `ListBucket`, multipart upload actions) scoped to the archive bucket
Dynamic partitioning (Stack 1)	jq extracts `owner`/`logGroup` to route 5 shared log groups into separate S3 prefixes. `CloudWatchLogProcessor` can't be used because it strips those fields; a jq fallback is required to handle `CONTROL_MESSAGE` records
Tiered lifecycle rules (Stack 2)	Standard → IA (30 days) → Glacier IR (90 days) → Deep Archive (365 days) → expiration (7 years)
Importing an existing log group (Stack 3)	Referenced via `LogGroup.fromLogGroupName()`. No new `AWS::Logs::LogGroup` resource is created
Export Task Lambda (Stack 4)	A concurrency guard that checks for running/pending tasks before calling `CreateExportTask`
Direct Lambda write (Stack 5)	`LambdaDestination` automatically manages invoke permissions. The payload is decoded and saved as custom JSON

Implementation Highlights

1. Pattern A — Kinesis Data Firehose subscription

For CloudWatch Logs to deliver to Firehose, it needs an explicit IAM role. We split responsibilities across two roles.

// CwlToFirehoseRole: assumed by logs.amazonaws.com. Only the trust policy is defined here
const cwlToFirehoseRole = new iam.Role(this, 'CwlToFirehoseRole', {
    assumedBy: new iam.ServicePrincipal('logs.amazonaws.com', {
        conditions: {
            StringLike: {
                'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*`,
            },
        },
    }),
});

// L2 SubscriptionFilter + FirehoseDestination(role: cwlToFirehoseRole)
const filterPattern = logGroupParams.filterPattern ?? defaultLogGroupArchiveConfig.filterPattern;
new logs.SubscriptionFilter(this, 'CwlSubscriptionFilter', {
    logGroup: this.logGroup,
    destination: new logs_destinations.FirehoseDestination(this.deliveryStream, {
        role: cwlToFirehoseRole,
    }),
    filterPattern: filterPattern
        ? logs.FilterPattern.literal(filterPattern)
        : logs.FilterPattern.allEvents(),
});

Why not attach firehose:PutRecord/PutRecordBatch permissions directly on the role?
Inside bind(), aws-logs-destinations's FirehoseDestination calls deliveryStream.grantPutRecords(role) on the role you pass in (or one it generates), automatically granting the necessary permissions. If you also add your own inline policy to the role, you end up with two AWS::IAM::Policy resources granting the same permission. The trick is to define only the trust policy yourself (scoped by SourceArn) and let the L2 construct handle the permission grant.

I used to write this with the L1 CfnSubscriptionFilter plus a hand-rolled inline policy, on the assumption that "the L2 SubscriptionFilter can't accept a roleArn for Firehose." That assumption was wrong. In the current aws-cdk-lib (2.261.0 in this repository), FirehoseDestination wires the role's ARN all the way through to CfnSubscriptionFilter for you, so the L2 construct alone is sufficient.

1a. Per-log-group dynamic partitioning (Stack 1 – Basic)

const s3Destination = new firehose.S3Bucket(archiveBucket, {
    dataOutputPrefix:
        'AWSLogs/!{partitionKeyFromQuery:owner}/CWLogGroup/!{partitionKeyFromQuery:logGroup}/!{timestamp:yyyy/MM/dd/HH}/',
    dynamicPartitioning: { enabled: true },
    bufferingInterval: cdk.Duration.seconds(60), // dynamic partitioning requires >= 60s
    bufferingSize: cdk.Size.mebibytes(64),        // and >= 64MiB
    processors: [
        new firehose.DecompressionProcessor({
            compressionFormat: firehose.DecompressionProcessorCompressionFormat.GZIP,
        }),
        firehose.MetadataExtractionProcessor.jq16({
            owner: '(if (.owner // "") == "" then "controlmessages" else .owner end)',
            logGroup:
                '(if (.logGroup // "") == "" then "controlmessages" else (.logGroup | ltrimstr("/")) end)',
        }),
        new firehose.AppendDelimiterToRecordProcessor(),
    ],
});

I hit two snags here.

2. Tiered S3 lifecycle (Stack 2 – Lifecycle)

For long-term archives, you can optimize cost by moving objects to progressively cheaper storage classes as access frequency drops.

archiveBucket.addLifecycleRule({
    id: `MoveToIAAfter${moveToIaAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.INFREQUENT_ACCESS, transitionAfter: cdk.Duration.days(moveToIaAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `MoveToGlacierAfter${moveToGlacierAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.GLACIER_INSTANT_RETRIEVAL, transitionAfter: cdk.Duration.days(moveToGlacierAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `MoveToDeepArchiveAfter${moveToDeepArchiveAfterDays}Days`,
    enabled: true,
    transitions: [
        { storageClass: s3.StorageClass.DEEP_ARCHIVE, transitionAfter: cdk.Duration.days(moveToDeepArchiveAfterDays) },
    ],
});

archiveBucket.addLifecycleRule({
    id: `ExpireCurrentObjectsAfter${expireAfterDays}Days`,
    enabled: true,
    expiration: cdk.Duration.days(expireAfterDays),
});

3. Attaching to an existing log group (Stack 3 – Existing)

Sometimes you don't want to create a new log group — you want to bolt archiving onto a log group an application is already writing to.

const existingLogGroup = logs.LogGroup.fromLogGroupName(
    this,
    'ExistingLogGroup',
    existingLogGroupParams.logGroupName,
);

4. Pattern B — Scheduled Export Task

The CloudWatch Logs Export Task API has a one concurrent execution per account limit. Before submitting a new task, the Lambda checks for running or pending tasks to avoid errors.

# export-task/index.py (core logic)
def lambda_handler(event, context):
    logs_client = boto3.client("logs")

    # Skip if a task is already running or pending
    running = logs_client.describe_export_tasks(statusCode="RUNNING")
    pending = logs_client.describe_export_tasks(statusCode="PENDING")
    if running["exportTasks"] or pending["exportTasks"]:
        print("An export task is already running or pending. Skipping.")
        return {"status": "skipped", "reason": "task_in_progress"}

    # Export the previous day's (UTC) logs
    yesterday = datetime.datetime.utcnow() - datetime.timedelta(days=1)
    from_time = int(yesterday.replace(hour=0, minute=0, second=0, microsecond=0).timestamp() * 1000)
    to_time = int(yesterday.replace(hour=23, minute=59, second=59, microsecond=999999).timestamp() * 1000)

    response = logs_client.create_export_task(
        taskName=f"export-{yesterday.strftime('%Y-%m-%d')}",
        logGroupName=LOG_GROUP_NAME,
        fromTime=from_time,
        to=to_time,
        destination=S3_BUCKET_NAME,
        destinationPrefix=f"{S3_PREFIX}/{yesterday.strftime('%Y/%m/%d')}",
    )
    return {"status": "created", "taskId": response["taskId"]}

The key detail is checking PENDING in addition to RUNNING. If a scheduled task is merely sitting in the queue and you submit the next one anyway, you get a LimitExceededException.

The S3 bucket side needs two resource policies allowing writes from the CloudWatch Logs service principal.

// 1. Permission to check the bucket ACL before exporting
archiveBucket.addToResourcePolicy(
    new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        principals: [new iam.ServicePrincipal('logs.amazonaws.com')],
        actions: ['s3:GetBucketAcl'],
        resources: [archiveBucket.bucketArn],
        conditions: {
            ArnLike: { 'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*` },
        },
    }),
);

// 2. Permission to write the exported objects
archiveBucket.addToResourcePolicy(
    new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        principals: [new iam.ServicePrincipal('logs.amazonaws.com')],
        actions: ['s3:PutObject'],
        resources: [archiveBucket.arnForObjects('*')],
        conditions: {
            StringEquals: { 's3:x-amz-acl': 'bucket-owner-full-control' },
            ArnLike: { 'aws:SourceArn': `arn:${this.partition}:logs:${this.region}:${this.account}:log-group:*` },
        },
    }),
);

The condition pinning s3:x-amz-acl to bucket-owner-full-control is explicitly required by CloudWatch Logs' official documentation. Omit it and the export task fails with a permissions error.

5. Pattern C — Subscription filter → Lambda → S3

With LambdaDestination, CDK automatically manages the resource policy (Lambda::Permission) that lets logs.amazonaws.com invoke your Lambda.

new logs.SubscriptionFilter(this, 'CwlSubscriptionFilter', {
    logGroup: this.logGroup,
    destination: new logDestinations.LambdaDestination(this.archiveFunction),
    filterPattern: filterPattern
        ? logs.FilterPattern.literal(filterPattern)
        : logs.FilterPattern.allEvents(),
});

On the Lambda side, the code decodes the CWL compressed payload and writes it to S3 in a JSON format it defines itself, per invocation.

# cwl-to-s3/index.py (core logic)
def lambda_handler(event, context):
    compressed = base64.b64decode(event["awslogs"]["data"])
    payload = json.loads(gzip.decompress(compressed).decode("utf-8"))

    if payload.get("messageType") == "CONTROL_MESSAGE":
        # Health-check message CloudWatch Logs sends, e.g. when a subscription starts
        return {"status": "skipped", "reason": "control_message"}

    now = datetime.datetime.utcnow()
    safe_stream = payload["logStream"].replace("/", "_").replace("$", "").replace("[", "").replace("]", "")
    s3_key = f"{S3_PREFIX}/{now.strftime('%Y/%m/%d/%H')}/{safe_stream}_{uuid.uuid4().hex[:8]}.json"

    record = {
        "logGroup": payload["logGroup"],
        "logStream": payload["logStream"],
        "owner": payload.get("owner"),
        "exportedAt": now.isoformat() + "Z",
        "events": payload["logEvents"],
    }
    s3_client.put_object(Bucket=S3_BUCKET_NAME, Key=s3_key, Body=json.dumps(record, ensure_ascii=False).encode("utf-8"))
    return {"status": "ok", "processed": len(payload["logEvents"]), "s3Key": s3_key}

Deploy & Verify

export PROJECT=your-project
export ENV=dev

npm run bootstrap -w workspaces/cloudwatch-logs-s3-archive   # first time only
npm run stage:deploy:all -w workspaces/cloudwatch-logs-s3-archive

To deploy individual stacks, use the per-stack stage:deploy:* scripts (each scoped to a cdk deploy stack selector like **/*Basic).

npm run stage:deploy:basic -w workspaces/cloudwatch-logs-s3-archive
npm run stage:deploy:lifecycle -w workspaces/cloudwatch-logs-s3-archive
npm run stage:deploy:export -w workspaces/cloudwatch-logs-s3-archive

# Pattern A Stack 1 (Basic): write test data into 5 log groups (/<project>/<env>/basic-*)
# (given the dynamic-partitioning buffering requirements of >= 60s / >= 64MiB, use the
#  repo's helper script rather than a one-off put-log-events call)
./write-test-logs.sh --project <project> --env <env>

# After ~60s, output appears partitioned under AWSLogs/<owner>/CWLogGroup/<logGroup>/...
aws s3 ls s3://<archive-bucket>/AWSLogs/ --recursive

# Pattern A Stack 2 (Lifecycle): single log group, no dynamic partitioning
./write-test-logs-lifecycle.sh --project <project> --env <env>
aws s3 ls s3://<archive-bucket>/ --recursive

# Pattern B: manually invoke the export Lambda
aws lambda invoke --function-name <project>-<env>-cwl-export-task --payload '{}' response.json
cat response.json

# Pattern C: write a test log entry (the subscription filter invokes the Lambda immediately)
aws logs put-log-events \
  --log-group-name /<project>/<env>/app-lambda \
  --log-stream-name test-stream \
  --log-events timestamp=$(date +%s000),message="hello lambda"
aws s3 ls s3://<archive-bucket>/subscriptions/ --recursive

Cost Estimate

💰 Rough monthly estimate (Tokyo region, 1 GB of log data per day)

Service	Applies to	Rough monthly cost
Kinesis Data Firehose	A	~$0.90 (at $0.03/GB)
Lambda	B, C	Effectively negligible at low invocation counts
S3 (Standard)	All	~$0.75 (at $0.025/GB-month)
CloudWatch Logs (ingestion)	All	~$0.76/GB (charged regardless of which archive pattern you use, and not included in the total below)
EventBridge	B	$1.00 per million events (negligible at minimal frequency)

Running all five stacks simultaneously at production-scale traffic for comparison testing will cost more than the above. Don't forget to clean up afterward with npm run stage:destroy:all -w workspaces/cloudwatch-logs-s3-archive.

Summary

What we learned from this pattern:

Pattern A (Firehose): Minimal code, well-suited for near-real-time archiving. Delivery from CloudWatch Logs requires FirehoseDestination (L2) plus two IAM roles with distinct responsibilities
Pattern B (Export Task): The lowest cost for batch use cases, but limited to one concurrent execution per account. A guard checking for running/pending tasks on the Lambda side is essential
Pattern C (Lambda): Full control over the output format, at the cost of ongoing Lambda code maintenance, including handling control messages
IAM trust relationships differ by service: For Firehose, FirehoseDestination grants permission to the trusted role via grantPutRecords; for Export Task, an S3 bucket policy with an s3:x-amz-acl condition is required; for LambdaDestination, CDK auto-generates the invoke permission — the same "delivery from CloudWatch Logs" concept wires up differently every time
Split tiered lifecycle rules by transition: rather than one giant rule, separate rules make the transitionAfter thresholds easy to tune independently as environment parameters
fromLogGroupName() is enough for bolting archiving onto an existing log group: pass the existing ARN into the trust policy directly and reuse the same wiring, with no new resource created
Firehose dynamic partitioning is incompatible with message extraction: CloudWatchLogProcessor strips owner/logGroup, so drop it if you're using jq to derive partition keys from those fields. You also need to account for CloudWatch Logs' CONTROL_MESSAGE records (where owner/logGroup are empty strings) with a jq if/then/else fallback (// doesn't catch empty strings)

References

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS CDK 100 Drill Exercises #008: CloudFront VPC Origin — Reaching an Internal ALB Directly, and Designing an Incident-Response Escape Hatch

TOMOAKI ishihara — Tue, 28 Jul 2026 15:19:43 +0000

Introduction

This is the 8th installment of "AWS CDK 100 Drill Exercises." See here for an overview of the series.

This time, we place two very different origins behind a single Amazon CloudFront distribution: a private S3 bucket serving a static site, and a private, internal ALB that CloudFront can reach only through a VPC Origin. No public IP, no NAT Gateway — CloudFront reaches the ALB directly in a private subnet.

And we don't stop at a "working architecture" — we also build in an incident-response mechanism based on a real AWS outage.

To keep this sample simple, the CloudFront⇔ALB communication uses HTTP (port 80) only. In production, you'd want to attach an ACM certificate to the ALB, add an HTTPS listener, and switch CloudFront's origin protocol policy to HTTPS_ONLY (or match-viewer).

What you'll learn in this article

How to serve content without ever making an origin public, using VPC Origin + Origin Access Control (OAC)
Automatic failover from a dynamic origin to a static one via Origin Group
How to allow-list IP addresses with AWS WAF
Recording denied requests to CloudWatch Logs via CloudFront Function + standard logging v2
How to deal with the constraint that some CloudFront APIs and metrics are pinned to us-east-1
A way of thinking about designing "what to do when it breaks" as a CDK parameter, assuming failure up front

📁 Code repository: GitHub

Architecture Overview

Feature	Benefit
VPC Origin	CloudFront reaches a private ALB directly, with no public IP and no NAT Gateway
Origin Group failover	Shows a static fallback page instead of a raw 502/503 when the ALB is unhealthy
Edge IP allow-listing	Disallowed IPs are rejected at the edge before ever reaching the ALB or S3
Geo restriction	Restricts delivery to only the explicitly allowed countries
Incident-response escape hatch	Routes around a VPC Origin-side outage with just a parameter switch — no code changes

Data Flow

Viewer
  │  HTTPS (min TLS 1.3, geo-restricted to JP/US/GB/CA/AU/NZ/IE)
  ▼
CloudFront distribution
  ├─ WAF (Web ACL, always active): managed rules + rejects IPs not on the allow list
  ├─ CloudFront Function (optional, disabled by default): rejects IPs not on the allow list
  │
  ├─ Default behavior "/*"
  │     └─ S3 origin (OAC) ────────► WebsiteBucket (private)
  │
  └─ Behavior "/alb/*"
      └─ Origin Group
          ├─ Primary: VPC Origin ──► Internal ALB (private subnet)
          └─ Fallback (403/404/5xx): S3 origin (OAC) ─► ErrorBucket (private)

The ALB's security group allows inbound traffic only on port 80 from the CloudFront-managed prefix list. CloudFront itself is reachable from the internet, but the ALB is not directly accessible.

Key Components

Component	Design Points
ALB (internal)	`internetFacing: false`, placed in a private-isolated subnet. The VPC Origin's target
ALB (Public, created only when `publicAlbFailover` is enabled)	`internetFacing: true`, public subnet. A dedicated security group also allows only the CloudFront-managed prefix list
WebsiteBucket / ErrorBucket	Private S3 buckets, readable only by CloudFront via OAC
CloudFront distribution	Min TLS 1.3, geo restriction, dedicated access log bucket
CloudfrontWafStack (pinned to us-east-1)	Managed rules (Core/KnownBadInputs) + IP allow list (before/after rules selectable), defaultAction is block, logs delivered directly to S3
CloudFront Function	Viewer IP allow list (optional, created only when `allowedCloudFunctionIps` is set), evaluated on `viewer-request`
CloudfrontLogDeliveryStack (pinned to us-east-1, created only when the CloudFront Function is used)	Delivers denied-request logs to CloudWatch Logs
CloudfrontMonitoringStack (pinned to us-east-1)	5xx error rate alarm + SNS notification

Implementation Highlights

1. An internal ALB reachable only from CloudFront

The ALB's security group opens port 80 only to the CloudFront-managed prefix list.

albSecurityGroup.addIngressRule(
  ec2.Peer.prefixList(props.cloudfrontManagedPrefixList),
  ec2.Port.tcp(80),
  'Allow inbound HTTP traffic from CloudFront managed prefix list'
);

const alb = new elbv2.ApplicationLoadBalancer(this, 'Alb', {
  vpc: this.vpc.vpc,
  internetFacing: false,
  securityGroup: albSecurityGroup,
  vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
});

The prefix list ID is region-specific. Check it beforehand.

aws ec2 describe-managed-prefix-lists \
  --filters "Name=prefix-list-name,Values=com.amazonaws.global.cloudfront.origin-facing"

2. Automatic failover with VPC Origin and Origin Group

CloudFront reaches the internal ALB directly via a VPC Origin. If the ALB stops responding, CloudFront automatically fails over to the static page in S3.

const originGroup = new cloudfront_origins.OriginGroup({
  primaryOrigin: cloudfront_origins.VpcOrigin.withApplicationLoadBalancer(alb, {
    httpPort: 80,
  }),
  fallbackOrigin: cloudfront_origins.S3BucketOrigin.withOriginAccessControl(errorBucket),
  fallbackStatusCodes: [403, 404, 500, 502, 503, 504],
});

distribution.addBehavior('/alb/*', originGroup, {
  viewerProtocolPolicy: cloudfront.ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
  // Without this, the default CACHING_OPTIMIZED policy applies (1-day default TTL). The ALB's
  // fixed responses carry no Cache-Control header at all, so once a successful response gets
  // cached at the edge, it keeps being served for up to a day regardless of the origin's or
  // Origin Group's actual state.
  cachePolicy: cloudfront.CachePolicy.CACHING_DISABLED,
});

Without this explicit cache policy, you'll run into a confusing situation while testing failover: "the origin has switched, but the browser/edge keeps serving a stale cached success response, so nothing looks different." /alb/* exists to show you the ALB's live response, so disabling caching outright is the straightforward choice here.

3. IP allow-listing and managed rules with WAF

Edge access control is handled first by a WAFv2 Web ACL attached to the whole distribution (CloudfrontWafStack). Since defaultAction is block, the effective behavior is that everything except allow-listed IPs is rejected.

const wafAcl = new wafv2.CfnWebACL(this, 'WafAcl', {
  scope: 'CLOUDFRONT',
  defaultAction: { block: {} },
  rules: [
    // Allow before managed rules (only created if IPs are actually specified. priority 1)
    { name: 'AllowSpecificIPsBeforeRules', priority: 1, action: { allow: {} }, ... },
    { name: 'CoreRuleSet', priority: 2, overrideAction: { none: {} }, /* AWSManagedRulesCommonRuleSet */ },
    { name: 'KnownBadInputsRuleSet', priority: 3, overrideAction: { none: {} }, /* AWSManagedRulesKnownBadInputsRuleSet */ },
    // Allow after managed rules (enabled by default. priority 100)
    { name: 'AllowSpecificIPsAfterRules', priority: 100, action: { allow: {} }, /* IPSetReferenceStatement */ },
  ],
});

In the current wiring, the deploying operator's IP (getMyGlobalIp()) is passed as allowedIpsAfterRules, and is allowed after the managed rules are evaluated (priority 100). In other words, even an allow-listed IP is first checked against the Core/KnownBadInputs managed rules for known attack patterns before being allowed through. The allowedIpsBeforeRules option (allow before rule evaluation) is there for cases where you want to verify behavior before the WAF rules take effect.

WAF logs are delivered directly to an S3 bucket whose name must carry the aws-waf-logs- prefix (CfnLoggingConfiguration). The authorization and cookie headers are masked via redactedFields before being recorded.

4. Edge control via CloudFront Function, and incident investigation via standard logging v2

Separately from WAF, a CloudFront Function-based allow list is also implemented for cases where you want finer, per-behavior control. When allowedCloudFunctionIps is set, every viewer request is evaluated and IPs not on the list are rejected. Denied requests are recorded as custom data via cf.logCustomData().

function handler(event) {
  var request = event.request;
  var allowedIps = ['203.0.113.10']; // baked in at synth time
  var clientIp = event.viewer.ip; // viewer IP accessor for the JS_2_0 runtime
  if (!allowedIps.includes(clientIp)) {
    cf.logCustomData(JSON.stringify({ clientIp: clientIp, allowedIps: allowedIps }));
    return { statusCode: 403, statusDescription: 'Forbidden', body: 'Access denied' };
  }
  return request;
}

The CloudFront Function is positioned as an optional feature you enable as needed.

5. The incident-response escape hatch (`publicAlbFailover`)

This is the point I most want to get across in this article.

On July 16, 2026, AWS CloudFront experienced a multi-hour outage in which 5xx errors spiked for customers using VPC Origins. The root cause was not in customers' own stacks — it was in the VPC Origin connectivity layer itself.

We identified the root cause of the issue as an internal constraint on the fleet that manages connections to private VPC origins. When this constraint was reached, the system responsible for distributing routing configuration to our network processors failed to load the updated configuration data correctly, affecting routing of VPC Origin connections.

AWS's guidance at the time was: "customers who are able to should temporarily switch away from the VPC Origin connection, and switch back once it's resolved."

The problem is that trying to do this by manually rewriting CloudFormation while the incident is ongoing makes both the decision-making and the work slower. So this stack prepares the switch ahead of time, as a single publicAlbFailover parameter.

// The internal ALB (for VPC Origin) is always created. A separate public ALB is created only
// when publicAlbFailover is enabled.
const publicAlb = publicAlbFailoverEnabled
  ? new elbv2.ApplicationLoadBalancer(this, 'PublicAlb', {
      internetFacing: true,
      securityGroup: publicAlbSecurityGroup,
      vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
    })
  : undefined;

// The VPC Origin used as the primaryOrigin in normal mode
const vpcOriginAlb = cloudfront_origins.VpcOrigin.withApplicationLoadBalancer(alb, {
  httpPort: 80,
  protocolPolicy: cloudfront.OriginProtocolPolicy.HTTP_ONLY,
});

// Escape hatch enabled: reach the separate public ALB as a plain public HTTP origin
const publicAlbOrigin = publicAlbFailoverEnabled
  ? new cloudfront_origins.HttpOrigin(publicAlb!.loadBalancerDnsName, {
      httpPort: 80,
      protocolPolicy: cloudfront.OriginProtocolPolicy.HTTP_ONLY,
    })
  : undefined;

// The fallback is always the S3 error page; only the primary swaps to the public ALB when enabled
const originGroup = new cloudfront_origins.OriginGroup({
  primaryOrigin: publicAlbFailoverEnabled ? publicAlbOrigin! : vpcOriginAlb,
  fallbackOrigin: cloudfront_origins.S3BucketOrigin.withOriginAccessControl(errorBucket),
  fallbackStatusCodes: [403, 404, 500, 502, 503, 504],
});

There are 4 design points here.

Traffic still only ever flows through CloudFront. Enabling this makes the newly-created public ALB internet-facing, but its security group — just like the internal ALB's — still only allows the CloudFront-managed prefix list, so it's never directly reachable by clients. The client-facing URL doesn't change either.
Setting publicAlbFailover.enabled: true makes cloudfrontManagedPrefixList a required field. Without it, there'd be no way to restrict the public ALB to CloudFront only, so it throws at synth time.
The prefix list alone isn't enough. The CloudFront managed prefix list is a shared IP range used by every CloudFront customer's distribution, so if someone else's CloudFront distribution pointed at this public ALB's DNS name as an origin, it would still pass the security group check. So, following AWS's official recommended procedure, we generate a secret value at synth time and send it as a CloudFront origin custom header, with the ALB listener rejecting (via its default action, 403) any request that doesn't carry that header.
It switches over in "minutes, not instantly." We originally tried a design where keeping the VPC Origin bound as the Origin Group's fallback would avoid recreating it. But CDK auto-derives an origin's logical ID from its "position" — which includes whether it's the OriginGroup's primary or fallback role — so simply swapping that role changes the VPC Origin's own logical ID, and CloudFormation ends up deleting and recreating it anyway (up to ~15 minutes). (I cover this CDK logical-ID auto-numbering mechanism in depth in my own article, "AWS CDK Logical ID Deep Dive: How Adding One CloudFront Origin Broke My Entire AWS CDK Deployment".) You can pin the logical ID with overrideLogicalId to avoid that, but then you trade it for a different constraint: CloudFront's UpdateVpcOrigin API rejects any in-place property change to the origin while it's still associated with a distribution. This sample favors simplicity and accepts that the VPC Origin gets recreated.

If you need genuinely zero-downtime, instant switching in production, consider the CloudFront continuous deployment approach (a staging distribution with header/weight-based gradual traffic shifting) or CloudFront Functions + KeyValueStore-based weighted routing, both covered in Migrate Amazon CloudFront public origins to private VPC origins. Both keep both origins alive at all times and shift traffic gradually by ratio — a step up in complexity from this article's Origin Group swap, but the right choice when you truly need zero-downtime switching.

Using it is simple.

// parameters/dev-params.ts
publicAlbFailover: {
    enabled: true, // false → true
},

npm run deploy:all

Once AWS resolves the issue, just set enabled: false and redeploy.

6. 5xx error rate alarm (also pinned to us-east-1)

CloudFront's request/error metrics (the AWS/CloudFront namespace) are only published to us-east-1, regardless of the distribution's own region. A CloudWatch Alarm can only evaluate a metric in its own region, so this alarm is also broken out into its own stack, CloudfrontMonitoringStack, deployed to us-east-1.

const alarm = new cloudwatch.Alarm(this, 'CloudFront5xxErrorRateAlarm', {
  metric: new cloudwatch.Metric({
    namespace: 'AWS/CloudFront',
    metricName: '5xxErrorRate',
    dimensionsMap: { DistributionId: props.distributionId },
    period: cdk.Duration.minutes(5),
    statistic: cloudwatch.Stats.AVERAGE,
  }),
  threshold: 5, // 5%
  evaluationPeriods: 3,
  datapointsToAlarm: 3, // sustained for 15 minutes before treating it as a real incident
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
});

This alarm watches exactly the metric AWS itself called out during the 2026-07-16 outage (the distribution's overall 5xx error rate). In other words, it's designed to plug directly into an operational flow: "alarm fires" → "check the AWS Health Dashboard" → "if it's a VPC Origin-side problem, enable publicAlbFailover."

Deploy & Verify

export PROJECT=your-project
export ENV=dev

npm run bootstrap   # first time only
npm run deploy:all

# Served from S3
curl https://<distribution-domain>/

# Served from the ALB via the Origin Group
curl https://<distribution-domain>/alb/

Cost Estimate

💰 Rough monthly estimate (Tokyo region, low traffic)

Service	Usage	Rough monthly cost
Application Load Balancer	Always on	~$25.00
CloudFront	Low request volume	~$1–2
AWS WAF	Web ACL + 3 rules + low request volume	~$8.00
S3 (3 buckets)	A few MB of static content + logs	< $0.10
CloudWatch Alarm + SNS	5xx error rate alarm notifications	< $0.10

Total: roughly $23–25/month

The ALB is billed hourly regardless of traffic. Don't forget to run npm run destroy:all to clean up after you're done experimenting.

If cdk destroy / npm run destroy:all gets stuck — a non-empty S3 bucket, a leftover ENI tied to the VPC Origin, or some other resource CloudFormation refuses to delete on its own — delstack, an OSS tool by AWS DevTools Hero k.goto (@365_step_tech), is worth having on hand. It forcibly cleans up the kind of resources that commonly block stack deletion.

Summary

What we learned from this pattern:

VPC Origin: CloudFront can reach a fully private ALB directly, with no public IP and no NAT Gateway
Origin Group: Automatic failover from a dynamic origin to a static one when a server error occurs
Defense in depth at the edge: Geo restriction, WAF, and CloudFront Function give you access control before traffic ever reaches the origin
CloudFront's us-east-1 constraint: Both the log delivery API and metrics are pinned to us-east-1, regardless of the distribution's own region
Bake incident response into code: Given a real outage, prepare "how to route around it" as a parameter ahead of time rather than as manual work
The prefix list isn't a silver bullet: Since it's a shared IP range across every customer, an internet-facing origin should also verify a secret custom header
Be explicit about cache policy on dynamic paths: Leaving the default CACHING_OPTIMIZED in place means a successful response from an origin that never sends Cache-Control gets cached for up to a day, masking your failover test results
CDK's origin logical-ID auto-numbering also depends on "role": Just swapping which origin is primary vs. fallback changes that origin's logical ID, so it can be recreated on every switch. Genuinely zero-downtime switching needs continuous deployment or CloudFront Functions + KeyValueStore-based weighted routing

References

Amazon CloudFront VPC origins
Using origin groups for failover
AWS CDK Logical ID Deep Dive: How Adding One CloudFront Origin Broke My Entire AWS CDK Deployment — my own deep dive into how CDK's origin logical-ID auto-numbering works
Migrate Amazon CloudFront public origins to private VPC origins
Using the managed cache policies (CachingOptimized / CachingDisabled)
Restrict access to Application Load Balancers
AWS WAF developer guide
CloudFront standard logging v2
Monitoring CloudFront distributions with CloudWatch metrics
AWS Health Dashboard
X: Amazon CloudFront Issue Resolved
delstack — an OSS tool by AWS DevTools Hero k.goto (@365_step_tech) for force-deleting CloudFormation stacks stuck on resources like non-empty S3 buckets or leftover ENIs

Let's keep learning practical AWS CDK patterns through the 100 drill exercises!
If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in my GitHub repository.

AWS or Amazon Quiz — How Well Do You Know AWS Service Prefixes?

TOMOAKI ishihara — Thu, 23 Jul 2026 15:16:57 +0000

Introduction

AWS (Amazon Web Services) offers 200+ services these days. That's a lot of names to keep straight.

And there's one thing that trips up everyone, beginners included. You get used to just saying "Route 53" out loud, and then when you're writing documentation you suddenly blank on the official name: is it "AWS Route 53" or "Amazon Route 53"?

Right?

AWS or Amazon

I figured turning that everyday confusion into a game could be fun, so I looked around for something similar. There are quizzes that show a service icon and ask you to name it, and quizzes that describe a service and ask you to pick the name — but nothing focused purely on "is it AWS or Amazon?"
Naming the full service correctly is hard for beginners, but just picking the prefix felt like something anyone could enjoy.

These days, if you have an idea, AI makes it quick to build. So I sketched out the rules for the game right away.

By the way, the distinction between the two prefixes is explained in plenty of places, but roughly:

Amazon: standalone services that work on their own
AWS: utility services typically used by other services

What I built

Here's what I ended up with. It's playable entirely in the browser.
It's hosted on GitHub Pages, and the source code is here.

Designing the game

I wanted something simple: a two-choice format you can breeze through. So I built it to pull 20 random questions from a dataset of AWS service info.
Beyond just picking an answer, each question shows a short explanation of the service and a link to the official docs, so you can actually learn something along the way.

Choosing where to host it

Since the game is just plain JavaScript and a single HTML file, GitHub Pages was the obvious choice for hosting.

Improving the UI

In the first version, the "Next" button appeared at the bottom of the screen after you answered. That worked, but it was too far from the answer choices, and on mobile it made moving to the next question clunky.

So I compared a few design options.

I also had a rank displayed based on your score, but it wasn't obvious whether a given rank was good or bad, so I simplified it to a plain evaluation instead.

Wrapping up

That's the story behind building "AWS or Amazon Quiz" and what I paid attention to along the way.
If you play it and catch yourself thinking "wait, I thought that one was Amazon" — that's a sign the naming conventions are starting to stick.

If you enjoyed it, a star on GitHub would mean a lot.
Bug reports and corrections (like "this service's description is wrong") are welcome via Issues or comments — the service descriptions were generated by AI since there are so many of them, so a few may not be entirely accurate.

The problem of putting Transit Gateway subnets at the "end" of a CIDR — existing tools couldn't solve it, so I built one myself

TOMOAKI ishihara — Sun, 19 Jul 2026 13:01:50 +0000

Introduction

How do you calculate CIDRs when designing VPC subnets? Do you work through them one by one with an online subnet calculator?

Or maybe you reach for the AWS Subnet Calculator, or the Excel-based subnet calculator spreadsheet referenced in AWS docs > Amazon EKS > Amazon VPC CNI?

Both are handy tools, but I ran into a wall: I needed the subnet reserved for Transit Gateway to sit at the very end of the VPC's CIDR block, and neither tool made that easy.

Here's the background. At one organization I worked with, on-premises connectivity was built using AWS Direct Connect and Transit Gateway. Whenever a new AWS account was issued, the VPC and its Transit Gateway subnet came pre-provisioned as part of the standard setup. On top of that, the Transit Gateway subnet was always carved out from the tail end of the VPC's CIDR block by design.

So when I needed to lay out subnets for a business application, I needed a tool that could compute the Transit Gateway subnet from the end of the CIDR range. None of the existing tools did that, so I built my own.

The tool is live on GitHub Pages, and the source is on GitHub.

Features

Variable VPC CIDR — enter any valid CIDR block (e.g. 10.0.0.0/22)
Multi-tier subnet design — add, remove, and configure tiers with custom subnet masks
Drag-and-drop reordering — reorder tiers by dragging them into place
Dual AZ support — automatically allocates subnets across ap-northeast-1a and ap-northeast-1c
TGW tail allocation — tiers named "TGW" get carved out from the end of the VPC address space (toggle on/off)
Free address space detection — surfaces every leftover range: alignment gaps, the space between forward-allocated and TGW subnets, and trailing space
AWS reserved address aware — shows the usable host count after subtracting the 5 AWS-reserved addresses per subnet
Japanese/English language switching — toggle the UI language with a button; auto-detects the browser's language on load
Dark mode support — automatically adapts to the system color scheme

How the CIDR calculation works

There's no server and no database behind this tool — the entire calculation runs as bitwise arithmetic in the browser. The process breaks down into four steps.

1. Turn the CIDR string into a 32-bit integer

First, an input string like 10.0.0.0/22 gets converted into a 32-bit integer — a network address plus a total address count — so it can be manipulated with bitwise operations.

function parseVPC(cidr) {
  var m = cidr.trim().match(/^(\d+\.\d+\.\d+\.\d+)\/(\d+)$/);
  if (!m) return null;
  var parts = m[1].split('.').map(Number);
  if (parts.some(function(p) { return p < 0 || p > 255; })) return null;
  var prefix = parseInt(m[2]);
  if (prefix < 8 || prefix > 28) return null;
  var base = ((parts[0]<<24)|(parts[1]<<16)|(parts[2]<<8)|parts[3]) >>> 0;
  var total = Math.pow(2, 32 - prefix);
  var network = (base & (~((1 << (32 - prefix)) - 1) >>> 0)) >>> 0;
  return { base: network, prefix: prefix, total: total, end: network + total - 1 };
}

Two things are worth calling out:

parts[0]<<24 | parts[1]<<16 | parts[2]<<8 | parts[3] packs the four octets into a single 32-bit integer.
The >>> 0 matters. JavaScript's bitwise operators treat numbers as signed 32-bit integers, so without it, any first octet of 128 or higher would come out negative.

~((1 << (32 - prefix)) - 1) then builds the subnet mask, and base & mask rounds down to the network address. So even if you type 10.0.0.5/22, with a non-zero host part, it automatically snaps to 10.0.0.0/22.

2. Forward tiers: round up to the alignment boundary

Regular tiers like Private-App and Public get packed from the front of the VPC, AZ by AZ. The key constraint here is alignment: a /26 subnet can only start on a 64-address boundary (x.x.x.0, x.x.x.64, x.x.x.128, and so on).

var fwdCursor = vpc.base;
for (var ni = 0; ni < normalTiers.length; ni++) {
  var ntier = normalTiers[ni];
  var sz = Math.pow(2, 32 - ntier.mask);
  for (var az = 0; az < 2; az++) {
    var aligned = Math.ceil(fwdCursor / sz) * sz;
    var end = aligned + sz - 1;
    if (end > vpc.end) { overflow = true; break; }
    allocRows.push({ kind:'subnet', tier: ntier.name, az: AZS[az], azIdx: az,
      cidr: ipToStr(aligned) + '/' + ntier.mask, first: ipToStr(aligned), last: ipToStr(end),
      total: sz, usable: sz-5, tail: false, startNum: aligned, endNum: end });
    fwdCursor = end + 1;
  }
  if (overflow) break;
}

The line that does the real work is Math.ceil(fwdCursor / sz) * sz: it rounds the current cursor position up to the nearest multiple of the subnet size. That means even when one tier is packed right up against the end of the previous one, the result always snaps to a valid CIDR boundary. If the allocation would run past the end of the VPC, an overflow flag stops the calculation.

3. TGW tiers: round down to the alignment boundary, from the end

Any tier whose name contains "TGW" gets allocated in reverse, starting from the end of the VPC. This is the feature that made me build the tool in the first place.

function floorAligned(addr, size) { return Math.floor(addr / size) * size; }

var bwdCursor = vpc.end;
var tgwBuilt = [];
var tgwRev = tgwTiers.slice().reverse();
for (var gi = 0; gi < tgwRev.length; gi++) {
  var gtier = tgwRev[gi];
  var gsz = Math.pow(2, 32 - gtier.mask);
  for (var gaz = 1; gaz >= 0; gaz--) {
    var galigned = floorAligned(bwdCursor - gsz + 1, gsz);
    var gend = galigned + gsz - 1;
    if (galigned < vpc.base) { overflow = true; break; }
    tgwBuilt.unshift({ kind:'subnet', tier: gtier.name, az: AZS[gaz], azIdx: gaz,
      cidr: ipToStr(galigned) + '/' + gtier.mask, first: ipToStr(galigned), last: ipToStr(gend),
      total: gsz, usable: gsz-5, tail: true, startNum: galigned, endNum: gend });
    bwdCursor = galigned - 1;
  }
  if (overflow) break;
}

This mirrors the forward-allocation logic. bwdCursor - gsz + 1 gives a candidate starting address, and floorAligned rounds it down to a valid boundary — the mirror image of rounding up. The TGW tier list is reversed before processing, and each result is pushed onto the front of the array with unshift, so on screen the tiers still appear in their original order even though they're being filled in from the tail.

4. Detecting free address space

Once forward and tail allocations are done, all subnets get sorted by starting address, and any gap between two adjacent subnets gets recorded as free space.

var allSubnets = allocRows.concat(tgwBuilt).sort(function(a,b) { return a.startNum - b.startNum; });

var gaps = [];
var cursor = vpc.base;
for (var si = 0; si < allSubnets.length; si++) {
  var s = allSubnets[si];
  if (s.startNum > cursor) gaps.push({ kind:'free', startNum: cursor, endNum: s.startNum-1 });
  cursor = s.endNum + 1;
}
if (cursor <= vpc.end) gaps.push({ kind:'free', startNum: cursor, endNum: vpc.end });

This is the classic "find the gaps between intervals" algorithm. A cursor walks forward from the start of the VPC; whenever there's a gap between the cursor and the next subnet, it gets recorded, and if the cursor never reaches the end of the VPC, the leftover range at the end is recorded too. The odd leftover addresses caused by alignment — the space between the last forward-packed subnet and the TGW subnets, for example — get caught by this same mechanism, with nothing falling through the cracks.

Put together, two simple algorithms — rounding up or down to an alignment boundary, and finding gaps between intervals — are enough to handle a fairly involved requirement (multiple tiers, multiple AZs, tail allocation) in a few dozen lines of serverless code.

If you found this helpful, please ⭐ the repository!

📌 You can see the entire code in My GitHub Repository.

CTF Event Report: Security-JAWS 10th Anniversary Day 2 — All 27 AWS Security Challenges Solved

TOMOAKI ishihara — Mon, 18 May 2026 13:13:37 +0000

Introduction

I participated in the CTF held on Day 2 of "Security-JAWS DAYS ~10th Anniversary Event~", organized by Security-JAWS, a Japanese AWS user community focused on cloud security.

The CTF was themed around a fictional SaaS company called "TechVault", and the scenario had us conducting a penetration investigation — starting from their employee portal and ultimately uncovering evidence of fraudulent transactions. It was an exceptionally well-crafted CTF with a cohesive narrative running through all challenges.

Event Overview

Duration: 13:00–17:00 / 4 hours
Total challenges: 27
- Tutorial: 6 / 290 pt
- Mainline: 12 / 2,300 pt
- Bonus: 5 / 1,400 pt
- Advanced: 2 / 700 pt
- Blue Team: 1 / 300 pt
- Finale: 1 / 600 pt
Setting: AWS environment of a fictional SaaS company "TechVault"

The story begins with an intrusion investigation of TechVault's portal service and culminates in gathering evidence of someone's fraudulent transactions. The level of polish in the scenario design was remarkable.

Results

Challenges solved: 27 out of 27
Score: 5,310 pt (max: ~5,590–5,650 pt)
Time to complete: 2 hours 48 minutes 36 seconds
Final ranking: 12th out of 125 participants
What went well:
- Using knowledge and CLI tools I don't normally touch in daily work, and working through them hands-on gave me a much deeper understanding of each technique.
What I'd improve:
- I should have set up my environment beforehand. I normally use devContainers, so my host machine only had the minimum: AWS CLI and Python. Docker, OpenSSL, Boto3, and similar tools were missing, which cost me more time than necessary.

Challenge Structure

Tutorial (290 pt)

A step-by-step introduction to web reconnaissance and AWS CLI basics.

ID	Title	Description
T1	Web Recon · robots.txt	Discover hidden paths from `robots.txt` Disallow directives
T2	The Unlocked Warehouse · Public S3	Retrieve files directly from a publicly exposed S3 bucket
T3	Behind the Page · HTML Source	Investigate credentials buried in HTML comments
T4	First Steps with curl	Check information embedded in HTTP response headers
T5	Leaked Config · .env File	Find a `.env` file mistakenly placed in the web root
T6	First Steps with AWS CLI	Use the key found in `.env` to run `sts get-caller-identity`

The T5→T6 flow was clever. You grab a key from .env and immediately use it with the AWS CLI — a hands-on demonstration of how a web vulnerability becomes an AWS entry point.

Mainline (2,300 pt)

The core of the CTF: an attack chain that follows the path of intrusion → privilege escalation → evidence collection, starting from Stage 0.

ID	Title	Description
Stage 0	The Forgotten Debug Mode	Extract AWS keys from debug output left in an auth API error response
Stage 1A	Flip the Bucket	Find a file hidden under a `.hidden/` prefix in an S3 bucket
Stage 1B	Who Am I?	Read IAM policy metadata to understand the compromised user's permissions
Stage 1C	The Past Never Disappears	Recover AWS keys left in a Git repository's commit history
Stage 1D	Ask the AI	Prompt injection against an AI assistant embedded in the employee portal
Stage 2A	The Deleted File	Recover a deleted file using S3 object versioning
Stage 2B	The Permission Map	Use `sts:AssumeRole` to pivot laterally into the DataAnalystRole
Stage 2D	The Function's Secret	Retrieve sensitive data stored in Lambda environment variables
Stage 2E	The Parameter Labyrinth	Navigate SSM Parameter Store paths to collect secrets
Stage 2G	The AI's Permissions	Extract S3 data via an over-privileged Bedrock agent
Stage 3A	The Vault Key	Retrieve the ZIP decryption password from Secrets Manager
Final	Consolidate the Evidence	Decrypt the evidence file using all collected information to expose the CEO's fraud

The Bedrock agent challenge (Stage 2G) was fresh. The agent was configured with direct S3 access, so data from a bucket I couldn't read directly could be pulled out simply by asking the agent "show me the project metadata." It drove home how important permission design is when integrating AI into your stack.

Bonus / Advanced / Blue Team / Finale (3,000 pt)

Additional challenges branching off the mainline, each requiring deeper technical knowledge.

ID	Title	Category	Description
Stage 2C	The Server's Shadow	bonus	Flag stored in EC2 instance tags
Stage 2F	Find It Automatically	bonus	Scan all branches for secrets using `gitleaks`
Stage 3B	The Invisible Voice	bonus	SSRF to IMDSv1 to steal EC2 role temporary credentials
Stage 3C	The False Face	advanced	Self-declare `custom:role=admin` during Cognito sign-up
Stage 3D	The Truth Inside the Image	bonus	Recover files deleted by `RUN rm` from Docker image layers
Stage 3E	The Neighbor's Vault	advanced	Read another tenant's data via wildcard permissions on S3 Vectors
Stage 4	Follow the Trail	blueteam	Identify attacker operation timestamps from CloudTrail logs
Stage 5	Suspicious Activity	finale	Decrypt CTO complicity evidence by tracing late-night activity in CloudTrail
Stage 5B	Combined Attack Surface	bonus	Call an internal API by combining intelligence from Stage 3D and Stage 3E

Stage 3D was by far the most time-consuming for me — because Docker wasn't installed in my CTF environment. Instead of using docker history (which would have shown it in seconds), I had to query the ECR API directly to fetch the image manifest, download each layer, and extract the tarballs manually. Painful on the clock, but I ended up with a much deeper understanding of how Docker image layers actually work.

Stage 4 and Stage 5 involved parsing large CloudTrail log files with jq to reconstruct the attacker's footsteps — a great taste of what SOC/incident response work feels like. Stage 5 in particular required chaining multiple steps: find the suspicious late-night (JST) operations in the logs, track down the Secrets Manager path they pointed to, and decrypt the encrypted evidence file with OpenSSL. OpenSSL wasn't available either, so I ended up implementing the decryption in Python.

The Full Attack Chain

Each challenge looks independent, but they're all connected as a single story.

Obtain AWS keys from debug API response (Stage 0)
        ↓
IAM recon reveals an AssumeRole-able role (Stage 1B)
        ↓
Pivot laterally into DataAnalystRole (Stage 2B)
        ↓
┌─────────────────────────────────────────────────────┐
│  Collect intelligence across multiple parallel paths │
│  · EC2 tags (Stage 2C)                              │
│  · S3 versioning — recover deleted files (Stage 2A) │
│  · Lambda environment variables (Stage 2D)          │
│  · SSM Parameter Store (Stage 2E)                   │
│  · SSRF → IMDSv1 (Stage 3B)                        │
│  · ECR Docker layer analysis (Stage 3D)             │
│  · S3 Vectors cross-tenant leak (Stage 3E)          │
└─────────────────────────────────────────────────────┘
        ↓
Retrieve password from Secrets Manager (Stage 3A)
        ↓
Decrypt ZIP to obtain CEO fraud evidence (Final)
        ↓
Trace CTO complicity via CloudTrail (Stage 4 → 5)

Each individual vulnerability might look limited in isolation, but chaining them together produces a critical breach. Stage 5B is the perfect example: an internal API only reachable by combining intelligence gathered from two separate advanced stages.

Key Takeaways and Mitigations

Information Leakage (Debug, Headers, etc.)

Debug output that's convenient during development can leak AWS keys if left enabled in production. HTML comments, response headers, and robots.txt are all reconnaissance vectors attackers regularly check.

Mitigations:

Disable debug mode in production environments.
Remove unnecessary response headers like X-Powered-By.
Never place secrets in front-end source code.

S3 Misconfiguration

Three distinct S3 issues appeared: public access enabled, a .hidden/ prefix used as security-by-obscurity, and deleted files recoverable via versioning. All three stem from treating S3 like a traditional filesystem.

Mitigations:

Enable Block Public Access on all buckets.
Prefixes are not access controls.
If versioning is enabled, also design lifecycle policies to expire delete markers and old versions.

Secrets in Git History

Even after deleting a .env file and committing the removal, git log -p surfaces it instantly. Tools like gitleaks can scan every branch and every commit in seconds.

Mitigations:

Integrate git-secrets or gitleaks as a pre-commit hook.
If a secret was already committed, rewrite history with git filter-repo and rotate the key immediately.

Prompt Injection

A single sentence — "ignore previous instructions" — was enough to extract the contents of the system prompt. Using the system prompt as a "hidden" information store is not a security boundary.

Mitigations:

Never put sensitive information in system prompts.
Validate both inputs and outputs. Make the boundary between user input and system instructions explicit.

Overly Broad IAM Permissions and AssumeRole

Having sts:AssumeRole allows switching to a different role. In this CTF, flags were embedded in IAM policy descriptions and EC2 tags for challenge purposes — but in the real world, metadata fields are an underappreciated place for sensitive data to accumulate.

Mitigations:

Apply the principle of least privilege rigorously.
When granting sts:AssumeRole, restrict the target resources.
Use Condition keys in trust policies to restrict callers.

Poor Secret Management

Three storage locations appeared: Lambda environment variables, SSM Parameter Store, and Secrets Manager. Even Secrets Manager provides no protection if the IAM permissions granting GetSecretValue are too broad.

Mitigations:

Manage secrets in Secrets Manager.
Scope the GetSecretValue resource policy to the specific secret ARN.

SSRF × IMDSv1

A URL preview feature in the dashboard was fetching external URLs server-side — and there was no filtering to block requests to http://169.254.169.254. IMDSv1 requires no token, so SSRF access to the link-local address yields EC2 role temporary credentials directly.

Mitigations:

Enforce IMDSv2 (HttpTokens: required) on all EC2 instances.
URL-fetching features should use an allowlist, and must block private IP ranges and link-local addresses.

Secrets Persisted in Docker Image Layers

A Dockerfile pattern like COPY secret.txt . → RUN python setup.py → RUN rm secret.txt produces a final image where secret.txt is not visible at runtime. However, downloading the image layers directly from the ECR API reveals secret.txt intact in a prior layer's tarball.

Mitigations:

Use multi-stage builds; never copy secrets into build contexts.
Retrieve secrets from Secrets Manager at runtime instead.

Broken Multi-Tenant Permission Design

The S3 Vectors resource policy was set to Resource: "*", allowing a role scoped to one tenant to query another tenant's vector data. Tenant isolation in SaaS demands rigorous permission separation.

Mitigations:

Constrain the Resource and Condition in resource policies to tenant-specific identifiers.
If sharing a vector bucket, scope queries and metadata access by tenant at the API level.

Cognito Authorization Design Flaw

Passing custom:role=admin in --user-attributes during aws cognito-idp sign-up was enough to self-declare administrator status, which the application then trusted for authorization decisions.

Mitigations:

Control role assignment server-side (e.g., in a Pre Sign-up Lambda Trigger).
Never use attributes that external parties can set as the basis for authorization.

AI Agent Over-Privilege

The Bedrock agent's IAM role had access to S3 buckets that the DataAnalystRole itself could not read. By asking the agent a natural-language question, data from otherwise-inaccessible buckets was pulled out indirectly.

Mitigations:

Apply the principle of least privilege to AI agent roles as well.
Explicitly enumerate and restrict the resources an agent is permitted to access.

CloudTrail for Evidence Preservation

With CloudTrail logs in place, "what happened, when, and by whom" can be reconstructed almost completely. A handful of jq filters were enough to trace the attacker's full activity.

Mitigations:

Enable CloudTrail in all regions and ship logs to S3.
Pair with GuardDuty for real-time detection.
Apply Object Lock to the log bucket to prevent tampering.

Closing Thoughts

All 27 challenges together gave me a visceral sense of how AWS misconfigurations cascade. Each individual problem represented a realistic "seen-in-the-wild" vulnerability — but what made this CTF special was that they were all woven into a single coherent story.

AWS certifications don't teach you why something is dangerous. Solving these challenges hands-on — making mistakes, working around missing tools, figuring out the low-level APIs when Docker wasn't available — built an intuition that studying documentation alone never could.

Highly recommend participating if a similar opportunity comes around. And if you're building on AWS, I hope this report serves as a useful checklist of things worth double-checking in your own environment.

DEV Community: TOMOAKI ishihara

Automating CodeCommit PR Reviews with Amazon Bedrock: 986 Reviews in 10 Weeks, $0.18 Each, and What We Could Not Measure

TL;DR

Table of Contents

Introduction

Architecture Overview

How the PR Review Works

Building the diff from blobs

Routing by path

Skipping a review

Combining multiple models with Agents-as-Tools

Comment limit: 10,240 bytes

Avoiding duplicate posts for the same commit

Measuring the Effect

① Receive review content as structured data, not free text

Agreement between models

② Cost and usage: add metadata to the invocation log

③ Record the time until merge

Visualization

CloudWatch dashboard

Daily, weekly, and monthly reports

Usage dashboard (optional)

Thinking About Cost

Results in Production

Estimate vs. actual

Design Decisions

Limitations and Future Improvements

Conclusion

Testing the AWS WAF Mitigations for PHP2Shell on Both CloudFront and an ALB

Introduction

TL;DR

Architecture

Telling which WAF and which rule acted, from HTTP headers

Result 1: rule conditions (reproducing the ALB test on both layers)

Result 2: string position vs inspection limit

Result 3: oversize rule in BLOCK (8,192 bytes)

Result 4: with WAF on both CloudFront and the ALB, the ALB blocks legitimate requests

Result 5: restricting ALB access to CloudFront

Other observations

Cost

Wrap-up

References

AWS CDK 100 Drill Exercises #013: CloudFront S3 Static Website — Serving a Private Bucket over HTTPS with OAC and Cross-Region WAF

Introduction

What you'll learn in this article

Architecture Overview

Data Flow

Implementation Highlights

1. Private bucket + OAC — the polar opposite of "who can read this"

2. Bulk security headers via ResponseHeadersPolicy

3. Gentle with the SPA, honest about real failures

4. CLOUDFRONT-scoped WAFv2 is pinned to us-east-1 — wire it with crossRegionReferences

5. Allow-listing before or after the managed rules

6. WAF logs go directly to S3, with sensitive headers masked

Deploy & Verify

Cost Estimate

Summary

References

AWS CDK 100 Drill Exercises #012: S3 Static Web Site — Getting a "Public" Bucket Policy Past Block Public Access

Introduction

What you'll learn in this article

Architecture Overview

Data Flow

Implementation Highlights

1. websiteIndexDocument alone doesn't finish the job — the tug-of-war with Block Public Access

2. enforceSSL: false — a setting that looks wrong at first glance

3. Auto-detecting the deploy operator's IP so the site is instantly viewable

Deploy & Verify

Cost Estimate

Summary

References

AWS CDK 100 Drill Exercises #011: A CodePipeline for CloudFront & S3 — and a self-reference bug

Introduction

What you'll learn in this article

Architecture Overview

Data Flow

Implementation Highlights

1. Let CDK grant IAM permissions per action — don't hand-roll a pipeline role

2. Split "upload" from "cleanup" instead of one aws s3 sync --delete

3. The continuation-token pattern for CloudFront invalidation

2. Bulk security headers via `ResponseHeadersPolicy`

4. CLOUDFRONT-scoped WAFv2 is pinned to us-east-1 — wire it with `crossRegionReferences`

1. `websiteIndexDocument` alone doesn't finish the job — the tug-of-war with Block Public Access

2. `enforceSSL: false` — a setting that looks wrong at first glance

2. Split "upload" from "cleanup" instead of one `aws s3 sync --delete`

5. Two more bugs that only a real `Template.fromStack()` run would catch

3. Cost Anomaly Detection's `thresholdExpression`

5. AWS Chatbot's guardrail policy default is `AdministratorAccess` (for both Slack and Teams)

6. Reusable cost constructs (`common/constructs/cost/`)