DEV Community: Pawel Zubkiewicz

AWS billing horror stories: 10 monsters living in your bill

Pawel Zubkiewicz — Sat, 31 Oct 2026 13:56:24 +0000

Halloween is a good time for scary stories. The thing is, I don't have any ghost stories to tell. I have something much scarier: AWS bills 👻.

I skipped the stories about access keys pushed to GitHub and crypto miners spun up in every region. That's a topic for the security team. What scares me more are the cases where nobody stole anything. The account was secure, the code worked, and the bill still went through the roof.

Because AWS bills you for the work done, not for its value to your business. If something does the same work a million times, you pay a million times more.

I collected ten such stories, including my own. Together they add up to about $258,000 before refunds (and one of them is an especially sneaky one). Where the source gives the volume, I recalculated the amount from the AWS price list. Sometimes it matches to the cent. For every story I also checked whether the monster is still alive, because AWS has closed some of these traps since. All prices are for us-east-1. In European regions they are sometimes a bit higher.

At the end you'll find a 15-minute checklist. You can run it on your own account tonight. 🎃

Strangers knocking at the door

The first three stories have one thing in common: the bill was generated by someone the account owner didn't know and didn't invite.

1. The haunted house: an empty bucket for $1,300

Let me start with a classic. With a Polish twist, too.

In April 2024, Maciej Pocwierz created a private, empty S3 bucket in eu-west-1. He needed it for a simple proof of concept. Two days later he checked his billing page to make sure he was still within the Free Tier. He saw more than $1,300.

His bucket name was identical to a default value in the configuration of a popular open-source backup tool. Every installation that nobody had reconfigured tried to upload its backups to his bucket. Almost 100 million PUT requests in one day. Every single one rejected with 403 Access Denied.

From a permissions point of view, everything worked: no stranger wrote a single file. From a billing point of view, not so much, because the bucket owner paid for every rejected request.

AWS first replied that this was "expected behavior". In the end the bill was cancelled, as an exception. The story went viral, and Jeff Barr publicly admitted that it had to be fixed.

I recalculated: 100 M PUT × $0.005 per 1,000 requests = $500 per day. So $1,300 is about two and a half days at that rate. On top of that, requests without a region first went to us-east-1 and were redirected, which added cost. The order of magnitude matches.

Is the monster alive? No. It's the only monster in this list that AWS really killed. The change was announced on 13 May 2024, and by August 2024 it covered all S3 APIs and regions. Watch out for one detail, though: only 403 errors that come from outside your account or AWS Organization are free. If your own application hammers a bucket without permissions, you still pay.

Exorcism: unique bucket names (with a random suffix), not backup, data or test.

2. The mummy: a file that woke up after years

In July 2020, Chris Short opened the email with his bill. He usually paid about $23 a month. For June: $2,657.68.

A 13.7 GB virtual machine image sat in a public S3 bucket. Cloudflare stood in front of the bucket, so Chris assumed the CDN would cache everything. But Cloudflare's Free, Pro and Business plans don't cache files larger than 512 MB. So every download went straight to S3, and S3 charged for data transfer to the internet. More than 30.6 TB in two days: $1,011.59 on 23 June and $1,639.07 on 24 June.

Who was downloading that file? Nobody knows to this day. Corey Quinn summed it up: Chris didn't do anything wrong. AWS refunded everything except about $40.

I recalculated: the first 10 TB × $0.09/GB + the next 20.6 TB × $0.085/GB ≈ $2,650. Less than 1% off the bill.

The same thing, 2026 edition: in March, an r/aws user described how, after his site was promoted on a subreddit, someone downloaded files from his bucket non-stop for three days. About 160 TB and about $15,500. AWS first reduced the bill only partially, and the author still couldn't pay the rest. AWS waived the remaining amount only after six weeks, and after the author, on a commenter's advice, wrote to Jeff Barr. Transferring 160 TB alone costs about $12,000 according to the price list, so the order of magnitude matches. A caveat: this is an account from an anonymous Reddit user, with no breakdown of the rest of the amount.

Is the monster alive? Yes. The 2024 change doesn't help here, because downloading a file from a bucket someone can access is a valid, billable request.

Exorcism: don't expose S3 directly to the internet. Serve files through CloudFront with Origin Access Control, and check whether your CDN really caches large objects.

3. A swarm of bats: 207 million requests in three hours

Web Dev Cody, a YouTuber well known in the JavaScript world, woke up in March 2024 to an email from AWS Budgets. He had set an alert at about $10. The bill was $269, almost $260 of it CloudFront.

For roughly three hours, while Cody slept, his Icon Generator app received more than 207 million requests. The /community page alone got about 25 million of them, and those responses came from cache. The app didn't have to compute anything, yet every HTTPS request was billed. Cody suspected a DoS attack or a script. The traffic stopped on its own. The bill stayed, and Cody doesn't mention any refund.

I recalculated: the bill in his video shows a line of 207,146,239 HTTPS requests in the US region. × $0.01 per 10,000 requests = $207.15. That's exactly the amount on that line. To the cent. Plus $31.37 for data transfer (these are two lines from the part of the bill shown, not the full breakdown).

The alert worked as a notification, not as a brake. Threshold $10, bill $269, and the traffic stopped before Cody woke up. Thanks to the alert, though, he found out the same day. He added a rate-limit rule in AWS WAF and closed the case.

Is the monster alive? Weakened. Since 18 November 2025 CloudFront has flat-rate plans (from the free Free plan to Premium at $1,000 a month) with no overage charges. Blocked DDoS attacks and requests rejected by WAF don't count against the allowance. But this is an option, not the default. A plan covers a specific distribution, not your whole backend, and an existing distribution doesn't get one automatically. On classic pay-as-you-go, the monster is alive and well.

Exorcism: a rate-limit rule in AWS WAF, or a flat-rate plan for projects that don't make money from traffic.

Loops that feed themselves

4. The poltergeist: a function that calls itself

June 2022. Ruslan Gainutdinov ran a Next.js site with Lambda@Edge. By mistake, the function started calling its own CloudFront distribution. According to the author, the recursive requests ran at about 17,000 per second, each with a 30-second timeout.

The bill: about $4,500.

The most interesting detail is another one, though. Ruslan had a budget alert set at $300. The alert arrived when the charges were already $1,484. CloudFront charges can show up on your bill with a delay of up to 24 hours. The author calculated that if the alert had arrived on time, he would have paid about $400.

I recalculated: assuming 128 MB of memory, one 30-second invocation is 3.75 GB-s × $0.00005001 ≈ $0.00019. So $4,500 is about 24 million invocations, which is just under half an hour at a steady 17,000 per second. The rate was probably a peak, but the order of magnitude matches.

Is the monster alive? In this form, yes. AWS detects and stops Lambda loops (since July 2023 for SQS and SNS, since October 2024 for S3, and according to the current documentation also for custom EventBridge buses). A loop through CloudFront and HTTP is not on that list.

Exorcism: a rate limit in WAF on the distribution, a header or counter that breaks the recursion, and a CloudWatch alarm on invocation count (Lambda@Edge metrics live in the regions where the replicas ran). The invocation count grows immediately. The dollars show up hours later.

5. My own poltergeist

I know this story best, because I wrote it myself.

When I was learning serverless and event-driven architecture, I built a simple flow. A file lands in S3, the event triggers a Lambda, the Lambda starts a Step Functions state machine, and the state machine processes the file and writes the result back to S3.

Nothing suspicious. Except that I put the input prefix and the output prefix in the same place.

So writing the result started the next round of processing. I built an infinite loop and simply didn't notice it for a while. The wake-up call came by email: I had used up the Step Functions Free Tier. After a few dozen minutes of playing around. 😅

Luckily, I stopped my tests at that point. And it was me who stopped the loop, not AWS. On accounts with the classic Free Tier, running out of it doesn't stop anything, it just starts charging.

Scale: the Step Functions (Standard) Free Tier is 4,000 state transitions per month. A state machine with a few states uses that up in a few hundred executions, and a loop does those in minutes. After that, every 1,000 transitions cost $0.025.

I don't have a six-figure bill to show here. What I do have is a very personal memory of the moment I understood that automation repeats my mistake just as efficiently as it does useful work.

Is the monster alive? Yes. Lambda loop detection covers S3, but my chain went through Step Functions. I wouldn't assume detection catches it.

Exorcism: input and output in separate buckets, or at least in separate prefixes with a filter on the S3 event.

Good intentions, terrible bill

In this category nobody did anything stupid. Everyone was trying to do things properly.

6. The vampire: a NAT Gateway that drinks every gigabyte

November 2025. The Geocodio team built a new process that synced large datasets from EC2 to S3. They knew that data transfer between EC2 and S3 in the same region is free. And that's true.

A few days later, an alert came from AWS Cost Anomaly Detection. In one day, 20,167.32 GB went through a NAT Gateway.

So where did the bill come from? From the path the data took.

The instances were in a private subnet. If they had been in a public one, the traffic wouldn't have gone through NAT at all. A private subnet has no direct route out of the VPC. Its route table sends all traffic leaving the VPC (0.0.0.0/0) to the NAT Gateway. And S3 isn't part of your VPC. It's a service with a public address. Without an S3 Gateway Endpoint, every request to S3 goes through the NAT Gateway, then through the Internet Gateway, and only then reaches the public S3 endpoint.

An important detail: this traffic doesn't go out to the public internet. According to the Amazon VPC FAQ, traffic between AWS resources stays on the AWS network, even when it uses public addresses. And the transfer from EC2 to S3 itself is still free, exactly as the price list says.

You pay for something else. A NAT Gateway charges $0.045 for every gigabyte it processes, no matter where that gigabyte is going. The transfer was free. The gate it passed through was not.

An instance in a public subnet (with a public IP and a route to the Internet Gateway) wouldn't have this problem, because the Internet Gateway doesn't charge for data processing. And a Gateway Endpoint adds a direct route to S3 to the route table, bypassing NAT. It costs nothing.

I recalculated: 20,167.32 GB × $0.045 = $907.53. Exactly the amount Geocodio reports for that one day. In January 2026 they added that AWS refunded the full amount.

Is the monster alive? Yes. The Gateway Endpoint for S3 and DynamoDB is free, but you still have to add it yourself.

Exorcism: a Gateway Endpoint for S3 and DynamoDB in every VPC, ideally in your IaC template, so nobody forgets it.

7. The ghost in the logs: one `print` for $1,500

On 4 August 2020, Michael Malis, CEO of Freshpaint, got a budget alert email. It was the fourth day of the month, and the company had already spent more than $1,600 against a budget of $1,000 for the whole month.

Almost $1,500 went to CloudWatch. The cause: a print statement added during debugging that logged every event passing through the backend. It made it to production and generated terabytes of logs. The whole cost was a single line item: DataProcessing-Bytes, meaning log ingestion into CloudWatch at $0.50 per GB ($0.63 in Frankfurt).

I recalculated: $1,500 / $0.50 = 3,000 GB, about 3 TB of logs. The author writes about "multiple terabytes". It matches.

An important detail: shortening retention wouldn't have changed anything. You pay for ingesting logs, not only for storing them.

Is the monster alive? Yes. The Infrequent Access log class is cheaper, but volume still costs money.

Exorcism: log level set in configuration, not in code, and an alarm on log volume for every log group.

8. The curse of defaults: CloudTrail for $200,000

In my opinion, this is the scariest story in the list. Precisely because it's so boring.

Bharath Nadar described how, while onboarding a security tool, someone accepted its default configuration: CloudTrail Data Events for all reads and writes in all S3 buckets, across the whole AWS organization. Full audit, compliance, security. Who would object?

The thing is, CloudTrail charges $0.10 for every 100,000 data events. And buckets that applications read billions of times a month generate billions of events.

The cost grew slowly: from about $1,000 a month in 2024 to about $30,000 a month in 2025. There was no sudden spike, so nothing raised an alarm. The author put it this way: "It didn't spike. It crept." He found the problem manually, in Cost Explorer, at the end of 2025. Total loss: about $200,000. The fix took 5 minutes: switching logging to writes only.

I recalculated: the author reports 8.98 billion S3 events in December, in us-east-1 alone. × $0.10 per 100,000 = $8,980 in one month and one region. That's consistent with the roughly $10,000 a month the author gives for a single production account.

A caveat: this is one person's account, with no company name. The mechanism and the numbers are consistent, though.

Is the monster alive? Yes. Data Events for high-volume S3 reads will always be expensive.

Exorcism: narrow Data Events with selectors to specific buckets and to writes. And before you enable a new tool, check how much the services it turns on will cost.

Monsters you can't see

9. The zombie: you deleted it, but it keeps walking

October 2024. Elliott King was building a side project on AWS Amplify Gen 2 and got a bill of about $1,125.

He deleted the sandbox environment with npx ampx sandbox delete. The DynamoDB tables disappeared. The Amazon OpenSearch Service domains stayed and kept running. Every new sandbox added another domain.

AWS adjusted the bill as a one-time courtesy. A long discussion grew under his post on Hacker News, so Elliott was definitely not alone.

Is the monster alive? I don't know. At the end of October 2024 the author wrote that the problem still existed. I found no confirmation that it has been fixed. What I do know is that the zombie is always alive when you trust a delete command without checking what it left behind.

Exorcism: after every environment deletion, check resources by tags (Tag Editor in Resource Groups). And set a budget on every account, including the one "just for fun".

10. The invisible monster: $30,000 for Claude Opus

This is the freshest story. May 2026, The Register. One of its readers used Claude Opus through Amazon Bedrock. He had Cost Anomaly Detection configured. The first sign that something was wrong was the invoice: $30,141.33 for April, plus about $675 in other charges.

Why did anomaly detection stay silent? Third-party models in Bedrock are billed through AWS Marketplace. And at that time Cost Anomaly Detection didn't see Marketplace charges at all. An AWS spokesperson confirmed this directly to The Register.

On top of that, AWS Activate credits quietly covered the first $8,026.54. When they ran out, nobody got a notification.

Is the monster alive? Weakened. On 19 August 2026 AWS announced that Cost Anomaly Detection now covers third-party models in Bedrock, with no extra setup. The documentation still says, however, that other Marketplace charges are not monitored. For those you need AWS Budgets.

Exorcism: a budget on the total account cost (it includes Marketplace) and a separate alert for credits running out.

Why the alarm didn't ring

When I put these stories side by side, I saw something more interesting than the amounts. Some people had no alarm at all. But where there was an alarm, it failed in four ways.

1. The alarm was late. Ruslan had a $300 alert that arrived at $1,484. Cody had a $10 alert and paid $269. That wasn't a bug. According to the documentation, AWS Budgets updates its data up to three times a day, usually every 8 to 12 hours. AWS itself writes that you can exceed the threshold before you get the email. Cost Anomaly Detection uses Cost Explorer data and needs up to 24 hours to detect an anomaly.

2. The alarm was blind. The Register reader had Cost Anomaly Detection, but it didn't see Marketplace costs. Credits hid the first $8,000. The alarm was looking the other way.

3. The monster crept. CloudTrail grew for more than a year. Anomaly detection learns your spending pattern and, according to the documentation, accounts for natural growth. So a steady, slow increase is easy to treat as normal. (The author didn't say whether he had anomaly detection. But even if he had, I wouldn't count on it.)

4. The monster was new. Few people know this detail: for a newly used service, Cost Anomaly Detection needs 10 days of history before it starts detecting anomalies. A service you turned on yesterday that got out of control today is invisible to it.

The alerts that did work (Cody, Geocodio, Freshpaint) gave one thing: knowledge within hours or days, not at the end of the month. None of them stopped anything. An alert is a smoke detector, not a fire extinguisher.

So did AWS finally give us a brake?

Partly. For years the answer was "there is no hard spending limit". Budget Actions can apply an IAM policy or an SCP, or stop specific EC2 and RDS instances. They won't stop Lambda, S3 or Bedrock.

On 16 September 2026, AWS introduced a new, project-based way of creating accounts. On the paid plan you can set a spend limit for a project. The minimum is $20 or AWS's estimate of your spend, whichever is higher. The limit applies to pre-tax costs and doesn't include credits. When you reach it, AWS pauses the project and stops its resources. There are also optional earlier thresholds, triggered a few days before the forecast overrun: blocking new resources, pausing idle resources, and even pausing the biggest cost drivers, including Lambda and Bedrock.

Sounds great. But there are three catches:

Spend limits work only in the new account model (AWS Settings). For traditional accounts, which in practice means every existing company, the comparison in the AWS documentation says it plainly: not available.
A spend limit is a type of AWS Budgets, so it uses the same delayed data. A loop that burns thousands of dollars in half an hour will manage to do so.
If a project is paused and you do nothing for 90 days, AWS permanently deletes its data. A brake that, instead of slowing you down, scraps the car.

For learning and experiments, it's a great direction. In production, you still have to rely on yourself.

Are these monsters still alive?

Monster	Mechanism	Status in October 2026
1. Empty bucket	Charges for rejected 403 requests	⚰️ Dead for requests from outside your account and organization (since 2024)
2. Mummy (13.7 GB)	Data transfer from S3 to the internet	🧟 Alive
3. 207 M requests	A charge for every CloudFront request	🩹 Weakened: flat-rate plans since 11.2025, if you choose them
4. Lambda@Edge	Recursion over HTTP	🧟 Alive: loop detection doesn't cover this path
5. My loop	S3, Lambda and Step Functions	🧟 Alive
6. Geocodio	NAT Gateway instead of a Gateway Endpoint	🧟 Alive
7. Freshpaint	Log ingestion into CloudWatch	🧟 Alive
8. CloudTrail	Data Events for S3 reads	🧟 Alive
9. OpenSearch zombie	Resources that survived `delete`	❓ No evidence of a fix
10. Claude on Bedrock	Charges outside anomaly detection	🩹 Weakened: detection since 08.2026

One dead, two weakened, six alive, and one I know nothing certain about. AWS does fix these traps, but slowly. And in two cases (1 and 10) only after someone described the problem publicly.

Monster hunting in 15 minutes

Every one of these monsters leaves a trace in your cost data. Open Cost Explorer, select the last 3 months, set Group by: Usage type and look for these traces. Usage types have a region prefix, for example EUC1-DataProcessing-Bytes for Frankfurt.

Monster	What to look for	What to do
🧛 Vampire (NAT)	A usage type with `NatGateway-Bytes` in its name (under EC2-Other)	Add a Gateway Endpoint for S3 and DynamoDB
🏺 Mummy (transfer)	`DataTransfer-Out-Bytes` for S3	Move public files behind CloudFront
👻 Ghost in the logs	`DataProcessing-Bytes` for CloudWatch	Find the log group with the highest volume
🧙 CloudTrail curse	CloudTrail cost growing month over month	Check your Data Events selectors
🦇 Bats (CloudFront)	CloudFront request cost higher than transfer cost	WAF rate limit or a flat-rate plan

Then a few quick checks in the terminal. Which VPCs have an S3 endpoint and of what type (compare the result with the list from aws ec2 describe-vpcs):

aws ec2 describe-vpc-endpoints --region eu-central-1 --filters Name=service-name,Values=com.amazonaws.eu-central-1.s3 --query 'VpcEndpoints[].{Id:VpcEndpointId,Vpc:VpcId,Type:VpcEndpointType}'

What exactly your CloudTrail trail logs (look for DataResources or AdvancedEventSelectors with S3):

aws cloudtrail get-event-selectors --trail-name YOUR_TRAIL_NAME

Whether you have a budget, an anomaly monitor and a subscription that will send an alert:

aws budgets describe-budgets --account-id YOUR_ACCOUNT_ID

aws ce get-anomaly-monitors

aws ce get-anomaly-subscriptions

An anomaly monitor without a subscription won't notify anyone. If you're missing a budget, a monitor or a subscription, you know what you're doing tonight. 😉

What this all means

After ten stories, one thing is clear: AWS almost always billed exactly as its price list says. In the Geocodio and Cody stories, my recalculation matches to the cent. Monsters rarely come from billing errors. More often they come from defaults and from a price list read without the context of your own architecture.

That's why before every deployment I want to be able to answer three questions:

What in this architecture can run over and over without delivering any new value?
How will I know it's happening?
How will I stop it, and who makes that call?

And after deployment I do three things:

I set up AWS Budgets and Cost Anomaly Detection. Knowing that they are smoke detectors, not fire extinguishers. They ring late and they don't see everything.
I look at the mechanism, not the service. An amount next to "S3" tells you little. What matters is whether you pay for storage, requests or transfer. An amount next to "EC2-Other" tells you even less.
I review trends, not just spikes. The most expensive story in this list didn't have a single spike. It grew calmly for a year.

One more thing. In at least four of these stories, AWS gave the money back. But usually as a one-time courtesy, often only after a public post. That is not a FinOps strategy.

I do this kind of review every day in CloudPouch, because this exact pain is why I built the app. But whatever tool you use, do the review. Before your story ends up in the next list like this one.

And if you have your own AWS bill horror story (no leaked keys!), share it in the comments. I'd love to read it. 🎃

Sources

Maciej Pocwierz, How an empty S3 bucket can make your AWS bill explode, InfoQ coverage and The Register coverage
AWS, announcement of 13 May 2024, announcement of August 2024 and Billing for Amazon S3 error responses
Chris Short, The AWS bill heard around the world
r/aws, $15,000 S3 Bill for DDoS, AWS reduced my $15k S3 bill to $10.5k after a DDoS, Update: My $15.5k AWS S3 DDoS bill has been fully resolved
Web Dev Cody, I got my first DDoS (and what you can do to help prevent it) (the bill is visible around 0:40)
AWS, CloudFront flat-rate pricing plans, plan documentation and CloudFront pricing
Ruslan Gainutdinov, I DDoSed myself using CloudFront and Lambda@Edge and got a $4.5k bill and the Hacker News discussion with the author's comments
AWS, Lambda recursive loop detection and the S3 announcement of October 2024
Mathias Hansen (Geocodio), The $1,000 AWS mistake and the Amazon VPC FAQ (traffic to public AWS addresses stays on the AWS network)
Michael Malis (Freshpaint), Avoiding a Surprise AWS Bill
Bharath Nadar, A $200,000 AWS bill nobody noticed
Elliott King, Amplify overcharge
The Register, Bedrock and a hard place: Claude adventure leaves AWS user staring down $30K invoice
AWS, Cost Anomaly Detection supports third-party models on Amazon Bedrock and the Cost Anomaly Detection documentation
AWS, AWS Budgets: data refresh frequency and Budget Actions
AWS, the new getting-started experience, 16 September 2026, spend limits in AWS Settings and comparison of sign-up options
AWS pricing: S3, VPC and NAT Gateway, CloudWatch, CloudTrail, Lambda, Step Functions

Originally published in Polish on Serverless Polska.

Why AWS Cost Explorer Isn't Enough: The Story of My Own Pain and How CloudPouch Was Born

Pawel Zubkiewicz — Fri, 30 Oct 2026 15:47:01 +0000

It was 2016. I was sitting in front of my monitor, staring at an estimated AWS bill: 300 dollars a month for my side project. $300! For an app used by maybe a handful of people. That was the moment I got it: either I give up on the dream of my own startup, or I find a way to tame cloud costs.

Today, after years of fighting with AWS cost optimization, I know one thing for sure: AWS Cost Explorer is only the beginning of your journey, not the end of it.

Painful beginnings with AWS

I came from the Java Enterprise Edition (JEE) world. When I decided to jump into the cloud in 2016 (at first I even gave Azure a shot!), I thought like a typical developer: "I need a server, a database, a load balancer... The standard enterprise setup."

The result? A staggering $300 a month for infrastructure that handled about as much traffic as a small blog.

That's when I discovered serverless on AWS. Lambda, DynamoDB, API Gateway. These technologies solved my cost problem. They also taught me to think "frugally" (as Dr. Werner Vogels put it many years later at re:Invent 2023).

In the cloud context, being frugal means choosing components with their cost in mind, not just their technical merits.

From $300 a month I went down to practically zero. The app ran for free within the Free Tier.

Cost Explorer: friend or foe?

After a few years with AWS, I started helping clients optimize their costs. And that's where the real headache began.

AWS Cost Explorer is a tool that seems powerful. Colorful charts, filtering by service, region, tag. At first glance it looks impressive.

The problem? Every question about the details ends in frustration.

Try answering these seemingly simple questions with Cost Explorer:

Why did this month cost more than the last one?
What exactly is driving the cost increase in EC2-Other?
And what does the usage type "EU-EBS:VolumeUsage.gp3" even mean?

Cost Explorer will show you a chart. It might even tell you that EC2 costs more. But it won't explain why, and it won't tell you how to fix it.

Working intensively on costs, I built my own techniques and checklists of what to check. Unfortunately, Cost Explorer forces (well, forced) me to keep clicking through the same chart configurations over and over again. A huge waste of time. I hate that, and I always try to automate everything. On top of that, when you're just starting out, the sheer number of options is overwhelming, and useful default charts for beginners are nowhere to be found.

The usage types puzzle: 10,000 ways to pay

As I kept digging into costs, at some point Usage Types became my obsession.

That's why today, when I want to check whether someone knows anything about costs yet, my opening question is: do you know what EC2 - Other is?

AWS has an estimated 10,000 different usage types. Ten thousand! From simple ones like S3 storage, through complicated ones like data transfer between regions, all the way to completely incomprehensible combinations that aren't worth dissecting here.

Example? S3 alone has dozens, maybe even hundreds of usage types:

TimedStorage-Standard: storing data
Requests-Tier1: every GET request
Requests-Tier2: PUT/POST requests
DataTransfer-Out-Bytes: outbound data transfer

And the AWS documentation? Some usage types aren't described anywhere. I even tried to find out what they mean through AWS Support. No luck.

It's like solving a crossword where half the clues have been hidden.

The turning point: the birth of CloudPouch

In 2021 I'd had enough. I was working with a client who paid serious money for AWS, and Cost Explorer couldn't answer my basic optimization questions.

So I thought: "If AWS won't give me the tools I need, I'll build them myself."

My starting point was simple:

See the costs: better than Cost Explorer
Understand them: explain all the usage types
Optimize: find concrete savings

That's how I came up with the tagline for my app: "See. Understand. Reduce AWS costs." After all these years it still holds up, and it shows the path for anyone who wants to become a cloud cost specialist.

CloudPouch was born as a desktop app that:

Connects directly to your AWS account through the API
Keeps all the data on your computer, so zero security risk
Explains Usage Types in plain language
Automatically detects 15 different types of waste (for details, take a look here)
Is built with Electron, so it runs on Windows, macOS, and Linux

When data tells a story

One of my favorite CloudPouch features is historical analysis. I can look at a client's costs and tell them what they've been up to over the past few months.

The cost detective

A real-life example: a client had a huge cost drop, $50,000 between September and October. In Cost Explorer, all they'd see is a line going down.

CloudPouch showed the details:

"NAT Gateway Bytes" down by $18,000 in Ireland
"Data Transfer Regional" down by another few thousand
Similar patterns in other regions

My diagnosis? The client had rolled out VPC Endpoints for S3 and DynamoDB. Data stopped going through the expensive NAT Gateways and started flowing straight to AWS services.

When I called the client with this "prediction", they were in shock. The answer I got: "How do you know? That's exactly what we did!"

This story repeats itself quite often, by the way. My clients are surprised that without talking to them, without access to any project documentation, from the costs alone, I can work out what changes they've rolled out over the last 12 months.

Real results: from 13% to 62% savings

After three years of building CloudPouch and working with dozens of clients, I can say one thing: AWS cost optimization is not a myth.

My stats:

Smallest saving: 13% of monthly costs
Biggest saving: 62% of monthly costs. Here even I was surprised, because my initial estimates pointed to "only" 41%
On average it's 18% to 26% savings per AWS account
Account range: from $3,000 to 1.5 million USD a month, those are the accounts I've worked on
Typical cases of waste:
- EBS volumes attached to stopped instances
- Unused NAT Gateways (no traffic for 14 days)
- RDS with no connections for over 2 weeks
- EC2 rightsizing
- Misconfigured DynamoDB (provisioned vs on-demand)
- Databases or EKS running in Extended Support mode
- and many more.

One example: a client was paying $262 a month for a NAT Gateway that carried... 2KB of data over two weeks. Two kilobytes!

The future: FinOps as the new standard

Building CloudPouch isn't just about solving my own problem. It's about getting ready for a future where cost awareness becomes a non-functional requirement, just like security is today.

The FinOps Foundation (finops.org) is building the only comprehensive framework for managing cloud costs. These aren't theoretical abstractions, but practical best practices drawn from the experience of the biggest companies.

For developers, architects, and cloud engineers this means one thing: soon every technical decision will have to take cost into account.

More than just savings

CloudPouch taught me that cost optimization isn't just math. It's a detective story, where every usage type tells a piece of your infrastructure's story, and I put on my hat and grab my magnifying glass.

And finally, a small note on sustainability: every dollar saved on AWS means less energy burned. When you switch off servers nobody needs, you save money and help the planet 🌍 at the same time. It may not be the main business motivation, but it does give me personal satisfaction.

Similarly, I like to imagine that with large annual savings (sometimes even millions of USD), thanks to my work companies don't have to lay people off, because they cut cloud costs instead of cutting jobs. And that makes someone's life a little better 😀.

What was your most unexpected saving on AWS? Share it in the comments. Maybe your experience will help others avoid costly mistakes.

CloudPouch offers a 7-day trial, no credit card required. Download it from cloudpouch.dev and **see for yourself* how much you can really save.*

What's next with the Serverless Framework in 2025?

Pawel Zubkiewicz — Wed, 05 Feb 2025 16:57:24 +0000

Recent Events Spark a Heated Debate

Recent events surrounding the Serverless Framework have ignited a heated discussion within the user community. As shown in the attached image, the CEO of Serverless Inc. officially closed the door on support for version 3 by rejecting a pull request that would have enabled it to work with the latest runtime environments—including Node.js 22. In his decision, he stated:

“We are no longer working on V3 and are focusing on V4.”

The rejection of this simple PR by none other than the CEO and founder of Serverless Inc. has elicited a range of responses—from understanding to outright hostility. What’s clear, however, is that this decision unequivocally signals the future direction of the Serverless Framework and raises the question: What’s next?

Houston, We Have a Problem!

This is a serious issue for several reasons. First, Node.js 20 (the last supported version in Serverless Framework v3) will reach its end of life in April 2026. Second, the company is firmly pushing users toward version 4, which introduces significant changes in its business model—changes that have effectively deterred the community from adopting this solution from the start.

Version 4 – Licensing Change, a New Business Model

At the end of 2023, Serverless Inc. released version 4 of the framework, introducing a major change: licensing fees for larger organizations.

How does it work in practice?

If your company generates more than $2 million in annual revenue, you’ll need to reach for your wallet. Fees are charged in the form of credits, where one credit equals one deployed service (a “Service Instance”) in a given region and environment for at least 10 days.

For example, a service named foobar deployed on the dev, test, and prod stages in Frankfurt would count as three Service Instances—that is, 3 credits per month. Deploying the same service in another region would count as an additional three Service Instances.

Pricing:

2 Credits: Free
15 Credits: $60 per month
50 Credits: $175 per month
300 Credits: $750 per month

Credits can also be purchased at a significant discount through a reservation model.

Free Usage Terms for V4

Fortunately, these changes don’t affect everyone. Version 4 remains free for:

Independent developers (i.e., us mere mortals)
Consultants working for clients
Open source projects
Startups and companies with under $2 million in annual revenue
Educational institutions

The Community Responds: Forking Serverless Framework V3

As is often the case in the open source world, when a problem arises the community takes matters into its own hands. Thus was born the fork of Serverless Framework v3, known as oss-serverless. This project, available on GitHub (oss-serverless/serverless), aims to extend the life of V3—eliminating the need to switch to the paid V4.

This fork is gaining traction because it allows users to continue working with up-to-date runtime environments, such as Node.js 22, Python, or .NET, without having to change framework versions. Essentially, it’s a copy of V3 with several improvements:

Added support for the latest AWS Lambda runtime versions
Patched security vulnerabilities
Streamlined and sped up the CLI
Removed unused components
All changes are detailed here.

Switching to the fork is ridiculously simple:

Remove the global installation of Serverless: npm remove -g serverless
Install the oss-serverless fork globally: npm install -g osls

What to Choose?

We are now faced with several options:

Switch to V4

If you’re a small company or an individual developer, this might be the natural path forward. For larger companies, the cost of using SFv4 isn’t prohibitive—especially since V4 offers features that can boost developer productivity, potentially recouping the licensing fees many times over.

Of course, bureaucracy can be an issue; sometimes convincing a client to spend an extra $100 on a project costing $100,000 is impossible or downright frustrating. Fortunately, credits can be purchased through AWS, which should ease the process somewhat.
Adopt the oss-serverless Fork

This is a solid alternative for those who can’t or don’t want to move to V4. The community has committed to supporting it for at least the next 5 years.
Stick with V3

Yes, the framework won’t suddenly stop working. However, the lack of updates and support could become problematic over time—especially if you’re looking for better performance and the latest features available in newer runtimes.
Consider Alternatives

Perhaps this is a good moment to explore other options like AWS CDK, AWS SAM, or SST.

Conclusion

The situation with the Serverless Framework teaches us two things.

First, every successful open-source tool eventually needs to find a business model that enables further development. Personally, I don’t hold it against Austen Collins (CEO) for blocking that PR—the change was announced long ago and shouldn’t come as a surprise. I’m hopeful that focusing on V4 and securing more financial resources will allow them to build an even better tool. I root for them and wish them the best of luck.

Second, the power of the open-source community is invaluable—as demonstrated by the oss-serverless fork. Thanks to this initiative, we have options, and that’s a great thing.

Regardless of which path you choose, it’s a good idea to start planning your strategy now. Although Node.js 20 will be supported for more than two years, time in the tech world moves faster than we think.

What do you think? Which option will you choose for your projects?

Bonus: What’s New in Serverless Framework V4?

Despite the controversies surrounding the licensing model, V4 introduces a host of intriguing improvements that may convince users to migrate:

Hybrid Developer Mode

V4 introduces an advanced approach that combines your local development environment with AWS Lambda functions—think hot-deployment. This means developers can enjoy:

Faster development cycles
Forwarding events from the cloud to local code
A more precise testing environment
Greater deployment flexibility

The framework also emphasizes increased freedom when choosing infrastructure:

The ability to switch between Function-as-a-Service (FaaS) and serverless containers
Reduced risk of vendor lock-in
Support for new AWS regions and features

Extensions and Integrations

A new flexible extensions system now offers:

The ability to create your own plugins
A partner program with an 80% revenue share for plugin developers
Support for various serverless service providers

Additional Features

V4 introduces a suite of useful tools:

Integration with HashiCorp Vault for secret management - which is very enterprise-friendly
Built-in support for CI/CD
Access to the Serverless Dashboard at no extra cost

Compatibility

The developers have ensured a smooth transition by:

Avoiding any breaking changes in backward compatibility
Preserving existing workflows

Migrating AWS Organizations: How I Did It and Why

Pawel Zubkiewicz — Sun, 24 Nov 2024 10:22:50 +0000

Migrating AWS Organizations is a rarely discussed topic that quickly becomes an administrative and technical challenge when you need to change the management account. This is because AWS Organizations does not allow changing the organization root account once it has been set. As a result, it's necessary to create a new organization and manually migrate all member accounts to complete the migration.

In my case, apart from migrating AWS accounts to a new management account within a new organization, I also changed the authentication method. I moved from IAM users to AWS Identity Center, formerly known as SSO. In this article, I share my experiences during this process, the challenges I faced, and insights that might be helpful to others.

Why Did I Have to Migrate?

My AWS Organization is my personal setup, which I use for learning and small projects. I created my first AWS account back in 2016 with the intention of learning serverless. As my needs grew, I added more accounts and eventually formed an AWS Organization. Due to a lack of experience at the time, I used my first account as the management account. This account already hosted various resources, primarily serverless projects. Over time, I learned that this setup violated best practices and needed improvement, but I couldn't make changes without a major migration. For a long while, I operated like this—after all, it was my personal organization, and I didn't need to pass any audits. 😉

Then, AWS Identity Center (previously called SSO) came along, and I wanted to enable it. However, I didn't want to do it in my old, disorganized setup, as I knew it would only make it harder to align with AWS best practices in the future. So, I kept postponing it.

Things changed in 2024, with the arrival of AWS Q Developer Pro, which I wanted to enable to assist me in coding. It requires AWS Identity Center, which finally pushed me to act. I wanted to deploy it, but not in the messy old organization. I knew that enabling Identity Center in my current structure would further complicate aligning with best practices.

The third reason, based on some hints and speculation, was that old AWS accounts seem to differ from those created recently. While I don't know the specifics, it's generally better to have a newer account as the management account for the organization. I recall that when AWS finally enabled multiple hardware MFA keys for IAM users, my old AWS account from 2016 only gained this capability a few weeks after the announcement, and AWS Support cited its age as the reason for the delay.

How Did the Migration Go?

In my case, the migration focused solely on transferring member accounts to the new organization, without moving resources between accounts. Here are the main steps:

Setting Up the New Organization – I started by creating a new AWS Organization, setting up a fresh AWS account that would serve as the management account.
Migrating Member Accounts – Next came the most time-consuming part—migrating the member accounts. Each account needed to be detached from the old organization, which involved removing SCP policies that blocked the account from leaving. For accounts created directly within the old organization, I had to set a root password, enable MFA, and—most crucially—define a payment method. The last part was key, as with consolidated billing, the payment method is usually only configured on the management account.
Accepting the Invitation to the New Organization – After detaching an account from the old organization, it became independent. In the new organization, I sent an invitation to the account, which had to be accepted from the root account level. Only then would the account join the new organization.

Challenges I Faced

Payment Issues – The main difficulty was setting up payment methods for accounts that had previously been created in the old organization without needing individual payment setups. I had to do this manually for each account.
Lack of Automation – Another significant challenge was the lack of automation from AWS. The platform doesn't provide tools for automatically transferring entire or partial organizations, which meant I had to execute every step manually. Fortunately, in my case, this involved only seven accounts, so it was manageable within a reasonable timeframe.
SCP and IAM Policies – I transferred the existing SCP policies via copy-paste (JSON definitions), and the IAM configurations on the accounts remained unchanged.

AWS Identity Center - A Fresh Start

Implementing AWS Identity Center in the new organization represented a major shift in my approach to access and security management. You can find more details in my previous article Simplified Configuration of SSO Profiles in AWS CLI Using SSO Sessions. Additionally, unlike in the old organization where resources were deployed directly on the management account, I decided to strictly adhere to AWS best practices and keep the management account empty.

Interestingly, AWS recommends configuring Identity Center on the management account, so it isn't entirely empty, but this is the only exception to the rule of not placing resources on the management account.

Recommendations

If you need help with setting up your AWS Organization or Identity Center, I recommend the excellent instructional videos by Łukasz Dorosz, which helped me get up to speed. In fact, if my memory serves me right, Łukasz was advising me to switch to Identity Center back in 2023. Thank you, Łukasz!

Summary

Migrating AWS Organizations is a complex task that may require many manual steps, but it is achievable with the right approach. In my case, creating a new organization and manually migrating accounts was the best solution, allowing me to organize resources and implement better security practices. Ultimately, it enabled me to use AWS Q Developer Pro.

For those considering a similar migration, the most important advice is to plan the process carefully and prepare all the necessary elements, especially payment setups and root account access for all AWS accounts being migrated. While the process is time-consuming, the benefits of a well-structured setup and increased security far outweigh the effort.

If you're just starting out with AWS Organizations, remember to create a fresh, dedicated management account without any resources. This will save you from the need for migration in the future and ensure you're aligned with AWS best practices from the start.

I hope my story will be helpful to those facing a similar challenge and that it will help you avoid my mistakes when creating your own AWS organization.

Simplified Configuration of SSO Profiles in AWS CLI Using SSO Sessions

Pawel Zubkiewicz — Sat, 23 Nov 2024 16:14:00 +0000

For some time now, configuring Single Sign-On (SSO) profiles in AWS CLI has been simplified thanks to the introduction of SSO sessions. Traditionally, setting up SSO profiles required specifying the SSO endpoint for each profile individually. The new approach uses the sso-session section in the configuration file to group SSO endpoint variables, which profiles can reference. This avoids redundant information and makes the configuration process easier.

Benefits of Using SSO Sessions

SSO sessions offer several advantages over previous methods:

Simplified configuration: Defining the sso-session section and linking it to a profile is easier than manually setting all the parameters.
Automatic token refreshing: SSO sessions automatically renew access tokens, eliminating the need to manually refresh credentials. AWS CLI manages the token renewal process in the background using a "refresh token." When the access token expires (typically after an hour), AWS CLI uses the refresh token to obtain a new access token without user intervention. This allows uninterrupted work and removes the need to repeatedly log in, which is especially useful for long-running operations or background scripts.
Reusable configuration: SSO session configurations can be shared across multiple profiles, making it easier to manage multiple accounts and roles.

Why Choose SSO Sessions Over IAM Users?

Using SSO sessions (IAM Identity Center) in AWS CLI offers significant advantages over traditional IAM users:

Enhanced security: Authentication via IAM Identity Center eliminates the need to manage long-term access keys, commonly used with IAM users. With SSO sessions, credentials are short-lived and automatically renewed, significantly reducing the risk of leaks or misuse.
Centralized access management: IAM Identity Center allows centralized management of user permissions through integration with existing identity providers (IdP), such as Microsoft Azure AD, or by creating your own user directory (I chose this option). Organizations can easily control access to multiple AWS accounts from a single place, facilitating audits and compliance with security policies.
One login, multiple accounts: SSO sessions enable users to access all accounts within an AWS Organizations structure using a single set of credentials. This means users log in once to access multiple AWS accounts without needing to create separate IAM users for each account or assign multiple roles to individual users. This simplifies identity management and enhances user convenience by eliminating the need to remember multiple passwords.

Configuring a Profile Using the `aws configure sso` Wizard

The easiest way to configure an SSO profile using SSO sessions is with the aws configure sso wizard. This command guides users through the setup process, including:

Creating a session name, e.g., myorg.
Providing the IAM Identity Center start URL, e.g., https://myorg.awsapps.com/start.
Specifying the AWS region where the IAM Identity Center directory is located.
Defining the registration scope (optional).
Selecting the target account from a dropdown menu in the terminal (as shown in the image below).

After completing the process, a new entry will appear in the ~/.aws/config file with the following structure. Notice how the profile references the session.

[profile my-new-profile]
sso-session = myorg
sso-account-id = 123456789012
sso-role-name = AdministratorAccess

[sso-session myorg]
sso-start-url = https://myorg.awsapps.com/start
sso-region = us-east-1
sso-registration-scopes = sso:account:access

In practice, the new configuration differs from the old one in that sso-session can be reused across multiple profiles. Previously, everything had to be defined together, duplicating information across profiles in the same organization.

Here’s how the old configuration looked:

; old config, don't use it
[profile my-old-profile]
sso_start_url = https://myorg.awsapps.com/start
sso_region = us-east-1
sso_account_id = 123456789012
sso_role_name = AdministratorAccess
credential_process = aws-sso-util credential-process --profile my-old-profile

Logging In and Using an SSO Profile

After configuring an SSO profile, users can log in using the aws sso login command. This command opens the default browser to authenticate the user. Once logged in, the credentials are cached, enabling AWS CLI to securely fetch AWS credentials for the specified IAM role.

Example usage of an SSO profile:

aws s3 ls --profile my-new-profile

If you use environment variables, you can still set the profile as the default:

export AWS_PROFILE=my-new-profile

BTW: I manage this setup with the direnv tool, allowing me to use different profiles in different directories.

Configuring Additional SSO Profiles

If you noticed in the screenshot, I have access to nine AWS accounts. To configure a profile for another account, I simply run the aws configure sso command again.

Then, I type the previously created SSO session name. This allows me to select another account from the list without re-entering the SSO URL or other session details. No manual edits or copying AWS account numbers from the web console to the config file are needed.

Logging Out of an SSO Session

After finishing with an SSO profile, users can log out using the aws sso logout command. This clears cached credentials.

My Experience with IAM Identity Center

In my AWS organization, I decided to enable IAM Identity Center specifically to use SSO sessions instead of IAM users with static access keys. I must admit, for many years, out of convenience and habit, I didn’t regularly rotate my access keys 😱 (see the image below). However, security concerns pushed me to change my approach. The new authentication method via SSO sessions is significantly more secure, eliminating risks associated with long-term access keys. I now sleep easier knowing my AWS accounts are better protected.

Gradual Transition to SSO and Temporary IAM Key Deactivation

When transitioning from IAM users with static access keys to SSO sessions, you can temporarily deactivate your access keys (Access Key and Secret Access Key) associated with the IAM user. Deactivation prevents the keys from being used for AWS operations but allows later reactivation without generating new keys. This approach provides flexibility during migration—you can configure SSO sessions on your system while disabling IAM keys as a security measure. If needed, you can quickly revert to traditional keys by reactivating them with one click in the AWS console. This makes the SSO migration process safer and less risky, allowing an emergency fallback to the old setup if required.

CloudPouch Supports the New SSO Profile Configuration

CloudPouch is an application for viewing costs and automatically identifying savings on AWS accounts. I’m excited to announce that starting with version 1.36.0, I’ve added support for the new SSO profiles, making it 100% compatible with configurations created by aws configure sso.

Summary

SSO sessions simplify the configuration and management of profiles in AWS CLI. Users can leverage the aws configure sso wizard to quickly and easily set up SSO sessions. This makes managing access to AWS resources more efficient, secure, and user-friendly.

Report from AWS Heroes Summit 2024

Pawel Zubkiewicz — Thu, 25 Jul 2024 10:23:26 +0000

I spent all of last week (July 15-19) in Seattle at Amazon's headquarters for a top-secret event exclusively for AWS Heroes.

How was the AWS Heroes Summit?

In short: fantastic.

Those who know me well know that I don't throw praise around lightly. 😉

Our conference days were packed with many long sessions (an hour and a half to two hours) across 4 tracks. This allowed everyone to choose something interesting for themselves. The sessions featured well-known figures like Developer Advocates who often speak at conferences, but also engineers building services and the managers responsible for them. With all due respect to the DAs, the sessions with the latter two groups were the most interesting. I wasn't familiar with most of these people before.

The engineers from AWS made an incredible impression on me; some of them are absolute rockstars and top-tier experts. Top of the top!

I'm not even sure how to best describe this impression 🤔 I imagine they're specialists of the same caliber as those working for Elon, and when he tells them to accomplish seemingly impossible things, they make it happen regardless of the scale of the problem. And AWS has the same kind of people working for them!

In summary, our favorite serverless services are in good hands. During these long sessions, there was plenty of room for discussions and questions. Interestingly, this worked both ways. The engineers from AWS were very interested in our problems, use cases, and generally what we think. Customer obsession in a new form.

Serverless is not dead!

In recent weeks (before the Summit in Seattle), several blog posts appeared once again proclaiming the death of or disappointment with serverless. This topic was amplified by the ubiquitous GenAI, which media-wise overshadows all other news and changes in AWS, as was particularly evident at the New York Summit, giving the impression that AWS has stopped investing in anything other than artificial intelligence. Luc van Donkersgoed, a Serverless Hero, critically described and summarized this situation in his article.

In a way, I feel obligated to address these opinions and the prevailing anti-serverless climate, as well as the accusations against AWS.

Judging by the AWS Heroes Summit agenda alone, there's no cause for concern. Roughly half of all sessions were about serverless, while the rest covered several topics: GenAI, security, databases, and containers (in no particular order).

Meetings and conversations with service teams show that AWS is developing normally, in many directions. Of course, GenAI, due to its nature, will touch many areas. Perhaps teams are encouraged to consider integrating such solutions into their services, but it's certainly not the case that AWS is investing only in artificial intelligence. The vast majority of new features and issues discussed in the sessions I attended had nothing to do with GenAI.

What's coming next?

Serverless is doing well, as I mentioned, rockstars are working on the services and will deliver many interesting and necessary new features in Step Functions, EventBridge, Lambda, and probably other serverless services, before and at re:Invent, and in the coming year.

The entire event and all content are subject to NDA. Therefore, I can't reveal anything specific.

I really liked several upcoming updates and can't wait for them to appear so I can use them myself. And I'm talking about fairly ordinary engineering updates to the aforementioned services. No GenAI magic here ✨😃
Some new features can definitely be classified in the "how long can we wait for this?" category 😉
The remaining things are less exciting for me, solely because I don't have clients/projects where I could apply them 🙁 Which by no means diminishes their importance. They're created to facilitate building or increase the security of solutions.

To be fair and not just paint a rosy picture, there was one thing in the [secret] service that I asked about and received an honest answer that they're aware of the need, but won't do it until 2025 because [secret]. That's a shame.

The level of openness of AWS employees towards Heroes elevated this Summit to levels previously unimaginable to me. Access to information covered by NDAs allowed for free communication both ways. This kind of relationship with service teams is the greatest value of being an AWS Hero.

Now I just need to find interesting projects where I can use this newly acquired knowledge 😉

The Summit is not just lectures

The AWS Heroes Summit was attended by 178 Heroes from 46 countries. As of July 2024, there are 260 Heroes worldwide in total across 56 countries. Never before in the ten-year history of this program have so many of us been in one place.

I'm the one in the Lambda / Half-life t-shirt 😃

In the first row in the middle stands Jeff Barr. He opened this mini-conference with an interesting keynote on the history and future of AWS.

Speaking of celebrities, the big surprise on the second day was a visit from Matt Garman, the new CEO of AWS. Interestingly, he didn't come to his keynote to dazzle us with a marketing presentation 🙂, but greeted us, said a few words, and then announced he would start answering questions from the audience.

Respect!

Heroes quickly grabbed the microphones, and Matt answered all questions, including the uncomfortable ones. As a person with a huge technical background, it was easy for him to win over our tech crowd.

Celebrating together

The Summit is a great opportunity for networking, as it doesn't have the crazy pace of re:Invent and is much more intimate. Every day we spent time together from breakfast until evening. I had the opportunity to deepen relationships with people I already knew and make new friends.

On the second day, during the afternoon networking session, we celebrated a special occasion – the tenth birthday of the AWS Hero program and the 10th anniversary of AWS Lambda. These beautiful cakes were prepared for us to mark the occasion. 🍰

The Polish 🇵🇱 representation at the Summit was very strong. Looking from the top left: Daniel Aniszkiewicz, Paweł Zubkiewicz, Łukasz Dorosz, Tomek Ptak, Karolina Boboli, Jarek Zieliński, Rafał Mituła, Tomek Dudek, Dominik Grzywaczewski, and Maciek Lelusz.

After such a wonderful week, I can only thank the organizers, wish for another meeting of everyone in Las Vegas, and say Go build!

Year in review, 2023

Pawel Zubkiewicz — Fri, 29 Dec 2023 10:05:23 +0000

Following the tradition I have maintained since last year, the time has come to summarize the past twelve months. Although this article is publicly available, I wrote it primarily for myself, treating it as a personal retrospection and creating a lasting record of events, challenges, and moments of joy that shaped the last year.

This text is not meant to be a self-congratulatory enumeration of achievements or a vain celebration of success, but rather a sincere look back, like a page from my diary, preserving memories and experiences from 2023.

As before, I have divided the summary into:

Career
Failures
Appearances and Publications
Travels
Concerts
Books
Other

Career

Here, there are practically no changes; I still collaborate with TT PSC as an AWS Architect, and as an independent consultant continue to develop services in the area of AWS cost optimization.

Successes

I can count as my biggest successes:

Creating from scratch and conducting FinOps Training for companies.
Conducting several successful AWS Cost Audits (savings up to 44%) - I can't boast about most of them due to contract clauses, but I have one client who publicly described our cooperation - for which I am very grateful 😃

Conducting cost optimization after a Cost Audit that brought annual savings of over a million dollars, specifically 1,319,110 USD.
Releasing as many as 15 new versions of the CloudPouch application. Expanding Cost insights functionality to include more AWS services: EC2, Load Balancers, DocumentDB, Glue Crawlers, and Jobs.
Co-organizing the AWS User Group Wrocław

Failures

Actually, I could almost copy and paste those from last year.

Poor adoption of CloudPouch
Small number of clients for AWS Cost Audit service

It would seem that in current times, savings are a priority, and anything that allows to significantly reduce unnecessary IT costs should sell like hot cakes, but it's not the case.

Faithful readers of the Serverless Poland Newsletter and blog have surely noticed that I publish content much less frequently. Lack of time is one thing, but also writing the newsletter and articles regularly brought me less joy, so I did it less often.
Not utilizing purchased training and access to closed communities.
I didn't get around to creating a page advertising the FinOps Training - there's no information about it on the internet.
Lack of significant engagement in Serverless Wrocław Meetup (we only organized 3 meetings)

Appearances and Publications

The year in a nutshell:

Automatic testing of serverless applications - presentation at Serverless Wrocław (March 9, 2023)
FinOps 4 Developers - presentation at the 4Developers conference in Warsaw (April 18, 2023)
Multicloud: is it worth floating in many clouds? - podcast with Maciej Lelusz (April 19, 2023)
The musty world of on-prem in AWS - with Maciej Lelusz - presentation at AWS User Group Wrocław (May 8, 2023)
How much code in the code - with Przemek Malak - presentation at AWS Community Day Warsaw 2023 (June 1, 2023)
AWS re:Invent re:cap - panel discussion at AWS User Group Wrocław (December 11, 2023)
Mini review of the ultra-wide Samsung LS49A950UIUXEN monitor
How to prepare for a trip to re:Invent in Las Vegas? ✈️
A Million-Dollar AWS Savings: A Case Study on EBS to Snapshot Conversion

Also, let's not forget about (almost) regularly sending, the largest in Poland, newsletter dedicated to AWS and serverless.

Travels

Travels are not just memories, but also a window to the world that becomes more valuable with each passing year. Together with my wife, every few years we immortalize our adventures in photo books, which I highly recommend to every travel enthusiast, as photos are perceived differently in physical form.

However, it's also nice to do an annual online summary, which can inspire others to pack their backpack and set off on a journey.

In 2023, the travel history is as follows:

We started the year with a bang, heading to colorful Mexico and Belize in January. A great trip, beautiful landscapes, friendly people, and fantastic food 😃

In the spring (April), we went to the Balkans. Traveling by car through Montenegro, Bosnia and Herzegovina, and Croatia was super interesting and allowed me to get to know this part of Europe better. And of course, Dubrovnik as the location of the Game of Thrones series was doubly interesting.

Mountain bike camp in Bielsko-Biała, June.

In July, we celebrated my birthday in Italy. The trip, whose climax was motorboating on the Mediterranean Sea, was wonderful. On land, we discovered the culinary treasures of Puglia and spent whole days wandering through the narrow streets of charming seaside towns.

The dream trip of my wife, who has wanted to see the world of Anne of Green Gables live since childhood. We spent two weeks in Canada, visiting Montreal, Toronto, Niagara Falls, and Prince Edward Island.

In the autumn, another dream of my wife led us to Africa for a safari. Masai Mara National Park in Kenya and Serengeti and Ngorongoro (extinct volcano) in Tanzania were filled with amazing views and encounters with wild animals.

Traditionally, my year of travel ends with a trip to Las Vegas for re:Invent. The best cloud event in the world!

To sum up, 2023 was a year of discovering as many as eight new countries, bringing me closer to the magic number of 50 visited countries.

Concerts

Last year I only mentioned concerts. Reflecting on them made me realize how much I enjoy them, so I decided to include a list of concerts in a separate section. Each of them brought me a lot of joy and I wouldn't want these precious memories to fade over time.

Elton John - May - Berlin
Pantera - June - Łódź
Metallica and Pantera - August - Montreal
Christmas Jazz - December - Wrocław
Bryan Adams - December - Gliwice

Books

I'm still not a bookworm and I prefer doing over reading. However, I managed to get my hands on a few things and even read and listened to a few. Although, to be honest, I buy more books than I read - which is obviously a problem, as the pile of shame 💩 is growing.

The Almanack of Naval Ravikant is not only a collection of business advice but also deep reflections on the meaning of life. In the book, you'll find an abundance of wisdom and intriguing observations from Naval Ravikant - an American entrepreneur, engineer, and investor. This book not only inspires but also provokes one's own thoughts. The history of its creation, which I discovered on Visualize Value, is also interesting. Jack Butcher presents its author as an example of a permissionless apprentice. The book was compiled by Eric Jorgenson, compiling content from tweets, podcasts, and interviews with Ravikant. The purpose of this publication is to make Ravikant's philosophy and wisdom available to a wider audience, touching on topics such as happiness, wealth, and personal development.
Decoded: The Science Behind Why We Buy by Phil P. Barden, released in 2022, is a fascinating exploration of the mechanisms governing our purchasing decisions. The book, purchased for the optimization of the CloudPouch sales page, turned out to be an invaluable source of knowledge in a much broader sense. Barden takes us on a journey through the nooks and crannies of the human mind to discover what drives us to buy. If I were Rafał Mazur, I would probably keep this title to myself, as it's material that, when used appropriately, can give an incredible advantage 😉
Since I've already mentioned this man, Become a Predator - Rafał Mazur, Marcin Osman.
After two, maybe three years, I finally finished the audiobook of Atlas Shrugged. While Ayn Rand's message is close to me, oh my God, the length is exaggerated!
Before the trip to Africa, I chose to read Factfulness by Hans Rosling, as it offers a novel perspective on reality, breaking negative stereotypes and showing how progress changes the world for the better. It's good to read about positive changes, especially when we are bombarded with negative news daily, such as those about wars.
Be Obsessed or Be Average by Grant Cardone - I read it for the second time. After a few years, equipped with new knowledge, I perceive it much better! The first time, Grant seemed like a charlatan. I don't see him that way anymore. Now I see many mechanisms and principles between the lines 🙂
Blood of Elves - Andrzej Sapkowski - to remind myself and compare with the next season of the Witcher series on Netflix.

And two classic titles on fascinating behavioral economics, which intrigued me a lot this year:

Predictably Irrational: The Hidden Forces That Shape Our Decisions - Dan Ariely
Thinking, Fast and Slow - Daniel Kahneman

Other

Mastermind Group - more in last year's post - it's still active, we manage to meet almost regularly, and what can I say, I really grew fond of these guys!
In 2022 in this section, I presented a few things I bought. This year, however, I will focus exclusively on recommending an ultra-wide screen monitor. This equipment significantly changes the way of working, elevating it to a higher level. You can find a detailed review of my monitor at this link.
It's my second year of learning Italian 🇮🇹. It brings me great joy, as I treat it as a hobby, but I am still at the stage of realizing how little I know ;-)

Since March, I have started amateur mountain biking training under the guidance of an experienced trainer. The goal is to improve my skills in navigating mountain trails. Regular exercises and refining my riding technique bring me immense satisfaction. At the same time, overcoming fear, especially before jumps, is a completely new challenge for me. I believe that these aspects together toughen the spirit and strengthen the willpower. An additional benefit is the fantastic group of people I train with. If you are from Wrocław and are keen on enduro, I warmly invite you to join our team!

It was a beautiful, successful year.

Thank you for your attention, and I hope you took something away from this post. Let me know if it inspired you to do anything.

I wish you a wonderful 2024!
Paweł Zubkiewicz

How to prepare for re:Invent in Las Vegas?

Pawel Zubkiewicz — Wed, 18 Oct 2023 08:43:48 +0000

[Last update: Nov 2025]

First of all, congratulations on attending re:Invent! It's an unforgettable experience, and you're sure to be amazed. I'm still excited at the thought of going, even though this will be my third time. I can't wait to be in Las Vegas again!
re:Invent is a fantastic four-day conference that dazzles with its scale and can be overwhelming at times. That's why it's worth being well-prepared.

I've gathered several tips for you. Some of them are obvious and widely known for years, but even if you've heard about them, read all the way through, as some are unique and based on my own experiences. 🙂

00. Travel insurance

This is obvious, but worth mentioning. You need to insure yourself against accidents, as healthcare in the US is paid for and very expensive.

01. Footwear

Everyone has been talking about this for a long time, and it's the number one advice to follow. Twenty thousand steps a day at re:Invent is not a big deal. Every day, for four days. That's why take comfortable shoes for walking. This time, I'm taking two pairs. I recommend lightweight, breathable running shoes for walking on pavement. It is also worth taking plasters for grazes with you.

02. Dry Air Essentials (Skincare)

Las Vegas is in the desert.

It's a known fact, but most of us don't deal with such a climate on a daily basis. Especially in the winter. So, it's worth stocking up on supplies to help us function in such conditions.

Creams (including for the face)
Lip balm
Eye drops / artificial tears

for me, this is a basic kit. These things can also come in handy on long flights, so it's worth packing them in your carry-on bag.

I've also heard that people bring air humidifiers with them or request a hotel room equipped with one to counteract dry air.

03. Throat Soothers (+medications)

At re:Invent, you'll meet thousands of people, and you'll probably be talking to hundreds. Excessive talking can cause a sore throat for me. To tackle it, it's a good idea to bring something for your throat. Strepsils candies work for me.
Additionally, when traveling, it's always a good idea to remember to bring tested medicines, for example, for colds, that we trust. This way, in case of sickness, there's no need to search for a CVS and figure out which American medications are equivalent to those you would buy in your own country.

04. Attire for re:Play

The Thursday party takes place outdoors and in huge tents. Last year, it was quite windy and rather cold. It's a good idea to have a jacket and warm shoes with you.
That's all that comes to mind regarding the weather and climate in Las Vegas.

05. US Power Outlets / Adapter / Extension Cord

This is rather obvious, but it's worth a reminder. The USA has different electrical outlets, so you'll need an adapter. Usually, we only use chargers for phones and laptops, so there's no need for a converter.

Additionally, whenever I have space in my luggage, I bring a regular extension cord. Firstly, in combination with one adapter, it gives me Polish sockets (I'm from Poland). Secondly, if there's no socket near the nightstand in the room, thanks to the extension cord, I'll always have my phone charging by the bed.

There's another life-hack regarding sockets. If you use Apple products, you can buy a different end for your charger. I did this and bought adapters for American outlets (on the left in the picture).

You can also buy a universal charger for phones and laptops with adapters, e.g., Baseus GaN5 65W – I've had one for a few years now, and it's awesome! It has two charging ports and allows you to quickly charge your phone (if yours is compatible with quick charging) and can easily replace the power adapter for a MacBook Pro with an M processor. And it has three plugs/adapters: for Europe/Poland, the US, and the UK and Asia (I recently used it in Hong Kong, for example). Overall, I recommend this device 🙂

06. USB-A Cables

Sometimes, the sockets by the bed in hotels have USB outputs. They're always USB-A. The same goes for multimedia panels on the plane. So even if you've been using newer USB-C for a long time, it's worth having a USB-A to USB-C (or Lightning if you use an iPhone) cable to charge your phone.

07. Power Bank

I take a lot of slide photos during presentations, so my phone quickly runs out of battery. That's why I bring my trusted power bank with me. There are charging stations at the conference, and companies will surely be giving away power banks as goodies, but in my opinion, it's better to have your own.

08. Mobile Internet in the USA

Wi-Fi is available everywhere at re:Invent. There can be minor issues with it, but personally, I think you can survive without mobile internet on a GSM network.

But why bother, right? 😉

I've been using eSIM cards for many international trips. They only work with iPhones and Samsungs (the more expensive ones). It's a pure software solution, so you don't need to buy a physical SIM card, e.g., at the airport. And you don't need a phone that supports dual-sim.

I highly recommend the Airalo app/service. I've used it in the USA, Canada, Montenegro, Mexico, and Belize. By creating an account through this link, you and I will each receive $3 to use for purchasing eSIM cards.

A few days before departure, I buy an eSIM and set it up on my phone while still at home. After landing in the destination country, the eSIM connects to its network and only becomes active from that moment on. Sometimes, the first network login takes up to 20 minutes, but I usually have internet even before I leave the plane.

09. Leave Empty Space in Your Suitcase

On the last re:Invent, I flew with a half-empty suitcase. I collected so much swag that I could barely close it. If you like cheap t-shirts, socks, and stickers, leave some free space in your luggage to bring back home all those things. 🎁

10. Online Hotel Check-In

If possible, do an online check-in at your hotel. It significantly reduces your time at the reception.

11. Unnecessary Items

Not everything you need to take with you, some things you will receive on-site.

Water bottle. Each participant receives one in the standard gadget pack, so I think it doesn't make much sense to bring one.
AWS has also been giving hoodies to everyone for years, so you can take one less from home.

What to Do in Las Vegas After Hours?

re:Invent is a cloud celebration, but it's also worth coming down to earth and exploring a bit. Below are a few suggestions from me.

12. Grand Canyon and Hoover Dam

When visiting Las Vegas, it's worth taking the time to visit the Grand Canyon and Hoover Dam, which are unforgettable attractions near the city.

The Grand Canyon is one of the most impressive natural wonders in the world, captivating with its immense depth and majestic rock formations. It's a place that provides unforgettable experiences associated with nature, tranquility, and peace, serving as a getaway from the hustle and bustle of Las Vegas.

The Hoover Dam, located on the Colorado River, is also an impressive structure, not only an important part of the infrastructure but also a tourist attraction. A walk across the bridge towering over the dam allows you to admire beautiful views of the Colorado River and the surrounding landscapes. Furthermore, the dam is a place steeped in history and engineering, making a visit there an educational experience.

The most impressive experience is admiring the Grand Canyon and Hoover Dam from a helicopter. It's a unique and unforgettable experience that adds even more majesty to these already impressive places. A helicopter ride over the Grand Canyon allows you to admire stunning views from an aerial perspective, fully showcasing the monumentality of the canyon and its diverse rock formations. The view of the surrounding landscapes and the impressive work of nature is breathtaking. Similarly, a helicopter ride over the Hoover Dam offers an extraordinary opportunity to see this monumental structure.

I went on the tour in 2019. I bought the ticket at one of the kiosks on The Strip. Its price was rather low for such a magnificent trip. I highly recommend it, as it is one of the best places in the world for helicopter tours.

13. The Strip

A walk along "The Strip," the main street in Las Vegas, is an unforgettable experience that allows you to feel the pulse of the sin city, gambling, and entertainment. Majestic hotels and casinos showcase their impressive facades, inviting you to explore interiors filled with luxury and opulence. Strolling along this iconic street, you can admire fascinating light shows and fountains that create spectacular displays, especially after sunset.

14. The Sphere

This is the newest attraction in Las Vegas. The world's largest dome that also serves as a huge display from the inside and outside. It is simply an entertainment hall that, thanks to its multimedia structure, can provide unforgettable experiences.

15. Old Las Vegas (Fremont Street Experience)

It's not the most fantastic place in the world, but it's worth visiting to form your opinion. It was a bit tacky for me, but I don't regret going there.

16. Accompanying Events

During re:Invent, many accompanying events sponsored by vendors take place. It's usually a good idea for a free drink and snack. Sometimes it's a good opportunity to personally meet someone we know from the internet. That's how I met the founder of A Cloud Guru in 2019 - a cool thing because I started learning about the cloud from his courses a long time ago.

You can find the list of events at https://conferenceparties.com/reinvent2025/.

Summary

Those are all the good tips I have for you. I hope they prove helpful and assist you in having the best week in Las Vegas! 😃

If you think something is missing from this list, please add a comment so that others can benefit from your knowledge and experiences.

Mini-review of the ultra-wide Samsung LS49A950UIUXEN monitor

Pawel Zubkiewicz — Fri, 14 Jul 2023 09:26:35 +0000

Working daily with the cloud, we are usually separated from hardware by many layers of abstraction and dozens of automations. Just look at the EC2 instance family.

You might even develop a dismissive attitude towards hardware. Assuming that we meet the requirements for computing power and memory, every machine will work well. The rest is tuning and cost optimization. We practically never consider choosing from a full range of instance types consisting of hundreds of items. As a result, hardware loses its significance (commodification).

However, there is another category of hardware that we should pay much more attention to. I am talking about devices we use personally, not remotely, because they affect our health and well-being.

While working in an office, our choice is limited to what the employer provides. But in remote work, we are only limited by budget (or wife 😉).

Today I will present my journey to decide on purchasing a new monitor and describe what it's like to work on it after six weeks.

Are you perhaps thinking about changing your monitor but have been putting it off? In that case, I hope this article will encourage you to consider changes at your own desk.

I unnecessarily lost a few months to deliberations and pondering the topic. If someone had advised me at the beginning, offered knowledge that I am about to pass on to you, I would have saved time and enjoyed a better workplace much earlier.

Choice - the path leading to the goal

The impact of a display on the eyes is obvious. This was the main driver for me to change.

I had two very old Full HD LG monitors, which did not have any solutions to protect my eyesight. I had them connected via a docking station and KVM switch to a PC and a work MacBook.

This is how they looked a few years ago. All in all, they were good to work with. Unfortunately, the autumn-winter period, when dusk falls right after lunch, strongly affected the fatigue of my eyes.

Around November or December, I started looking for new monitors.

Since from my first job in 2006 I have always used two monitors, I still wanted to stay with this setup. The space I have allows for the installation of two twenty-seven-inch (27") monitors. That's what I started looking for.

While it is currently difficult to find hardware without eye protection features (no flickering + blue light filtering), I had other requirements that posed a challenge. I wanted to get rid of the tangle of cables under the desk along with additional devices (docking station and KVM switch).

My requirement became Daisy Chaining, also known as MST, allowing two monitors to be serially connected together, so that only one cable needs to be run from the computer to the first monitor.

Yes, it definitely solved my first world problem: connecting a MacBook Pro with two UBS-C/Thunderbolt cables - was to become a thing of the past 😉

Unfortunately, it soon turned out as follows (this is a story for at least two 🍻 beers, so I'll shorten it a lot):

Apple does not support MST over USB-C
Apple may support MST over Thunderbolt 3/4, but it is safest to have M2 chip hardware
Monitors with Thunderbolt are at least 2 times more expensive than those with USB-C

In general, dozens of hours spent looking for equipment resulted in finding a model that might work with a MacBook, supported Daisy Chaining, and also had a built-in KVM switch.

The price was only ~1000 USD. Per piece. (BenQ PD2725U)

Unfortunately, someone figured out that when a monitor has Thunderbolt ports, it falls under the category of applications "computer graphics" and such equipment should be priced accordingly.

As you might guess, I don't care about the colors in VS Code and Chrome 😉 so I wasn't going to pay for parameters I didn't need.

At that point, I returned to the idea of a single Ultra-wide monitor, which physically replaces two 27-inch monitors. I write "returned" because such a thought crossed my mind at the very beginning, but I quickly dismissed it. At that time, I believed that 49-inch monitors were a segment for gamers (crazy refresh rates, low input lag, etc.) - and again, I would be paying for something I wouldn't use.

And here's another digression. It's us, programmers, who push this entire IT business forward, and yet there isn't even a segment of monitors for coding and the internet for us. And nobody will convince me that "office" is our segment because "office" is simply ordinary equipment without anything special for pro users like us.

The obvious advantage of a single monitor is the lack of the need for MST and Thunderbolt support. This significantly lowers the price and also positively impacts the minimalism of the final effect on the desk. That's why...

I started looking for a 49-inch Ultra-wide monitor.

The choice isn't vast (at least in Poland), and from several models available on the market from different manufacturers, I quickly decided on Samsung.

Wide, but not for gamers! Exactly what I needed 🎯

The exact model is LS49A950UIUXEN.

Source: Manufacturer's materials, Samsung.

I bought it for less than 1300 USD. That's half the price of two BenQ 27-inch monitors. 👍
Unfortunately, it quickly turned out that I needed to buy a desk mount, as the Samsung stand takes up a ton of space. I chose the Ergotron HX Desk Monitor Arm for only 380 USD 😬.

The monitor is 5K, meaning its resolution is 5120 by 1440 pixels, and the screen aspect ratio is 32:9.

Curved R1800 screen

Source: Manufacturer's materials, Samsung.

Since my old monitors were also set at an angle, I essentially wanted a curved monitor. However, there are two types of them: R1800 and R1000. 'R' as in the radius of the circle of which the screen is a part. 1800 gives a lesser curvature, while 1000 gives a greater one.

First impression

Very positive.

Although I must admit, on such a wide monitor this curvature could be greater, so R = 1000 would also be suitable. This type dominates in monitors intended for gaming due to a greater immersion in the game world. Unfortunately, I didn't find any 49" monitor on display anywhere in my home town to sit in front of, so I bought it blindly.

The transition from twice Full HD (1920 x 1080) to 5120 x 1440 pixels also did the job. The difference in the vertical is very noticeable for the better, although the physical dimension is the same as in the old equipment.

The monitor, even with a new desk mount, protrudes much more from the wall than my old monitors, thereby taking up space above the desk and shortening the distance to the screen. In the center it is 22 cm, and on the sides 32 cm from the wall.

The last note is about the rear panel of the monitor and connecting cables - a real torment, they made it very inconvenient.

Current setup

My Mac Book Pro is connected to the monitor with a single USB-C cable (and it is charged from it). The old PC, on the other hand, is connected via two HDMI cables (in Picture-by-Picture monitor mode - for the PC it's like two physical 27" monitors). I got rid of the docking station and KVM switch under the desk (along with dozens of cables). It's now simpler and more minimalistic.

Unfortunately, the KVM switch built into the monitor only half works. That is, it only switches the image, but not everything that is plugged into the monitor via USB. In practice, this means that my Logitech mouse and keyboard must be switched separately between computers (each device has such a switch - the Easy Switch technology in Logitech Unifying Receiver).

Perhaps I am too stupid to figure it out, and honestly, I didn't want to invest more time in it because I use the PC sporadically, and these three clicks don't annoy me (monitor, keyboard, and mouse). In summary, I have a lot of devices plugged into the monitor, and everything communicates with the laptop via a single USB-C cable along with the image - and that's a great thing.

Impressions after 6 weeks of use

Not much to say - it's awesome and that's why I'm writing about it.

Perhaps you are thinking about changing your monitor, but you have put it off until later? Or maybe you didn't even know that there are such ultra-wide ones? In that case, I hope this article will encourage you to consider upgrading your workspace. If someone had advised me earlier, I could have enjoyed better equipment for longer. I, personally, recommend it.

For me, the biggest surprise is how big a difference the lack of bezels (frames) in the middle of the screen makes compared to a setup of two monitors.

For years I sat slightly crooked because my left monitor was the main one. Now I sit totally perpendicular and look straight ahead (area no. 2 in the picture).

The lack of a bezel in the middle enables better work. For years, I worked in two work areas (right and left monitor). Now I work on three.

I have divided the screen into 3 areas, where the middle one is centrally located and is about 20% wider than the side ones. It works incredibly well! The side areas are auxiliary, for example, email and company messenger. To define these working areas and manage windows, I use an application on MacOS called Rectangle Pro.

The comfort of work and the health benefits of sitting straight and looking straight are incomparably better than with two monitors. I know, I am very excited about it, but I still feel that I do not accurately convey how big a difference the lack of bezels makes!

I must also add that before the change, I was very afraid of not having two monitors, because of how the fullscreen mode works on Mac. For those who don't know, I will just say that it creates a virtual desktop for each maximized window. This was a feature I used heavily with two monitors. On one, it only makes sense with some applications (for example, Miro looks awesome 😀👍, video editing or VS Code), but in most cases it is hopeless, for example, maximized Chrome, or YouTube leaves a lot of empty, unused space on the sides. Fortunately, the aforementioned screen areas work so well that I don't miss the Mac's virtual desktops at all!

Summary

I am very pleased with the change, even though it is not perfect (lack of one-button computer switching). The quality of work has definitely improved, and importantly, I have taken care of my body (eyes and spine).

Ultra-wide 49" monitors are still relatively rare equipment, so I hope this article will be helpful for you in making or not the purchase decision.

Year in review, 2022

Pawel Zubkiewicz — Sat, 31 Dec 2022 10:51:10 +0000

Year 2022 was full of many events that I decided to gather in one place.

I think this text will provide you with a lot of value, whether professionally or even travel-wise (inspiration for trips). After reading it, you will certainly get to know me better.

However, I must admit that I am also writing this summary for myself, to possibly come back here in the future and reminisce.

I divided the summary into:

Career
Failures
Presentations and publications
Travels
Books
Other

Career

While working for TT PSC, after more than 3 years, I ended my project cooperation with the US client PTC and moved on to another project in the PMI organization as Master Solutions Architect. Such a bit of a promotion, and a bit of a change of environment, new challenges and other problems. Standard.

It was much more interesting on the private front

In June, I responded to the Call for Papers for members of the AWS Community Builders program and sent my presentation proposals for the re:Invent 2022 conference. In mid-September, I received an extremely nice email:

Congratulations on becoming a re:Invent speaker!

The joy was endless 😀 It is an incredible honor because I was the first non-formally affiliated with AWS speaker from Poland at the re:Invent conference! It also meant that I would fly to Las Vegas, which was not so obvious before. At that time, I was only at re:Invent 2019 once, but I really wanted to be there again in 2022. Being chosen as a speaker meant a ticket to the conference, literally and figuratively 😉. Fortunately for me, TT PSC agreed to sponsor the trip. My goal and dream was now within reach.

All I had to do was prepare the presentation. The deadline for submitting finished materials was set for October 17th. I remember well the last two weeks of sitting on the speech every day. Luckily, hard work paid off, and the presentation was completed on time.

Dreams come true!

Will you be our Hero?

I couldn't believe it when I saw this titled email in my inbox. Although I had been dreaming for a long time to be among the prestigious group of AWS Heroes, on that November evening I was completely surprised because I didn't think it would happen this year.

On November 15, 2022, the information was officially announced, and I became the first Polish AWS Serverless Hero. At the end of 2022, there were 253 AWS Heroes in total, and in the serverless category only 42. This is a really small group and an incredible honor to be one of them.

re:Invent

So, as a brand-new hero, I went to Las Vegas and gave my presentation COM314 Automated testing for serverless applications. I rehearsed the speech itself a dozen times while still in Poland and then in the hotel room to deliver it as well as I could. The work paid off.

I remember that during the presentation, the connection to the display went down three times and ended up with me switching to a different laptop. However, these failures did not deter either me or the audience. 97 people stayed until the very end, and then a large group of listeners besieged me with additional questions. Unfortunately, my speech was not recorded.

Professionally, I ended the year in December by bringing AWS User Group Wrocław back to life, along with a phenomenal group of friends. At our first meeting, we discussed the conference and the news announced at re:Invent.

Failures

Successes do not necessarily mean the absence of failures. They are naturally incorporated into everything we do, so there is no point in hiding them.

So, what sucked in 2022?

broken ribs on a bike (if at least there was an impressive jump, unfortunately there wasn't 🤦🏻)
poor adoption of CloudPouch
lack of customers for AWS cloud cost consulting

It might seem that in these times savings are a priority and everything that allows for significant reduction of unnecessary IT costs should sell like hotcakes (not to mention ecology). Nothing could be further from the truth. We devoted the last half year to the Polish market, and the results are far from satisfactory.

Presentations and publications

Year in a Nutshell

Article: AWS Step Functions - Refactor to native integrations with AWS API
Presentation: conference 4Developers - April 5, 2022 - Cost optimization in the AWS cloud (in Polish)
Article: 4 ways of executing Lambda function via HTTP endpoint - a comparison
Article: Simple Serverless Scheduler
Presentation: conference Cloud Summit 2022 - May 19, 2022 - Cost optimization in the AWS cloud (in Polish)
Presentation: conference AWS Community Day Warsaw 2022 - October 14, 2022 - Industrialization and other mechanisms of public cloud evolution (in Polish)
Article: Reduce AWS costs: How I've saved $18,890 monthly on EBS drives?
Article: Add types to your JavaScript Lambda functions
Article: Serverless pre:Invent announcements (in Polish)
Presentation: conference AWS re:Invent - November 30, 2022 - COM314 Automated testing for serverless applications
Resurrection and co-organization of the AWS User Group Wrocław
Presentation: Lighting talk - December 12, 2022 - re:Invent from a participant's perspective

It is also worth mentioning the (nearly) regular weekly dispatch of the largest AWS and serverless newsletter in Poland.

Travels

The first year after the pandemic, in which covid restrictions disappeared completely, was very generous to me in terms of travel.

It started with Portugal, specifically the beautiful, green, and mountainous Madeira in February.

Then my wife and I did a road trip from home through the Czech Republic, Slovakia, Hungary, and Austria.

In May, we visited Italy. Our base was Pisa, from which we went to see Siena, Florence, and the sea to visit the charming villages called Cinque Terre.

The next trip was in June, after a few days in Istanbul we began a journey through Turkey. We visited the east of the country, Cappadocia (that's where you fly hot air balloons 😉).

At the end of August and beginning of September, we made a quick trip to the Baltic countries (Latvia and Estonia) and Finland on one trip.

In October, there was a city break in Barcelona - a birthday present for my wife.

We ended the year of travels with a winter trip to Cyprus. When Poland turned gray in November, we were swimming in the sea under the palm trees. 🏝

All of the above trips were organized by my wife ❤️, who is a master of travel planning. Of course, except for her present (Barcelona), which I planned. I also went to a conference in Las Vegas in November, which I mentioned earlier.

The year 2022 was full of travel, but only a few countries were new to me: Latvia, Estonia, Finland, and Cyprus. However, I managed to reach the level of 40 visited countries in the world.

Books

Lately, I am not a fan of reading. Of course, I'm only talking about books related to technology, business, marketing, or sales. But I rarely read for entertainment either. Reading is a veiled form of procrastination. To make matters worse, it reinforces the pattern of "I have to know and predict everything before I start", which is impossible and effectively immobilizes you from the very start.

That's why I replace books with video trainings and mental masturbation with recognition in battle. Or at least I try to act that way 😉

Taking all this into account, the following completely short list of positions should not surprise you:

Product-Led Growth: How to Build a Product That Sells Itself - a compendium of knowledge on product building. Don't let the name fool you, this book is equally about discovering the needs of your customers and fulfilling them as it is about marketing and sales itself. Reading it with CloudPouch in mind, I have already made several decisions about changes in my application that have not yet been implemented (lack of time), but will certainly contribute to greater customer satisfaction and wider use of this tool.
Traction - a highly recommended, popular book about the fact that in marketing everything depends on, and you need to try many ways to verify what will work for us. It doesn't blow your mind, but it is very concise.
The 48 Laws of Power - a definitely interesting position with patterns identified throughout history for those wanting to gain and maintain power. But even if you don't have such ambitions, you can treat this list as a cheet sheet and have fun analyzing the behavior of characters from various series such as Game of Thrones, Succession, or others and identifying which patterns they are using. 😉

Other

Mastermind group - another factor contributing to the abandonment of reading is a group of three people I regularly meet with and discuss all topics for almost two years now. Originally, we started with a plan to study Wardley's maps, but quite quickly we have moved beyond that and cover all topics that interest us. Of course, we didn't part with the maps, but rather treat them as a tool for a better understanding of the world around us and the changes taking place in it, rather than the goal itself. In other words, we get excited about driving a car, not the machine itself. These sessions give me more than any book could offer! They show different points of view, stimulate reflection, and challenge me with topics outside my bubble. One of the really interesting topics I learned about through this group is the theory of psychopaths in companies. 🤯 I'm sure you haven't heard of it, even though the concept is over 10 years old. This is not the time or place for me to develop it here, so I'll leave you with just the link. It fits in perfectly with the 48 Laws of Power 🙂
Purchase of Sennheiser Momentum 3 headphones - all my life I used some crappy headphones, these are still not audiophile grade, but for me the difference is huge. The money spent is definitely worth it for good sound every day at home and when walking the dog. I'm writing about this because if someone had told me beforehand: Dude, get yourself some good headphones I would have been able to enjoy this a long time ago... so if you haven't already, get yourself some good headphones! 😃
While we're on the subject of music, I enjoyed myself at concerts: Tool, Iron Maiden, Ed Sheeran, Machine Head, Martin Garrix.
I also purchased a Scott Spark bike after waiting for over a year. It's a blast 💪😃
Since January, I've been learning Italian 🇮🇹. It brings me a lot of joy because I treat it as a hobby. It's also a nice reminder of how difficult the basics can be for someone who is just starting - it gives me food for thought in the context of my trainings and advocacy of serverless architecture.

That, in a nutshell, is what my 2022 looked like on paper and through the lens.

Add types to your JavaScript Lambda functions

Pawel Zubkiewicz — Sat, 29 Oct 2022 15:48:01 +0000

If you need to add types to your Lambda functions written in JavaScript (Node.js) then you can do it very easily, using VS Code's built-in JSDoc support. How it works in other editors I don't know, because I haven't had the need to leave VS Code for many years 😃

Types library includes events and other objects that occur in AWS. They come in handy when defining an event that calls Lambda functions, which is the first parameter of the handler method:



const handler = async (event) => {
  // extract data from event
  const response = '...'
  return response
}

By installing this library as a dev dependency



npm i --save-dev @types/aws-lambda

we can add this information to our code:



/**
 *
 * @param {import('aws-lambda').APIGatewayProxyEventV2} event
 * @returns
 */
const handler = async (event) => {
  // extract data from event
  const response = '...'
  return response
}

Such a small change allows VS Code to display a relevant list of prompts as we write code.

This is what it looks like for the API Gateway HTTP API:

My next Lambda function is triggered by DynamoDB Stream, so we define event as DynamoDBStreamEvent. Please notice that even inside the loop editor prompts valid types in sub-properties:

Of course, JSDoc definition can be added on any method, not just on the handler called by the Lambda service. You can annotate any JavaScript function that process AWS type with this method to have better intellisense.

There is a huge number of types in the whole package @types/aws-lambda, and not just events. We also have responses and other objects available.

As an example, consider the context object, which we sometimes need in a Lambda function.

I'm sure you will find this small convenience useful. I hope it would ease switching from strongly typed languages to JavaScript. This article, was based on this short video published by Luciano Mammino.

At this point, it's probably worth repeating myself (since I wrote about it a long time ago) and answer the following question.

Why am I writing in JavaScript and not TypeScript?

The transition from Java to JavaScript was neither easy nor pleasant for me. The lack of types didn't help. But that was years ago and now I wouldn't go back to Java for the world's sake.

Of course, there is still (for some time) place for Java in this world. 😉

However, in Lambda functions the scope of the problem and the amount of code is so small that we don't need types to understand what's going on. By complete coincidence, in recent days Yan Cui - AWS Serverless Hero - also wrote about this, and I paste his graphic and comment here:

In a Node.js app, say, an Express.js API, the complexity ceiling is really high - multiple abstraction layers, lots of modules and a "user" variable can mean different things every time you see one. Types are crucial for managing that complexity and communicating intent.
But the complexity ceiling of a Lambda function is much much lower. I only need to look at 2/3 modules to understand what the function does. To the point that the (relatively small) overhead of bringing in TypeScript just doesn't feel necessary in most cases.

Source: Yan Cui in LinkedIn

The bottom line is that it's better to break through and implement functions in JavaScript than to harness TypeScript, just to have types everywhere. Especially since TypeScript entails some inconveniences.

The lack of defining explicit types is very convenient because the code is shorter and easier to read, and sometimes you can even go a little bit crazy with it... 😉

Reduce AWS costs: How I've saved $18,890 monthly on EBS drives?

Pawel Zubkiewicz — Fri, 03 Jun 2022 11:26:58 +0000

This is a case study of a real project, where one simple change in configuration resulted in almost a quarter of a million dollars savings per year.

There was a single AWS account which had around 17,000 EBS volumes, which totaled to 944,875 GB (almost 1000 TB). That's a serious amount of disk space for which you pay serious money.

A CloudPouch tool identified possible savings on that account (screenshot above) regarding EBS Volumes. That was a starting point, which started the whole endeavor.

Before optimization, EBS:VolumeUsage.gp2 cost was around $100,000 monthly. That's just disk drive space, nothing else! All of those drives were of type GP2, which stands for General Purpose 2.

According to the CloudPouch tool, conversion to newer drive type GP3 would save 20%, which was $18,890 monthly in this case.

EBS `GP3`

AWS introduced new GP3 drives in December 2020. They provide up to 20% lower price-point per GB than existing GP2 volumes, and their performance is no longer coupled with their size! Which means you don't need to over-provision size to get expected performance.

If you're interested in detailed comparison this AWS blog post is the best place to start. The below table summarizes most important differences pretty well; it was copied from the over-mentioned blog post.

Migration from `GP2` to `GP3`

After learning more about GP3, migration sounded like a no-brainer. 😃

The newer drive type should result in immediate decrease in costs by 20% without any performance impact on our drives, which were between 50 to 250GB each.

From a technical perspective, migration seemed pretty easy to conduct. AWS states:

Customers can easily migrate gp2 volumes to gp3 volumes using Elastic Volumes, which is an existing feature of Amazon EBS. Elastic Volumes allow customers to modify the volume type, IOPS, or throughput of their existing EBS volumes without interrupting their Amazon EC2 instances.

After confirming with AWS representative, that conversion from GP2 to GP3 does not impact running EC2 instance at all, I was ready to start.

The change can be applied manually in the web console, as showed in the image below. All you need to do is to select new type:

Of course, when you have 17,000 drives, you want to automate that!

Fortunately, you can do that using AWS SDK. It's surprisingly simple, given how much money you can save 😀

You need to get the list of all your EBS volumes using EC2.describeVolumes() method, and then modify those having VolumeType === 'gp2' to gp3 using another SDK method EC2.modifyVolume(). It really doesn't get any simpler than that. 😀

The EC2.modifyVolume() request asynchronously starts the migration, so the whole script runs pretty fast. Based on my research and discussions with AWS representatives, the whole operation can take up to 6 hours per drive, depending on size and other factors.

What's really important and amazing at the same time is that migration takes place in the background without interrupting EC2 instances. Also, we pay only for GP3 drive since the very start of the migration.

Results

Well, you already know the results but lets show them in the form of a chart.

The EBS:VolumeUsage.gp2 UsageType costs (yellow) were replaced with EBS:VolumeUsage.gp3 (red).

As expected, EBS Volumes costs dropped by 20%. In the case of this particular account, I was able to save $18,890 monthly - almost a quarter of a million dollars per year!

The migration took place at the end of 2021 year, we haven't had even a single problem with any EBS or EC2 instance using them.

Ongoing migrations

In the environment, where you can't control newly deployed EC2 instances and their EBS drives, you can deploy a Lambda Function, that would regularly migrate newly created drives to GP3 version. It's really easy to write and can save a lot of money.

Disclaimer

In most of the cases migration from GP2 to GP3 can be safely executed. However, preliminary analysis is recommended, especially to check if your existing drives have any special settings, like provisioned IOPS. Don't do this without thorough preparation.

CloudPouch

Everything described in this article can be done without CloudPouch.

However, if you want to examine all your AWS accounts and have immediate response about potential savings, CloudPouch is the tool to use!

Start in the Current month and scroll down to the AWS Services table.

Select the EC2 - Other row, it will expend showing UsageType cost chart.
Next, click on Check Cost Insights 🔍 button. Now CloudPouch is connecting to your AWS account to get information about deployed resources.

After a moment, a summary is displayed.

The Cost Insights summary includes not only GP2 to GP3 migration check, but also recommendations about:

"orphaned" drives - unattached EBS Volumes that are not used by any EC2 instance,
Elastic IPs - reserved but not used,
and needless NAT Gateways - sometimes people forget to remove them from VPCs that are no longer used.

All those resources can be considered waste - you don't use them, but you pay for them.

The summary also gives you an amount of potential savings, so you can make a data-driven decision whether it's worth optimizing.

DEV Community: Pawel Zubkiewicz

AWS billing horror stories: 10 monsters living in your bill

Strangers knocking at the door

1. The haunted house: an empty bucket for $1,300

2. The mummy: a file that woke up after years

3. A swarm of bats: 207 million requests in three hours

Loops that feed themselves

4. The poltergeist: a function that calls itself

5. My own poltergeist

Good intentions, terrible bill

6. The vampire: a NAT Gateway that drinks every gigabyte

7. The ghost in the logs: one print for $1,500

8. The curse of defaults: CloudTrail for $200,000

Monsters you can't see

9. The zombie: you deleted it, but it keeps walking

10. The invisible monster: $30,000 for Claude Opus

Why the alarm didn't ring

So did AWS finally give us a brake?

Are these monsters still alive?

Monster hunting in 15 minutes

What this all means

Sources

Why AWS Cost Explorer Isn't Enough: The Story of My Own Pain and How CloudPouch Was Born

Painful beginnings with AWS

Cost Explorer: friend or foe?

The usage types puzzle: 10,000 ways to pay

The turning point: the birth of CloudPouch

When data tells a story

The cost detective

Real results: from 13% to 62% savings

The future: FinOps as the new standard

More than just savings

What's next with the Serverless Framework in 2025?

Recent Events Spark a Heated Debate

Houston, We Have a Problem!

Version 4 – Licensing Change, a New Business Model

Free Usage Terms for V4

The Community Responds: Forking Serverless Framework V3

What to Choose?

Conclusion

Bonus: What’s New in Serverless Framework V4?

Hybrid Developer Mode

Extensions and Integrations

Additional Features

Compatibility

Migrating AWS Organizations: How I Did It and Why

Why Did I Have to Migrate?

How Did the Migration Go?

Challenges I Faced

AWS Identity Center - A Fresh Start

Recommendations

Summary

Simplified Configuration of SSO Profiles in AWS CLI Using SSO Sessions

Benefits of Using SSO Sessions

Why Choose SSO Sessions Over IAM Users?

Configuring a Profile Using the aws configure sso Wizard

Logging In and Using an SSO Profile

Configuring Additional SSO Profiles

Logging Out of an SSO Session

My Experience with IAM Identity Center

Gradual Transition to SSO and Temporary IAM Key Deactivation

CloudPouch Supports the New SSO Profile Configuration

Summary

Report from AWS Heroes Summit 2024

How was the AWS Heroes Summit?

Serverless is not dead!

What's coming next?

The Summit is not just lectures

Celebrating together

After such a wonderful week, I can only thank the organizers, wish for another meeting of everyone in Las Vegas, and say Go build!

Year in review, 2023

Career

Successes

Failures

Appearances and Publications

Travels

Concerts

Books

Other

How to prepare for re:Invent in Las Vegas?

7. The ghost in the logs: one `print` for $1,500

Configuring a Profile Using the `aws configure sso` Wizard

EBS `GP3`

Migration from `GP2` to `GP3`