DEV Community: Subham Ghosh

Design a Scalable Notification System

Subham Ghosh — Sun, 01 Nov 2026 13:27:48 +0000

Designing a scalable notification system sounds straightforward until - A 10M marketing blast delaying a single login OTP by 4 minutes will kill your checkout conversion faster than a database outage.

The naive architecture of dumping every notification into a shared message broker with a priority header guarantees head-of-line blocking the second a surge hits.

Here is how I design a notification engine to enforce a strict <5s p99 OTP SLA at 16M+ daily events without dropping a single packet:

The Ingestion & Isolation Path: Upstream Service ➔ Stateless Gateway ➔ Redis Dedup Shield (atomic SETNX) ➔ HTTP 202 Accepted ➔ Physical Kafka Priority Topics
The Smart Core & Egress Path: Dedicated Worker Pool ➔ Smart Core (Quiet Hours + Redis Debounce + Hydration) ➔ Distributed Token Bucket Limiter ➔ Vendor APIs (APNs / Twilio / SendGrid)

When scaling this to production, watch out for these 3 traps:

The Priority Flag Illusion Relying on priority headers inside a shared Kafka topic fails because consumer threads spend all their cycles de-queueing bulk marketing payloads while urgent OTPs sit starved in partition lag. We solve this with physical isolation: dedicated, high-priority Kafka topics consumed by autonomous worker pools provisioned for zero queuing, keeping OTP p99 delivery latency under 3s.
The Vendor Throttling Meltdown Calling external provider APIs (Twilio, SendGrid, APNs) synchronously inside your consumer workers will exhaust thread pools the millisecond an upstream vendor experiences a latency spike or throws HTTP 429s. You must decouple egress using distributed Token Bucket rate limiters in Redis and enforce message custody in Kafka before external invocation, allowing bounded exponential backoff instead of silent drops.
The "Double-Tap" Resend Storm Querying a relational database to verify if an alert was already dispatched collapses under load when anxious users tap "Resend Code" 6 times in 5 seconds during an outage. We drop a Redis Deduplication Shield directly at the API Gateway using atomic SETNX with a sliding 60-second TTL on userId:eventType, returning an immediate cached HTTP 202 without
generating downstream broker traffic.

I broke down the complete architecture - including the sub-5s OTP failure latency budget, the Two-Stage Kafka topic pipeline, and how chronological Kafka delay topics replace database retry polling - in my full system design write-up here:
https://engineering.by-tech.workers.dev/articles/scalable-notification-system-design/

The Golden Rule for Notification Egress: If an authentication OTP doesn't arrive in under 5 seconds, the user assumes your entire platform is down. If a promotional discount notification arrives 30 minutes late, nobody notices.

Never let bulk traffic share the same connection pool, Kafka partition, or egress rate limiter as auth traffic.

SystemDesign #DistributedSystems #BackendEngineering #Kafka #Redis

Hotel Booking System like Booking.com

Subham Ghosh — Sun, 25 Oct 2026 14:29:08 +0000

When 500 travelers simultaneously click "Book" on the last 3 rooms for a holiday weekend while 9,000 search QPS runs multi-table availability joins, a single relational database melts down. Slow search queries starve the connection pool, concurrent checkouts deadlock on overlapping dates, and naive hold timers risk billing users after their room expires.

Hospitality traffic follows a steep 1,000:1 search-to-booking funnel, keeping search and checkout fast requires splitting the architecture across a strict CQRS boundary:

𝟭. 𝗥𝗲𝗮𝗱 𝗣𝗮𝘁𝗵 (9,000 Peak QPS, p95 < 200ms):
Client -> API Gateway -> BFF -> Search Service -> Elasticsearch (Geo & Facet Filters) -> Redis (Room Availability Counters) + Rate Service

𝟮. 𝗪𝗿𝗶𝘁𝗲 & 𝗦𝘆𝗻𝗰 𝗣𝗮𝘁𝗵 (300 Peak Hold QPS, p95 < 500ms):
Client -> API Gateway (Idempotency Lease) -> Booking Service -> MySQL Primary (Inventory Lock + Outbox) + SQS (10-Min Hold Timer) -> CDC Stream -> Kafka -> Redis & Elasticsearch

The full two-tier idempotency design, the search pagination vs. sold-out rooms dilemma, and the 10x database sharding strategy are covered in the complete system design write-up here:

https://engineering.by-tech.workers.dev/articles/hotel-booking-system-design/

Even after separating reads from writes, three edge cases will still break checkout in production:

𝟭. 𝗧𝗵𝗲 𝗠𝘂𝗹𝘁𝗶-𝗡𝗶𝗴𝗵𝘁 𝗗𝗲𝗮𝗱𝗹𝗼𝗰𝗸
If User A locks Dec 20 then Dec 21 while User B locks Dec 21 then Dec 20, the database deadlocks. Instead of tracking individual physical rooms, we store one daily counter per room type and always lock rows in strict chronological order (Dec 20 -> Dec 21), keeping lock times under 5ms

𝟮. 𝗧𝗵𝗲 𝗟𝗮𝘀𝘁-𝗦𝗲𝗰𝗼𝗻𝗱 𝗣𝗮𝘆𝗺𝗲𝗻𝘁 𝗥𝗮𝗰𝗲
If a guest clicks "Pay" at minute 9:59 right as the 10-minute hold timer expires, the timer might release the room while the bank charges their card. To prevent charging for a lost room, checkout atomically flips the booking status to PAYMENT_PROCESSING before calling the payment gateway—which tells the background timer to back off.

𝟯. 𝗧𝗵𝗲 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗟𝗼𝗰𝗸 𝗧𝗮𝘅
Putting Bookings and Room Inventory in separate microservices with separate databases forces you into slow two-phase commits during checkout. Keeping reservations and daily inventory counters (73M rows for 2 years = 7.3 GB in RAM) in the same MySQL database lets a single local ACID transaction secure the hold in 2ms.

SystemDesign #DistributedSystems #BackendEngineering #MySQL #Kafka

Designing an Email Service at Scale

Subham Ghosh — Sun, 18 Oct 2026 02:48:57 +0000

How do you build a distributed platform that handles 1 billion users, 50 billion daily messages, and petabytes of storage with zero data loss?

Dive into the complete engineering breakdown here:
https://engineering.by-tech.workers.dev/articles/designing-an-email-service/

At a high level, modern email architecture operates as two independent, one-way pipelines bridging legacy protocols and cloud-native systems:

Outbound Delivery: Users submit messages via HTTPS REST APIs, where payloads are immediately offloaded to S3. Delivery tasks are queued in Kafka, and outbound workers manage DKIM signing and DNS MX resolution before delivering over SMTP.
Inbound Ingestion: Edge gateways terminate global Anycast connections on port 25 and perform lightweight SPF/DMARC checks. Heavy message bodies stream to S3, while a Kafka claim-check pointer triggers deep scanning, Cassandra metadata writes, and real-time WebSocket pushes to active devices.

Key technical constraints discussed in the article:

The Custody Contract: Why returning an SMTP 250 OK is an irreversible legal commitment that must wait for multi-AZ Kafka replication.
The Dual-Write Trap: How using a single-partition Cassandra atomic batch for transactional outboxes guarantees emails are never saved without also being queued for delivery.
Backscatter Prevention: How sending an automated bounce to a forged address can instantly get your outbound IP blacklisted by major ISPs.
Data Modeling: Sharding Cassandra by user_id and monthly time buckets to keep read latency under 200ms without unbounded partition growth.

Design a metrics monitoring system

Subham Ghosh — Sun, 11 Oct 2026 05:15:08 +0000

How do you design a metrics monitoring system for 100,000 servers?

I’ve shared my approach here: https://engineering.by-tech.workers.dev/articles/metrics-monitoring-system-design/

Here’s the high-level flow:

Agents collect metrics; a durable message queue buffers them.
A time-series database stores metrics for dashboard queries.
A stream processor evaluates alert rules independently of dashboard storage.
Notification workers deliver alerts and retry failed sends.
Older measurements become summaries to keep storage manageable.

I also cover capacity planning, push vs. pull collection, data retention, and recovery from failures - along with alert rules for sustained spikes and missing data, since a brief CPU spike needs a different response from prolonged overload.

What Does It Take to Build a Correct Real-Time Leaderboard?

Subham Ghosh — Sat, 03 Oct 2026 15:14:45 +0000

What happens when a gaming leaderboard must stay fast, accurate, retry-safe, and recoverable—even while millions of players are competing?

The article below captures my thought process and understanding of how to design this system:
https://engineering.by-tech.workers.dev/articles/real-time-gaming-leaderboard-hld/

The challenge isn’t simply displaying the Top 10. It’s preventing duplicate score updates, handling out-of-order events, returning exact player ranks, recovering after data loss, and safely finalizing tournament results.

The design covers idempotent score processing, Redis sorted sets, live updates, deterministic tie-breaking, sharding, and multi-region consistency.

SystemDesign #DistributedSystems #BackendEngineering #Redis

Under the Hood: Distributed Message Broker Design, Storage, and Failure Modes

Subham Ghosh — Sun, 27 Sep 2026 12:30:11 +0000

Most discussions about message brokers stay at a high level, placing generic broker boxes on a system architecture diagram. In practice, building or operating a distributed message broker requires understanding specific mechanics around storage, replication, and consumer coordination.

I wrote a detailed walkthrough analyzing how a fault-tolerant message broker works under the hood. It focuses on implementation details, edge cases, and failure modes rather than surface-level component diagrams. It is structured to be useful for engineers at any experience level.

Full article: https://engineering.by-tech.workers.dev/articles/distributed-message-broker-system-design/

Key Concepts Covered

Ordering vs Throughput: Using partition-based append-only logs to achieve horizontal scaling while maintaining strict ordering per key.
Replication Semantics: What acks=all and min.insync.replicas actually protect against during leader failures, and where failure windows still exist.
Duplicate Processing: Why consumer crashes lead to duplicate downstream side effects, and how generation fencing, idempotency keys, and transa ctional outbox patterns prevent state corruption.
Operational Boundaries: Managing backpressure, slow consumers falling behind retention windows, and broker disk exhaustion.

Architecture and Parameters Evaluated

Storage Engine: Partitioned, append-only retained logs optimized for sequential disk I/O.
Replication Factor: 3 replicas with min.insync.replicas = 2.
Consumer Coordination: Generation-fenced offset commits to reject stale consumer writes after rebalances.
Reliability Features: Retry topics, Dead Letter Queues, quotas, and retention cleanup policies.

The goal was to connect every architectural decision directly to a specific failure mode and detail the trade-offs involved.

Feedback, questions, and discussion are welcome in the comments!

How Google Drive Syncs Files and Resolves Concurrent Edits

Subham Ghosh — Sun, 20 Sep 2026 04:41:28 +0000

File synchronization looks simple until devices go offline, files become large, and multiple users edit the same content.
This article examines a Google Drive–style sync architecture and the problems behind reliable file updates:

Offline edits reconnecting at the same time
Large files with only small changes
Out-of-order retries, metadata updates, and notifications
Concurrent edits and version conflicts
The accompanying HLD diagram maps the flow across the client, API gateway, metadata server, block server, metadata database, and object storage. The article explains how these components interact during synchronization.
Read the full article: https://engineering.by-tech.workers.dev
Any feedback, questions, or discussion are welcome.