System Design 101 - The Building Blocks Every Developer Should Know

As a software developer why do we need a system design, isn't this one extra, "UNNECESSARY" step?? If we can directly jump into coding, make applications and make changes on the go. Well... right and wrong too.

For very simple applications like a todo list or a color changer, no system design is required. But when we talk about masses, lot of data moving from one point to another, lot of users accessing the services at the same time, sudden surge in the usage etc. there we need system design.

Think of it like this, when you are making noodles for yourself, you don't think a lot. But when you have to cook for 20 people, you prepare a whole plan first and only then execute it. That is system design. Designing the whole plan of how things will work and how things will take place, before actually picking up the tools (here, coding). It gives a clear cut idea of what we are building, and in case we are stuck, we can always refer back to it.

In one line, System Design = Components + a Common Goal. Lots of small pieces (database, server, cache, load balancer...) all working together to achieve one goal.

In this article I will walk you through the 7 core components of a system and the main ideas around them, so treat it as your System Design 101.

Requirement - Little bit of technical knowledge before hand (what a server, database and API roughly are).

What Are We Even Dealing With? Data!

At the end of the day, almost every application is moving data around. Mainly it is one of these:

1. Video
2. Audio
3. Text
4. Images

Keep this in mind, cause every component we discuss below exists to store, move, or serve this data better.

The Components

Here is the list of components we will look at:

1. Database
2. Server (Application Layer)
3. Frontend (Client Application Layer)
4. Cache Layer
5. Load Balancer
6. Message Queue
7. Monitoring and Logs

Let's start with the first one.

1. Database

The whole point of an application is to manage, deliver, present and analyse data, and give it to the users in a usable manner. A highly sophisticated application without data is of no use. So to hold this data for future operations, we need a place to store it, and that place is called a database (DB).

Now we have two types of databases.

SQL (Relational Databases)

SQL stands for Structured Query Language. Technically SQL is the language we use to talk to the database, and the databases that use it are called Relational Databases (MySQL, PostgreSQL, Oracle etc.), but in day to day talk everyone just calls them "SQL databases".

It's all about storing data in a structured manner. It uses tables (rows and columns) and close relations between these tables to present data. Every entity gets its own table, like Users, Posts, Pictures, Comments, Blogs.

Here is a small users table, every row is one user:

These are good for complex queries and transactions. A transaction is a group of operations which either all happen or none of them happen. Imagine sending money to a friend, money should go out of your account AND come into theirs. If only one of these happens, that's a disaster. SQL DBs guarantee this using ACID properties:

• A - Atomicity - all or nothing
• C - Consistency - data always follows the rules we set
• I - Isolation - two transactions running together don't mess up each other
• D - Durability - once saved, it stays saved, even if the power goes off

No-SQL (Non-Relational Databases)

Good for unstructured data and ingesting data in bulk. Stores data as key-value pairs, documents, columns or graphs (we'll see each type below). No fixed structure, in the first object we can save only one value while in the second we can store 50 values, no issue. Meanwhile SQL is structured, every row follows the same structure.

Here we will focus more on SQL first.

Constraints

While creating a table in SQL, we can set some rules which need to be followed to ensure consistent and good data. These rules are called constraints. The common ones are:

• NOT NULL - value can't be empty/null
• UNIQUE - every value in that column has to be unique (like email)
• DEFAULT - sets a default value if nothing is passed
• CHECK - value must pass a condition, e.g. age >= 18
• PRIMARY KEY - the unique identity of every row, like the id column. It is UNIQUE + NOT NULL together, and a table can have only one primary key.
• FOREIGN KEY - a column which points to the primary key of another table. This is how tables get connected. E.g. a posts table has a user_id column which points to users.id, so every post knows which user it belongs to.

This is important cause users will feed data, and it's our responsibility to ensure that it's correct and following a particular format.

Joins

Used to combine two or more tables (using the related columns, usually the foreign key) and then run a query on it to extract the required data. Most common ones are:

• INNER JOIN - only the rows which have a match in both tables
• LEFT JOIN - all rows from the left table + matching ones from the right
• RIGHT JOIN - all rows from the right table + matching ones from the left
• FULL JOIN - everything from both, matched where possible

Example - "Give me all posts along with the name of the user who wrote them" = join posts and users on posts.user_id = users.id.

Relations

This describes how two tables are connected with each other.

• One to One - one user has one profile, one profile belongs to one user
• One to Many - one user can write many posts
• Many to One - many posts belong to one user (same thing as above, just seen from the other side)
• Many to Many - a student can enroll in many courses, and a course has many students. For this we use a third table in between (enrollments) which holds student_id and course_id.

DB Scaling (the basics)

We know that each DB has a limit, it can't store unlimited data magically. To ensure it's healthy and running, we need to scale it as the users/data increase. Ways to scale it up:

• Vertical Scaling - increase the resources of the same machine - RAM, storage, CPU etc. But it can be done to a particular extent only, you can't keep adding RAM forever.
• Horizontal Scaling - add more DBs (more machines).

Issue with horizontal scaling in SQL is that it's tedious/hard. SQL data is connected through relations, so once it's spread over many machines, a single JOIN or transaction may need data from 3 different machines, and that gets slow and complicated. While in No-SQL every document stands on its own, so it can sit on any machine without worrying about the others. (We will dig deeper into DB scaling later in this article.)

No-SQL in short

• Easy to scale
• Schema less
• No heavy linking between different entities, unlike SQL's relations
• Every entity stands on its own

When SQL, When No-SQL??

• SQL - payments, transactions, places where data is structured and consistency is a must
• No-SQL - scale, speed, unstructured data

Consistency  >>  Availability                      =>  SQL
Availability + Scalability  >>  Strict consistency  =>  No-SQL

Types of No-SQL

1. Key-Value DB

The simplest one. Every piece of data is stored as a unique key and its value. The value can be anything - JSON, string, number, blob, array, or a combination of them. Think of it like a giant dictionary, give the key, get the value, super fast.

Used for - cache, cookies/sessions, simple data storage.
Ex - Redis, Amazon DynamoDB.

2. Columnar DB

Similar to a table in SQL, but it stores and reads data column wise. Normal SQL DBs read the table row by row, so in case we want only the marks column, a relational DB will still read row wise. Columnar DBs allow us to read only the columns we need.

Ask "what's the average of marks?" and a columnar DB reads only the marks column (90, 95), it doesn't even touch the names.

Ex - Google BigQuery, Amazon Redshift, Snowflake, ClickHouse (these actually speak SQL, but they store data column wise, which is the idea here).

• Good for analysis, like reading millions of rows to build a report
• Writing in this is slower than other No-SQL DBs

3. Graph DB

Data is stored as Nodes (the entities) and Edges (the relationship between them). Both nodes and edges can have properties. Like a Student node connected to a Course node with an "enrolled" edge, and that edge holds the year, marks and year of passing. Good to find hidden patterns in analysis/research, like "friends of friends" on social media or fraud detection.

Ex - Neo4j.

Cons - complex to design, and slow for normal bulk queries which don't need relationships.

4. Document DB

This one is used widely. Data is stored as documents, usually JSON-like objects (each document is like a key-value pair, where the value is a whole object).

• Flexible
• Schemaless
• No relations
• Ex - MongoDB

Good for - logging, user profiles. Like an Instagram post, one document can hold the image, video and text together.
Reads are fast cause one document has everything, no joins needed. But if the same data is copied into many documents, updating all of them gets messy.

2. Server (Application Layer)

The backbone of any application. It is responsible for doing the heavy lifting, does all the operations, fulfils users' requests and acts as a connecting bridge between the frontend and the database. It hides the complexities of the DB and gives easy access to it through APIs -> Endpoints (CRUD - Create, Read, Update, Delete).

3. Frontend (Client Application Layer)

The device you are using currently to read this article, the website or the mobile app. The GUI, or the front face of any application through which the user interacts, is called the frontend.

4. Cache Layer

A cache is a small but very fast storage (mostly kept in memory/RAM) which keeps the most asked data, and it mostly sits in between the server and the actual DB.

Imagine a store, you walk into it and see that most of the items are kept on their shelves while some of the most used items are kept right on the counter. The shelves are our DB, while the items on the counter are our cache, for quick and easy access.

Now imagine a situation, your website has a hero section where you are showing 6 courses, and each course needs an image, course details, price and a preview.

Single user  = 6 courses x 4 APIs        =        24 requests
10,000 users = 24 x 10,000               =   2,40,000 requests

So 24 requests just to populate the hero section for a single user. Now imagine 10,000 users land on the site, it becomes 2,40,000 read requests to the DB, all asking for the exact same data. So we cache it and serve it from there only, cause it's not going to change, it's the same for every user. It saves bandwidth, decreases latency (the delay between asking for data and getting it) and improves user experience. And remember, high latency = poor performance = bad UX = business loss.

Important terminology

1. Cache Hit - data is present in the cache = fast response
2. Cache Miss - data is not in the cache, so we go to the DB = slower response
3. TTL (Time To Live) - expiry time of the data in the cache. Once the TTL is over, the data is removed and fetched fresh next time.

Storage in cache

Data in a cache is stored like this:

And cache size <<<<<<<<<<<< actual DB. The cache is kept small cause it lives in RAM, which is expensive. So we only keep the hot data, the stuff that's asked for again and again, otherwise it will become just another simple DB.

Caching Strategies

1. Read Through Cache (RTC) - for reading

First we will read the cache, if it's a cache miss then only we will read the DB and update the cache on the way back. (In pure read-through, the cache itself goes to the DB on a miss. When our app code does that job instead, it's called cache-aside, but the idea is the same.)
For write requests, it always goes to the DB first.

2. Write Through Cache (WTC) - for writing

Every write goes to the cache and the DB together, the write is done only when both are updated. So the cache always has the latest data.
Good for fast paced data where the latest data matters - like stock values.
Downside - writes are a bit slower, cause we write at two places.

3. Write Around Cache (WAC)

Writes go directly to the DB, no cache update. Reads always go through the cache, on a hit we return, on a miss we go to the DB, update the cache and then return.
Good when data is written once and rarely read immediately after, so we don't fill the cache with stuff nobody asks for.

4. Write Back Cache (WBC)

In this, the data is first written in the cache and the response is sent back to the BE (backend) right away, the write on the DB happens asynchronously. The issue is that there may be inconsistency, what if an error occurs while writing to the DB, or the cache crashes before saving? That data is lost.
Read is similar to above.

Speed >>> Consistency is the trade-off here.

Ex - high write traffic, like Swiggy/Zomato/Uber Eats where order status and delivery location update very quickly.

Cache Eviction Strategies

Cache is small, remember? So when it's full and new data comes, something has to go out. Who goes out is decided by eviction strategies.

1. LRU - Least Recently Used - remove the data which hasn't been used for the longest time. The most common one.
2. MRU - Most Recently Used - remove the data which was used most recently. Sounds weird, but think of streaming services, once you've watched an episode, you are not going to watch it again, the next episode is what you need.
3. LFU - Least Frequently Used - remove the data which is used the least number of times.
4. FIFO - First In First Out - remove the oldest added data, like a queue.
5. LIFO - Last In First Out - remove the newest added data, like a stack of plates.

5. Load Balancer

As the application grows, we can't just rely on a single server, can we? What will happen if that server faults and goes down? The whole application will be unusable. To mitigate this and to give quick responses to the incoming requests, we have a cluster of servers working together. Now how will a request know which server to hit? For that we have the Load Balancer (LB). It decides which request will go to which server and maintains the load on the servers, so no one server dies under heavy load.

Responsibilities of LB

1. Choosing the server
2. Checking servers' health

LB Algorithms

1. Round Robin

Selects servers in numerical order, checks nothing. With 4 servers it goes 1, 2, 3, 4, 1, 2, 3, 4...

• Easy to implement
• Equal distribution

Issue - what if the servers aren't of the same config?

Here S1 will face difficulties to keep up with the others, cause it isn't capable enough to match them, but it still gets the same number of requests.

2. Geo Based Algo

Based on the server's and user's location. The servers are spread across the globe and the nearest one to the user is selected to entertain its requests. A user sitting in India gets the India server, not the US or EU one.

• Less latency
• Increases performance

Cons

• LB needs to manage extra details - location of servers, user location, and constantly check servers' health
• DBs across regions need to be maintained, ensuring the data is uniform everywhere
• VPN/Proxies can mislead the traffic. If your VPN says you are in the US while you are sitting in India, you get routed to a US server, that's one of the reasons your internet feels slower on a VPN.

3. Least Connection Algo

The server with the least number of active connections gets the next request.

Disadvantages

1. What if the server with the least connections is handling a heavy task, the next request is still sent to it. It will choke.
2. LB needs to keep records of the connections of each server to implement it.
3. What if some servers have fewer resources, still they will get the request -> slower response.

4. Least Time Algo (Least Response Time)

Selects the server with the least response time, eventually giving less average time. If S1 is answering in 50ms and S2 in 10ms, S2 gets the next request.

Less response time = usually high config server.
Ex - real time apps like trading.

In this the LB needs to calculate the average response time of each server, which depends on the type and config of the server, number of requests etc. It's complex, plus it keeps changing with the traffic.

5. IP Hash Algo

The user's IP is passed through a hash function (a function that turns any input into a number, and the same input always gives the same number), and the result decides the server. So the same IP always lands on the same server, good for keeping the user's session on one server. Good thing, the IP rarely changes in a single session, but the moment the connection breaks or the service provider changes, the IP changes. If the server changes, then the user's data needs to be there on the new server too. E.g. hash values 1-10 go to S1, 11-20 to S2, 21-30 to S3.

Major disadvantage - Scaling. New server -> hash ranges change in the LB -> new mapping for every server, so lots of users suddenly get shifted to a different server. (Consistent hashing is the technique used to reduce this reshuffling.) Not a great fit for microservices where servers come and go often.

6. Weighted Round Robin

In this, each server is assigned a weight/value, and the load balancer uses these weights to distribute the requests in a sequential manner.

Imagine server A with weight 1 and server B with weight 3. The LB will send 1 request to server A and 3 requests to server B, then again 1 to A, and so on. This fixes the issue of normal round robin with different config servers, give the powerful one a bigger weight.

In reality we use hybrid algos, we don't rely solely on a single algo.

Health Check

How does the LB know a server is down? By checking its health.

• Passive way - checks the responses of the already sent (actual) requests. If they keep failing, something is wrong.
• Active way - by sending extra requests (pings, like hitting a /health endpoint) after a particular interval to check if the server is up and running.

Parameters for Health Check

1. Interval - how often we check our server's health. Every 5 sec, every 2 hrs etc.
2. Timeout - how long do we wait to get a response before calling it a failure. E.g. if a server normally responds in 50ms (min) to 5 sec (max), then a response taking 10 sec is alarming.
3. Threshold - setting the limit for when to change the server's status.
   • Unhealthy threshold - after how many failed checks in a row we mark the server unhealthy and stop sending traffic to it. E.g. 3 failed checks in a row and the server is out.
   • Healthy threshold - after how many successful checks in a row an unhealthy server is considered healthy again and gets traffic back. E.g. n successful calls -> everything is going good, bring it back.

DB Scaling - Deep Dive

We know vertical scaling is good but it has its limit, after that we need to add more DBs. And we can do that in a better way too, using Replication and Partitioning.

Replication

Keeping copies of the same data on multiple DBs.

• Avoids Single Point Of Failure (SPOF) - one DB dies, others are still there
• Keeps data closer for faster retrieval
• Read throughput increases

Single DB -> 10,000 reads/s
2 DBs     -> 10,000/s x 2
3 DBs     -> 10,000/s x 3

Writes don't scale like this though, every write still has to reach every copy.

Replication Algos

1. Single Leader Replication

One leader, n followers. The leader (L) takes all the writes and passes them on to the followers (F). Reads can be served by any of them.

In this, the write operation can be done in two ways, sync and async.

Sync

The leader waits till every follower confirms the write, then only says "done" to the user.

Advantages

• Followers are always updated
• If the leader fails, the followers still have up to date data

Disadvantages

• Longer response time - need to wait for the response from each follower DB
• Blocks resources
• Impractical - if one follower is slow or down, every write gets stuck

Async

The leader saves the write, says "done" to the user straight away, and updates the followers in the background.

• No blocked resources
• Fast response

Disadvantage - mismatched data. A user may get stale data from a follower which isn't updated yet, while another user gets the updated data. (This delay is called replication lag.)

In real life, many systems go in between - one follower is sync and the rest are async (semi-sync), so at least one copy is always up to date.

How to add a new follower

1. First create a snapshot of the current system, at that moment, with a timestamp.
2. Use this snapshot to feed data into the new follower, assume it took 3 hours.
3. In those 3 hours new data came in. So now get only the changes made after the first snapshot (the DB keeps a log of all changes, so we just replay it).
4. Populate this data.
5. Connect it with the actual system.
6. Congrats, new follower added!

How to elect a leader if L fails

1. Check which follower has the latest data.
2. Promote F1 as the leader and update F2 to see F1 as its leader.

(Also make sure the old leader, if it comes back, knows it's not the leader anymore, otherwise we get two leaders, called split brain.)

2. Multi Leader Replication

More than one leader, usually one in each data center. Each leader first populates its own followers and the other leaders, which then populate their followers.

• Users write to the nearest data center, so writes are faster
• Multiple leaders = more fault tolerance
• Collaborative editing (like Google Docs)
• 2 leaders can write on the same data in parallel, but this increases the chances of conflict.

To resolve conflicts:

1. Last write wins - the latest timestamp wins
2. Replica/Leader with the higher value assigned wins - each leader gets an ID/priority, higher wins
3. Do nothing - pass the conflict to the user and the user will pick the value (like git merge conflicts)

3. Leaderless Replication

No leader. Every query (read/write) goes to multiple nodes to get the response. (Ex - Apache Cassandra, ScyllaDB, Riak.)

Say User1 updates some data, DB1 and DB2 get it but DB3 misses it. Now when User2 reads, DB1 and DB2 give the updated data while DB3 gives stale data.

So do users need to wait for the response from every DB??

Nope. There are different algos and we can follow one of them. One of them is Quorum. We wait only for enough DBs to make w + r > n (explained below), not all of them.

Servers  n = 3
Write    w = 2   (wait for 2 out of 3 to say ok)
Read     r = 2   (read from 2 out of 3)

Rule:  w + r > n   ->  2 + 2 > 3

Why does this work? Cause if 2 DBs have the new write and we read from 2 DBs, at least one of them will surely have the latest data.

For a read, say:

• Response 1 = updated data
• Response 2 = stale data

Every value carries a version number, so we pick the one with the latest version, and also fix the stale DB on the way (read repair).

Partition (Sharding)

Issue with replication:

• What if the data itself is huge? Every copy still holds ALL of it.
• Time for searching/any other action keeps increasing.

So here we divide the DB into smaller pieces, called partitions (also known as shards), and each partition can then be replicated too.

Only rules:

1. When the partitioned DBs are connected together, they should give the entire data back.
2. Data should be equally distributed. (One partition with 90% and the other with 10% is no good.)

Partition Algos

1. Partition by 'key' (range)

Say we have 10,000 users. Users 1 to 5000 go to P1, and users 5001 to 10000 go to P2.

Hotspot - a partition getting much more traffic/requests compared to the others, which may lead to node failure. E.g. if a big chunk of the newly active users fall in one range, that partition gets hammered while the other one chills.

2. Partitioning the data with 'hash' of key

Based on the value of a hash function, the data is spread more randomly and evenly. U1 hashes to 1 so it goes to P1, U2 hashes to 2 so it goes to P2, U3 to P2, U4 to P1 and so on.

Issue still persists, what if a single key becomes a hotspot (think a celebrity account with millions of followers)? Plus range queries like "users 1 to 100" now have to check every partition.

3. Partition by Secondary Index (local index)

We often search by something other than the key, like "all users from India". So each partition maintains its own secondary index for its own data. Writes are fast (just update the local index), but a read has to ask every partition and combine the results.

4. Common Global Index

One global index covering the data of all partitions (and the index itself is partitioned).

• Better reading - we know exactly which partition to go to
• Slower writing - one write may need to update the index on another partition too

CAP Theorem

• C - Consistency - every read gets the latest data
• A - Availability - the system always responds to every request
• P - Partition Tolerance - the system keeps working even when the network breaks and some nodes can't talk to each other

⚠️ Careful, the "P" here is not about partitioning/sharding our data like above. It's about a network partition, when the connection between nodes breaks.

CAP theorem says that in a distributed system, we can only provide two among the three at a time.

Combinations:

1. CA - Consistency + Availability, no partition tolerance. Only really possible with a single DB, cause there is no network between nodes to break.
2. CP - Consistency + Partition tolerance. When the connection breaks, the out of sync node stops serving (or the system waits) until it is restored and updated with the latest data. Users may get an error, but never wrong data.
3. AP - Availability + Partition tolerance. When the connection breaks, everyone keeps serving, so users may see stale or outdated data for a while.

• AP = Instagram - seeing a like count a few seconds late is totally fine
• CP = Bank - showing a wrong balance is NOT fine
• CA = Small app - single DB, no distribution

In reality, network failures will happen in any distributed system, so P is a must. The actual choice is between C and A when things break.

Sync vs Async Requests

Before message queues, we need to know that there are types in requests too, Sync and Async.

• Sync - the client sends a request and waits till the response comes back. Used in finance apps, security checks, and dependencies where one service depends on the result of another.
• Async - fire and forget. The client sends the request and moves on, the work happens in the background. Used for broadcasting, emails, SMS/WhatsApp notifications.

Example - E-commerce. Once a product is purchased, updating the inventory is sync (we must know the stock is updated), while sending the email and starting the delivery are async.

6. Message Queue

What happens when you place an order on a food ordering app? It first takes your order, then sets it to preparing, then packaging, picked up, delivered etc. Or when you book a flight, you get a message, email, WhatsApp, saying thank you for choosing xyz airlines, here is your ticket. Ever wondered how this works? If each of these features is handled by the main server, it will put additional load on our servers, while these don't need to happen instantly.

So we have a Message Queue (MQ). Once the ticket is booked, the server pushes the email, message and WhatsApp tasks into the message queue, from where they are picked up by their respective services and the task is done. These aren't as urgent as the booking itself, therefore they can be done asynchronously.

A message queue is like a buffer placed between the app (backend) and a service, and it's async in nature. It's helpful as it allows the app to shed some load onto the queue. So in case a request fails, the queue handles it on its own (retries), otherwise the backend had to handle it, which would be an additional load on it.

The one sending messages is the Producer/Publisher, the queue in between is the Broker, and the one picking them up is the Consumer/Subscriber.

Ex - RabbitMQ, Apache Kafka, Amazon SQS.

Qualities of MQ

1. Can hold requests
2. Communicates with services
3. Handles failed calls
4. Handles the load of services

MQ Algos

1. FIFO

• Strict order - messages are processed exactly in order. If one fails, it will not proceed, it will wait for its completion and then only move forward. Blocking in nature. 1 ✓, 2 ✓, 3 ✗... and the queue waits on 3.
• Unordered queue - skips the failed request and moves on to the next one, the failed one is retried later (or goes to the DLQ). 1 ✓, 2 ✓, 3 ✗ (skip), 4 ✓, 5 ✓.

2. Priority Queue

Attach a priority number with the call and higher priority ones get processed first. E.g. OTP SMS before a promotional email.

3. Pull Based

Requests are fed into the queue and the consumer pulls the requests on its own, whenever it's ready.

4. Push Based

Here the MQ pushes the request/call to the consumer as soon as it arrives.

Pub Sub Model

In a normal queue, one message goes to one consumer. In Pub/Sub (Publish/Subscribe), publishers publish a message to a topic, and every subscriber of that topic gets its own copy. Subscribers can get it pushed or pull it themselves.

Ex - an "order placed" event, the email service, the inventory service and the analytics service all subscribe to it and each one does its own job.

DLQ - Dead Letter Queue

Calls which couldn't be processed by the consumer, or had issues even after retries, are moved into a separate queue called the Dead Letter Queue. Later a log is given to the producer/developers for analysis of what went wrong and where. This way the bad message doesn't block the main queue, and it's not lost either.

Use Cases of MQ

• Async tasks
• Analysis / analytics events
• Load balancing / load levelling - absorbing sudden spikes so services work at their own pace
• Deferred systems
  • do them later
  • scheduled processes

When not to use it:

• Don't use it for a small app with low traffic, a queue is one more service to pay for, run and maintain.
• Real time stuff where the user is waiting for the result right now, big no. If we need an immediate result back from the other service, call it directly.

7. Monitoring and Logs

As developers, our work doesn't stop when the project is deployed or handed over, we need to ensure that it's up and running all the time. For that we need logs to see where the issue is, so that we can recreate it and fix it. Creating the app is just step one, maintaining it is the longer journey.

Fault/Error

Any condition where the system isn't working the way it's supposed to.

• Hardware based
  • Server, DB, RAM failures
  • Overheating
  • Power supply
• Software issue
  • Bad code
  • Edge cases
  • Config issue
  • Deployment issue
  • Merge requests/conflicts
  • Latency
  • Performance
• Human fault
  • Unpredictable
  • Unreliable
  • Yet this is our power too, which makes us different, humans learn from experience and improve.

Monitoring and Observability

Monitoring tells us that something is wrong (CPU at 95%, errors increasing). Observability helps us understand why it's wrong, using the three pillars:

• Logs - what happened, line by line
• Metrics - numbers over time (CPU, latency, error rate)
• Traces - the full journey of one request across all the services

Monitoring API

1. Throughput - how many requests the server is handling per second
2. Error codes (response codes) - how many 4xx/5xx we are throwing
3. Health check - passive or active
4. Latency - how long a response takes

Machine Monitoring

1. CPU usage - 75% -> alert
2. Memory - 90% -> scale up
3. Disk I/O - near limit -> alert
4. Network I/O

What Type of Application Is It?

Before jumping on to designing the system, we first have to understand what type of application it is. Is it:

1. Data Intensive
2. Compute Intensive

Data Intensive Applications

Here more focus is on data, how to gather it, store it, move it, operate on it and how to present it. Imagine Instagram, it is filled with data, be it reels, messages, photos, comments etc. WhatsApp messages, banking transactions, analytical dashboards etc.

From the point of technology, the main things we need to look at are the response from the database, network calls and server upgradation.

Primary goal - serve data from the DB to the user faster. Not much computation is required.

Worries

1. How fast can we read the data
2. How safely are we going to store the data (privacy)
3. How many users can access it simultaneously
4. What happens when a machine dies

Most of the work happens on the server side, where the data lives:

1. DB
2. Sharding (partitioning)
3. Caching
4. Replication
5. Consistency

Compute Intensive Applications

Challenge - heavy computation. These applications are CPU/GPU bound (computation ↑, data ↓).
Like - image processing, video rendering, ML model training, simulation, cryptography.

Worries

• How fast can we compute
• Can we parallelize the work
• Reduce computational cost
• Can we use a GPU instead of a CPU

Mixed Applications

Currently we have mixed applications too, which rely on both, like YouTube:

• Data intensive - serving videos to users
• Compute intensive - recommending videos

Tip - Ask, where is the time being lost? If time is lost in moving data, it's a data intensive feature. If time is lost in computation, it's a compute intensive feature.

Functional and Non-Functional Requirements

1. Functional - WHAT the system should do. Product features, the requirement list, all the features/functions of the application. For example in Amazon:

1. User can register
2. User can login
3. User can see products
4. Search products
5. Filter products
6. Add to cart, apply coupon, place order etc.
7. Make payment
8. Track order

2. Non-Functional - HOW the system should do it. Product properties, user expectations.

Example of Amazon (assume 1 million users):

1. Should handle 1 million users
2. Response time < 200ms
3. 99.9% availability
4. Data encryption
5. Scalable up to 10x traffic (sales)

Common non-functional requirements:

• Scalability - horizontal/vertical
• Availability - uptime
• Reliability - no data loss
• Performance - latency
• Security - authentication (who are you), authorization (what are you allowed to do), encryption
• Maintainability - code modularity
• Observability - monitoring and logging

Functional -> WHAT        Non-Functional -> HOW

DNS - Domain Name System

Ever wondered what happens when you type www.example.com in your browser and press enter? Computers don't understand names, they understand IP addresses (like 19.28.38.41). DNS is the phone book of the internet, it converts the name into the IP address. Explaining it in short:

1. First the browser checks its own cache, then the OS cache. If we visited the site recently, we already have the IP. Done, super fast.
2. If not, the request goes to the DNS Resolver (usually run by your ISP - Internet Service Provider, or something like Google's 8.8.8.8). It also has a cache, if the IP is there, it passes it back to the user.
3. Otherwise, the resolver hits the Root Server. There are 13 root server addresses in the world (with well over a thousand physical copies spread across the globe). The root doesn't know the IP, but it knows where the TLD - Top Level Domain server for .com, .in, .dev, .org etc. is.
4. Now we are redirected to the TLD server of the request, e.g. .com. This TLD server gives us the address of the Authoritative Name Servers of that website.
5. The authoritative name server finally gives us the IP address of our required server/site.
6. Now the request is made from the user to that IP address and the application is used. The resolver also caches it (for its TTL) for future use.

If nothing is cached, all 6 steps can take tens to a few hundred milliseconds, but most of the time step 1 or 2 already has the answer, so it's almost instant.

Cache levels

Remember our store counter analogy from the cache layer? Caching isn't only between the server and the DB, it happens at several levels:

1. Browser level
2. Operating System
3. DNS Resolver (already looked into)
4. Server side (Redis etc. between server and DB)

API - Application Programming Interface

APIs are the bridge which connects the frontend with the backend (or one application with another), and using this the applications work in real time. Now there are two types of APIs based on who can use them:

1. Access to another (Public / Third party APIs)

You must have come across platforms like Google Gemini, ChatGPT where you saw developer APIs, or IMDb APIs. These are public APIs maintained by these companies, and other developers can use them. Like with IMDb APIs you can look for movies, their ratings, cast, duration and show them in your own app. For this feature, some companies charge an amount, some keep it free.

2. Single App (Private / Internal APIs)

Here the APIs are for the application only, no public access. Like your own backend serving your React website and your Swift iOS app.

Types of APIs

1. REST - Representational State Transfer

The most widely used API style. It's light, gives freedom and is very easy to work with. Most modern applications use this. It mostly uses the JSON format. (REST is actually a set of rules/style on top of HTTP, not a format, JSON is just the most common choice.)

2. SOAP - Simple Object Access Protocol

It was widely used before JSON became popular and still a lot of legacy apps (old banking, enterprise systems) are using it. It uses XML, which is bulky in nature, and it has strict rules about everything from the header to the body. If an app wants to use these APIs, it has to handle the XML format, like a modern app consuming some old bank's APIs.

3. GraphQL - a query language for your API

This exposes a single API (endpoint) but a query language is attached. So the frontend asks for exactly the fields it needs. So instead of several APIs each responsible for a single task, we have a single API that does all of the work. We decide what data to choose and we get only that, nothing else. This saves bandwidth plus improves the response (no over-fetching or under-fetching).

4. gRPC (Protocol Buffers)

gRPC is developed by Google and uses Protocol Buffers to send data. Protocol buffers are a binary format, smaller than JSON/XML, that's why they are faster to transfer.

The downside - since it's binary, it's not human readable (you can't just open and read it like JSON), you need to define a schema (.proto file) first, and browsers don't support it directly. Therefore it's not great for browser-facing APIs, but it is a great fit inside a microservice architecture. Imagine several microservices talking to each other all the time, they just need small information to work on, there protocol buffers shine.

5. WebSockets

Best option for real time and persistent connections. In normal APIs, the client asks and the server answers, that's it. In WebSockets, the connection stays open and both sides can send data anytime (bi-directional). Widely used in chat apps where a constant connection is required. Chatting, multiplayer gaming, live scores and the live chat on a stream need WebSockets for real time bi-directional connection. But it is expensive to use, as every open connection eats server resources, and we need additional components to keep it alive since it stays on all the time.

RESTful APIs

Now we will be expanding on RESTful APIs as they are used widely in the industry. As mentioned, they use JSON to transfer data from one side to another. JSON holds data in key-value pairs, and a value can be a string, number, boolean, array or another object.

An endpoint for an API consists of two things:

Endpoint = Method + Path
           GET    + example.com/api/v1/resources

Methods in REST APIs

• GET - used to get data from the backend
• POST - used to send data from the frontend to create a new entry in the DB
• PUT - used to update data in the DB, this replaces the whole object with the data passed
• PATCH - also used to update, but this updates only the specified fields in the object
• DELETE - used to delete the entry from the DB

Ways to Pass Data Through an Endpoint

Now there is a need to pass data from the frontend to the backend, how can we do it?

1. Path (Path Parameter)

We can directly use the path to pass data to the backend.
Example - example.com/api/v1/user/{id}
Here id is the placeholder for the user's id. Then in the backend we can read the id and use it to retrieve the data from the DB.
Good for - identifiers, like user id, item id, slugs. Also for clear relations by nesting, like user/comments/{id} or blog/{id}/comments.

2. Query Parameter

Here also we pass the data in the URL, but it comes after a ?, like example.com/blog?page=4&sort=asc.
Good for - filtering, sorting, searching, pagination. Remember it's exposed in the URL.

3. Body of request

In this we send the data inside the request body. The better option for big or sensitive data (passwords, forms etc.), it needs slight additional handling. It's not visible in the URL, so it doesn't end up in browser history or server logs. (The actual security comes from HTTPS, which encrypts the whole request.)

API Request Example

POST          /api/v3/users                                     => Method + URL
Content-Type: application/json                                  => Meta data (headers)
Body:         {"name": "Testing", "last_name": "Testing2"}      => Body

API Response Example from Backend

1. Status Code

It tells about the status of the request in a nutshell, like whether it was a success or not.

• 2xx - everything is fine
• 3xx - just redirecting to another place
• 4xx - something is wrong from your side (client)
• 5xx - something is wrong from the server side

Common ones:

• 200 - OK, success
• 201 - Created (after a successful POST)
• 400 - Bad request
• 401 - Unauthorized (who are you? login first)
• 403 - Forbidden (I know you, but you're not allowed)
• 404 - Not found
• 500 - Internal server error

2. Body

We also send a body in the response, that's how the frontend gets the data. We can send any type of data back, but a good rule of thumb is to always send the data wrapped inside an object, don't directly return an array or a plain value. This way future changes are manageable, tomorrow if you need to add pagination info or a message, just add a new key without breaking the frontend.

{
  "users": ["value1", "value2", "..."]
}

Bonus - The Five Step Framework

Whenever you sit to design any system (or face a system design interview), follow these 5 steps:

1. Clarify Requirements - functional and non-functional, what are we building and for how many users?
2. Data + APIs - what data do we store (SQL/No-SQL) and what endpoints do we need?
3. High Level Architecture - draw the boxes: frontend, LB, servers, cache, DB, MQ.
4. Scale & Edge Cases - replication, partitioning, caching, what if a server dies?
5. Wrap Up - summarise the trade-offs you made and why.

Quick Recap

• System design is the plan before the code, components + a common goal.
• Database - SQL for structure and transactions, No-SQL for scale and flexibility.
• Server - the backbone, exposes the DB through APIs.
• Frontend - what the user sees and touches.
• Cache - small, fast, saves the DB from repeated reads.
• Load Balancer - spreads requests across servers and keeps an eye on their health.
• Replication & Partitioning - copy the data for availability, split it for size.
• CAP - when the network breaks, choose consistency or availability.
• Message Queue - do the not-so-urgent work later, asynchronously.
• Monitoring - our work doesn't end at deployment.

That's System Design 101. Your next step: pick an app you use daily, like Instagram or Swiggy, and sketch which of these components it uses and where each one sits.

I hope you found it helpful.
Until next time.
Peace ✌️