DEV Community: Adolfo Pedernera

Spurious Correlations and Fake Patterns: How Arithmetic Manufactures Proof

Adolfo Pedernera — Sat, 24 Oct 2026 09:25:50 +0000

Four levers — an arbitrary base, an exotic unit, an accommodated operation and a pre-chosen target — turn any measurement into any number. How numerology, gematria, Bible codes and sacred geometry use the same recipe, and how to test a claim that arrives with arithmetic attached.

Introduction

In 1994, the journal Statistical Science published a paper reporting that the names and death dates of a list of famous rabbis were encoded in the book of Genesis, spelled out at fixed intervals — take every 12th letter, or every 50th, or every 477th, and a name appears. The statistics were not obviously broken. The journal was peer-reviewed. The authors were serious people, and the result was presented as a discovery about the text.

Five years later, four mathematicians re-ran the experiment. They found the same kind of result in a Hebrew translation of War and Peace, using the same protocol applied to a text nobody considered sacred . Whatever the experiment was measuring, it was not measuring the book of Genesis.

That episode is the engine of this guide. Its lesson is not "those people were stupid." The lesson is that arithmetic is not a detector of truth. Division manufactures precision from any two numbers. Exponentiation manufactures drama. A ratio between two measured quantities always exists, because measuring produces quantities — and any ratio you can compute, you can also name. Give the operation enough freedom and it will hand you a number that looks like a message.

Why does the number feel like proof? Three properties do most of the work. Precision: decimals read as measurement, and measurement reads as care. Reproducibility: anyone who runs the same steps gets the same answer, which feels like a law of nature but is only a property of arithmetic. Impersonality: the calculation does not appear to have an author, so it does not appear to have a motive. None of those three properties is evidence about the world. Together they are enough to make a coincidence look like a finding.

The rest of this guide is organized as a demolition manual rather than a sermon. First, the recipe: four levers that turn any measured fact into any number you like, worked end to end on the Great Pyramid. Second, the same recipe in the wild — numerology, gematria, Bible codes, sacred geometry — plus a live specimen you can operate yourself in about ten seconds. Third, why the output feels like discovery even to careful people. Fourth, a four-question test you can run in a conversation. Finally, the places where this argument stops working, because a guide that only teaches suspicion is just a different way of being wrong.

One note before starting: almost nobody in this story is lying. Numerology, gematria and sacred geometry were built by people who believed them, and most of them were doing something recognizably human — looking for order in a world that does not volunteer it. The recipe is worth learning precisely because it does not require a fraud to operate it.

The Recipe: Four Levers and One Target

Every fabricated numerical "discovery" follows the same skeleton. There is a real measurement, a conversion, an operation, and a target. Four knobs get turned, and one of them decides the answer in advance.

An arbitrary base. The measurement itself: a length, a date, a count, a name. It has to be real, or at least checkable, because that is what makes the final number feel grounded.
An exotic unit. The base gets re-expressed in a unit you choose — varas, cubits, furlongs, letters, "sacred inches", megahertz. Changing units changes the number without changing the world.
An accommodated operation. Multiply, divide, square, take a root, raise to a power, sum the digits. Each one produces a different number from identical inputs.
A target chosen in advance — or afterwards. Either you fix the number you want and search for operations that reach it, or you fix the operations and search for a number that feels meaningful. The freedom has to live at one end of the pipeline. It is never absent.

Here is the canonical example, assembled step by step. Every individual step is true, or close enough that arguing about it would miss the point.

Step 1 — a real measurement. The Great Pyramid of Giza has a base side of roughly 230.33 metres . Different surveys give values between about 230.25 and 230.45 m, because the original casing stones are gone and the corners are eroded. Keep that range in mind; it will matter later.

Step 2 — an exotic unit. The vara is a Spanish length unit of about 0.8359 m, used in Spain and its American territories from the 16th century onward. It has no connection whatsoever to the Old Kingdom. Convert anyway:

\frac{230.33 m}{0.8359 m/vara} = 275.5 varas

Step 3 — a biological ruler. Call a dromedary's ordinary walking stride 1.96 m, which is 2.345 varas. Divide the pyramid by the camel:

\frac{275.5 varas}{2.345 varas} = 117.5

Step 4 — a cosmological multiplier. The sun crosses 15 degrees of sky per hour, so in fifteen minutes it covers 3.75 degrees of arc. Multiply:

117.5 \times 3.75 = 440.6 \approx 440

Step 5 — the announcement. Four hundred and forty. Which is the frequency of the A above middle C, in hertz. Which means the pyramid encodes concert pitch.

Stop and look at what just happened, because the arithmetic is flawless. 230.33 divided by 0.8359 is 275.5. Divide by 2.345 and you get 117.5. Times 3.75 gives 440.6. Nothing was miscalculated. The problem is not in the arithmetic; it is in the connection between the arithmetic and the claim.

Now the four levers, named:

Lever one, the base. 230.33 m is a modern metric average of a ruined structure. The builders did not know the metre. If you prefer 230.25 or 230.45, you get a slightly different answer, and you would pick whichever one lands better.

Lever two, the unit. The vara exists to convert. So does the cubit, the span, the furlong, the "pyramid inch". With five plausible units and a stride, you have five shots at a memorable number before you have done anything dishonest.

Lever three, the operation. Here is the part the published versions always omit: the operation is also a choice. Raise 117.5 to the power 3.75 and you get about 58 million. Divide it by 3.75 and you get 31. Add 3.75 and you get 121. The same three ingredients support at least four "discoveries", and only one of them resembles a musical note. The recipe reports the operation that worked.

Lever four, the target. In this example the target came first: 440 is famous, so the operations were tuned to reach it. In mystical systems the same freedom sits at the other end — the operations are fixed by tradition and the target is chosen after seeing what comes out. Either arrangement is the same trick with the receipt filed differently.

Now the detail that should end the conversation, and that the recipe cannot survive: hertz is cycles per second. The second is a modern unit, defined since 1967 by counting oscillations of a caesium atom. Concert pitch itself is an orchestral convention of the twentieth century — before that, "A" was whatever the local organ said it was. The pyramid was built around 2560 BCE, roughly 4,500 years before anyone could measure a frequency, in a civilisation with no electricity and no seconds. The number 440 is dimensionless: it is a ratio of a length to a length, multiplied by an angle in degrees. Attaching "megahertz" to it is not a measurement. It is a costume.

One more thing. Camel strides are not a constant of nature; they vary with the animal, the gait, the load and the sand. Run the band from 1.90 m to 1.98 m and the "discovery" sweeps from about 455 down to about 436. The target sits comfortably inside that band. So the claim is not even wrong in an interesting way — it is underdetermined, and the uncertainty was spent on making the number pretty.

The Same Recipe in the Wild

The Giza example is a demonstration built for this guide. The genre is older, larger, and in most cases entirely sincere. Four traditions are worth knowing, because they are the same machine with different paint.

Numerology maps letters to digits. The Pythagorean version assigns A=1 through I=9, then starts over at J=1, so the alphabet folds into nine buckets:

V = 1 + ((i - 1) mod 9)

where i is the position of the letter in the alphabet. Sum the letters of a name, reduce the total to one digit, read the meaning. Three separate choices are hiding in that sentence: which alphabet (Latin, Hebrew, Cyrillic, and their transliterations give different sums), which reduction rule (digital root, master numbers kept, vowels only, consonants only), and which form of the name (full, short, maiden, married, stage). Nine possible outputs, dozens of possible inputs, and the reading is supplied by the person who recognises themselves in it.

Gematria is the older and more disciplined cousin, and it does not need a mapping table at all: in Hebrew and in Greek, letters are numbers. Aleph is 1, bet is 2, tav is 400. A word already has a value. The freedom moves elsewhere — into spelling (plene or defective), into which word boundaries you count, into whether you add, multiply or compose values, and into which verse you decide is the relevant one. Each of those decisions is another draw from the machine.

Bible codes industrialise the process. Instead of words, take letters at equal intervals: every n-th letter, in either direction, across the whole text. The search space is enormous — text length times skip range times direction times spelling variants — and "near" is a threshold somebody has to set. In the 1994 experiment, the list of rabbis, the spelling of their names, and the definition of proximity were all settled in ways that later turned out to be doing the work . The tell was not the arithmetic. It was that the protocol was tuned until the text cooperated.

Sacred geometry skips letters and goes straight to architecture. Take two lengths from a building, form a ratio, compare it to a constant you like: π, φ, √2, 2π, the fine-structure constant. Six candidate ratios, four units, five constants — that is a hundred and twenty comparisons before anyone has lied. Some will land within one percent, and the one that lands gets published. The Great Pyramid is a generous donor here, because its outer casing is gone: you may measure the base you can see or the base you infer, the height as built or the height as eroded, and both choices are defensible.

The modern version is not mysticism; it is a Tuesday in a research lab. The same flexibility — which outcome to report, which covariates to include, when to stop collecting data, which subgroup to highlight — produces "significant" findings from noise. Simmons, Nelson and Simonsohn demonstrated it by showing that a completely impossible hypothesis (that listening to a song makes you older) could be made to pass conventional significance tests through undisclosed analytical choices , and Head and colleagues documented how widespread the practice is across scientific fields . No sacred text required. The lever is the same one.

System	What becomes a number	What the operator chooses	When the target is fixed
Pythagorean numerology	the letters of a name	alphabet, reduction rule, name form	after the reading
Gematria	Hebrew or Greek letters	spelling, word boundary, which operation	after the verse
Bible code (equidistant letters)	letter positions in a text	skip distance, direction, spelling, proximity rule	after the search
Sacred geometry	dimensions of a building	which lengths, which unit, which constant	after the survey
Pyramid frequency recipe	base, camel stride, solar arc	which unit, which animal, which operation	before the search

Look at the last column. That is the whole taxonomy. In the mystical systems the operations are inherited and the target is discovered; in the modern and contrived versions the target is named first and the operations are reverse-engineered. Either way, one end of the pipeline is free, and the freedom is what generates the result.

A live exhibit. The Love Calculator is the machine with the casing off, and it is worth two minutes because you can operate it yourself. It offers four methods for the same pair of names. Letter Match counts the unique letters two names share and divides by the unique letters in the longer name. Vowel Harmony compares vowel-to-letter ratios. Numerology applies the A=1 to I=9 mapping above, sums each name and reduces it. FLAMES reduces the pair to a count and uses that count to index a six-letter word: Friends, Lovers, Affectionate, Marriage, Enemies or Siblings. Romeo and Juliet score about 16.7% under Letter Match and about 70% under Vowel Harmony.

Same two names. Same calculator. Two very different verdicts, because the mapping is a design decision, not a measurement. That is precisely the gematria move: the letters are converted by a rule somebody chose, and the rule determines the answer. The difference is honesty — this calculator is labelled entertainment, and its own documentation says the algorithms are arbitrary heuristics. Most of the genre forgets to print that label. Use it as a specimen: try the pair of names you care about, then try two random strings of letters. A percentage and a message appear in both cases, delivered with exactly the same confidence.

Now fabricate your own

The exhibit above hides the machinery in someone else's code. Here the choices are made in front of you. The generator below holds 30 real reference measurements - the Great Pyramid's base at 230.33 m, the Hope Diamond at 45.52 carats, the Earth-Moon distance at 384,400 km, the speed of light, the number of keys on a piano - together with 15 operation templates and a free decimal point. Give it any target and it searches until something closes, then names the three levers it used: the operation, the units, and the decimal point.

(had to upload a static image here - can't embed the actual component - check the original guide if curious)

Run it twice: the same target gets a different "derivation" each time, which is the first sign that the derivation is not about the target. Then pick a number that means something to you. The machine does not know what it means and does not care - it will reach it anyway. A derivation that can be produced for any target is evidence for none of them.

Press Test it with a smaller corpus to see the part that makes the genre work. At 0.01% precision, six measurements cannot reach 440 at all; twelve still miss; the full corpus hits it in a handful of ways. Nothing changed but the size of the pile you are allowed to search.

Why a Fabricated Number Feels Like Discovery

If the recipe is this transparent, why does it work on intelligent people — including the people who build it?

The pattern detector is not optional. Human perception is tuned to find structure, and it fires on noise as readily as on signal. Psychology has a name for the disposition: apophenia, the tendency to perceive meaningful connections among unrelated phenomena, studied as a general bias toward false positives rather than as a defect of a few minds . You cannot switch it off, and you would not want to: the same machinery that finds faces in clouds finds the actual patterns that keep us alive.

Confirmation closes the loop. Once a candidate number exists, attention moves to the confirming instances — the other measurements that "fit", the other words with the same value, the other verses that mention the same thing. The misses are not collected, because nobody enumerated the search. This is the same selective accounting that makes gambling systems feel reliable: the wins are memorable and the losses are explained away.

Multiple comparisons do the arithmetic. This is the part that can be counted, so count it. If you test one hypothesis at the conventional α = 0.05, you have a 5% chance of a false positive. Test several independent hypotheses and the chance that at least one looks significant climbs fast:

P (at least one hit) = 1 - (1 - α)^{n}

Comparisons run (n)	Chance of 1+ hit at p < 0.05	1
1	5.0%	1: 5.0
5	22.6%	5: 22.6
10	40.1%	10: 40.1
20	64.2%	20: 64.2
50	92.3%	50: 92.3
100	99.4%	100: 99.4

Probability of at least one false positive as the number of independent comparisons grows, at alpha = 0.05.

Twenty comparisons is an afternoon with a spreadsheet. A sacred-geometry survey, a gematria reading that tries four spellings, and a Bible-code search over a few thousand skip distances are all running dozens or thousands of comparisons. None of them reports the denominator, and the denominator is the whole story.

What a p-value is not matters here too. The American Statistical Association's statement is explicit that a p-value does not measure the probability that a hypothesis is true, nor the size or the importance of an effect . It measures how surprising the data would be if a specific model were true. Reversing that conditional — "the data are surprising, therefore the model is false, therefore my story is true" — is the single most common error in popular statistics, and it is the exact error the recipe depends on.

Scale makes it worse. Ioannidis showed that when studies are small, effects are small, the analytical flexibility is large, and many teams chase the same questions, most published research findings will be false . That is not a claim about the honesty of scientists; it is a statement about what happens when selection operates on noisy measurements at scale. The mystical version of the same process — many texts, many measurements, many interpreters, one memorable hit — has been running for centuries.

And the arithmetic is genuinely reproducible. This is the last piece of the illusion, and the strongest. Run the Giza steps again tomorrow and you get 440.6 again. That stability feels like corroboration. But reproducibility is a property of division, not of pyramids: any two numbers you can measure will keep producing the same quotient forever, and that tells you nothing about whether the quotient means anything. Diaconis and Mosteller made the point precisely in their work on coincidences: the question is never "how improbable is this event", it is "how improbable is this class of events" — because the class is what you were really sampling from .

How to Test a Claim: Four Questions

The checklist is deliberately short enough to use while someone is still talking. It works on a pyramid ratio, a numerology reading, a headline about breakfast, and a dashboard that discovered a trend.

1. Was the target chosen before the search? Ask the counterfactual: would the claim's author have been equally impressed by 441? By 438? By 0.44? If the answer is no, then the target did the work, and the arithmetic was a search, not a test. Preregistration is the formal version of this question, and its absence is not a moral failing — it just means the finding is a hypothesis, not a result.

2. Is the unit the natural unit of the phenomenon? Would the people who built the thing have used that unit, and would they have had that concept at all? A vara on the Giza plateau fails this test before breakfast; hertz fails it permanently. The test also catches the reverse error: a unit that is unusual but intrinsic — natural units in physics, for instance — is not automatically suspect. Unconventional is not the same as arbitrary.

3. How many combinations were tried? Ask for the denominator, plainly: how many ratios, units, spellings, subgroups, outcomes, or stopping points did you look at before this one? If the answer is "we didn't count", assume at least twenty and read the table above. An honest large denominator is not disqualifying — a finding can survive a wide search if it is then tested on fresh data. Concealing the denominator is what turns exploration into fabrication.

4. Does it need rounding, truncation, or a favourable variant to land? Recompute the band. If the underlying measurement is uncertain by a few percent and the hit survives only at one edge of that range, the hit belongs to the input, not to the object. 440.6 is not 440; it is 440.6, and the distance between them is where the choice lives.

Two extensions are worth carrying around. First, the negative control : apply the exact same protocol to a text, building or dataset where the effect should be absent. McKay's team did this with War and Peace and the code followed them there . If your method finds the pattern everywhere, it is not finding the pattern. Second, ask whether the claim is a derivation or an observation. A proof is valid or invalid on its own terms; a numerical claim about the world has to survive contact with the world. Confusing the two is how "the math checks out" becomes a substitute for "the evidence checks out".

A claim that survives all four questions is not proven. It is merely worth the next test, which is the most any honest claim can offer.

Limitations: Where This Argument Does Not Apply

A demolition manual that leaves you unable to believe anything is not an improvement on credulity. Four boundaries matter.

Genuine correlations exist, and they survive the checklist. Smoking and lung cancer, CO₂ and global temperature, vaccination and antibody titres — these are not numerology with better branding. What separates them is not the sophistication of the arithmetic but the structure of the evidence: effects measured on prespecified outcomes, replicated out of sample, robust to reasonable changes in units and model specification, and predicted in advance. The recipe in this guide describes claims that were never subjected to any of that. It says nothing about claims that were.

Exploration is legitimate. Most real discoveries begin with someone noticing a pattern that was not the point of the study. Looking at data to generate a hypothesis is not p-hacking; publishing it as though it were a confirmed finding is. The remedy is a fresh test on fresh data, not abstention.

Mathematics is not arbitrary. Within mathematics, a proof is not a pattern claim: it is checked against axioms and rules, and it is either valid or it is not. What is arbitrary is the bridge between a number and a fact about the world — the choice of unit, the choice of measurement, the choice of what to compare. Confusing the two produces both errors at once: credulity toward numbers that were chosen to fit, and nihilism toward mathematics itself, as if the Pythagorean theorem were a matter of opinion.

Improbable events really do happen. The birthday paradox is not a trick of units or a rounding artifact; it is a correct probability about a class of events, and it is why shared birthdays in a room of twenty-three people are ordinary. The lesson of the recipe is not that improbable things are impossible. It is that the probability has to be computed for the event class you actually sampled, and that you have to count what you looked at .

Nothing on this page is a probability error, and separating it from one is the whole point. A coincidence is an event that happened and was noticed afterwards, so its odds can be computed honestly - that is what the It's Not a Coincidence guide does with the birthday paradox, the law of truly large numbers and Littlewood's law. Here the odds were never the issue. The target was chosen first, and the derivation was assembled to reach it. If the number survives the question "how many opportunities were there?" you are dealing with a coincidence; if it only survives "was the target chosen before or after the search?" you are dealing with a fabrication.

Two more honest caveats, specific to this page. First, the pyramid example is constructed here , not quoted from a published author: the measurements are real, the unit is real, the solar arc is real, but the assembly is ours, and it exists to show the method rather than to accuse anyone. If you meet a version of it in the wild with a straight face and a citation, the four questions will handle it. Second, the checklist is a filter, not a court. It will reject some true claims that happen to be poorly presented, and it can be gamed by anyone willing to run a fresh preregistered study — which is exactly the behaviour we want to reward, and the reason the filter is worth applying anyway.

The line worth drawing is consequences. A love calculator that produces a number for two names is a party game, and its own documentation says so; the machinery is visible, the stakes are zero, and the amusement is real. The same machinery in a hiring decision, a medical choice, a loan application or a relationship ultimatum is a different object entirely. Arbitrary arithmetic is fine as entertainment. It is not fine as a basis for decisions that change someone's life.

Frequently Asked Questions

Q: Does the Great Pyramid really encode the note A at 440 hertz?

A: No. The example in this guide is constructed to demonstrate a method, not quoted from a researcher. It uses a real base length, a real historical unit and a real solar arc, then multiplies them together until the number 440 appears. Hertz measures cycles per second, a unit defined in the twentieth century, so it cannot describe a building from 2560 BCE.

Q: What is a spurious correlation?

A: A statistical relationship that appears in data but does not reflect a causal connection between the two things measured. It can arise from chance, from a shared third factor, or from the way the data were selected and grouped. With enough variables, some pairs will always line up.

Q: Is numerology always wrong?

A: As a factual claim about the world, yes: letter-to-number mappings are chosen by people, so the results are design decisions rather than measurements. As a cultural practice and a source of entertainment, it is not a factual claim at all. The problem begins when a mapping starts influencing decisions about money, health, hiring or relationships.

Q: What is the difference between numerology and gematria?

A: Numerology usually assigns values to letters through a table, such as A=1 through I=9 in the Pythagorean system. Gematria works with alphabets where letters already carry numeric values, as in Hebrew and Greek, so a word has a value without any conversion table. Both then rely on the interpreter's choices about spelling, boundaries and operations.

Q: Why do so many measurements come out close to the golden ratio?

A: Because a ratio of two lengths you are free to choose can be steered toward almost any constant, and because with five or six candidate constants and several ways to measure each length, one comparison will land close by chance. Getting within one percent is easy when you have a hundred attempts and report only the successful one.

Q: What is p-hacking in one sentence?

A: P-hacking is the practice of trying multiple analyses, subgroups or stopping points and reporting only the one that crosses a significance threshold, which converts a search into a fake confirmation.

Q: Does a p-value tell me the probability that my hypothesis is true?

A: No. The American Statistical Association states explicitly that a p-value is not the probability that the hypothesis is true, nor a measure of the size or importance of an effect. It describes how surprising the data would be if a specified model were correct.

Q: If the math is correct, how can the conclusion be wrong?

A: Because correct arithmetic only guarantees that the output follows from the inputs. If the unit was chosen to fit, the target was named after the search, or the denominator was never counted, every step can be flawless and the conclusion still says nothing about the world.

Q: What is apophenia?

A: Apophenia is the tendency to perceive meaningful connections among unrelated phenomena, including patterns in noise. It is best understood as a general bias toward false positives in pattern detection rather than a rare disorder, and it is the perceptual half of what makes fabricated numerology feel convincing.

Q: Can the Bible code be tested scientifically?

A: Yes, and it was. Researchers extracted letter sequences at fixed intervals from Genesis and compared hit rates against control texts. When the protocol was applied to a Hebrew translation of War and Peace, comparable results appeared, which indicates the effect came from the flexibility of the method rather than from the text.

Q: How do I check a claim a friend sends me?

A: Ask four questions: was the target chosen before the search, is the unit natural to the phenomenon, how many combinations were tried, and does it need rounding or a favourable variant to land. If the answers are vague, the claim is a hypothesis rather than a finding.

Q: Isn't math supposed to be objective?

A: Mathematics is objective about what follows from what. It is silent about whether the quantities you fed it were the right ones, whether the units mean anything, or whether the comparison was chosen in advance. Objectivity in the calculation does not transfer to the story built on top of it.

notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.

🔗 Original guide

Data Engineering Basics for Developers: Pipelines, Warehouses, and Streaming

Adolfo Pedernera — Fri, 09 Oct 2026 20:10:27 +0000

What every developer should know about data engineering: ETL vs ELT, warehouses vs lakes, batch vs streaming, dimensional modeling, and pipeline patterns.

The Report That Ran for 14 Hours

A data analyst at an e-commerce company spent a week building a report that joined customer, order, and product data across three databases. On Monday, it worked — for 10,000 customers. By Friday, the company had a flash sale, and the customer table grew to 2 million rows. The query that used to take 20 minutes now took 14 hours. The analyst's colleagues asked why the report was so slow. The analyst said: "the data's bigger now."

The data was bigger — but the real problem was that no one had built a pipeline. Every query was hammering the production databases directly, joining across transactional systems that were designed for writes, not analysis. The data lived in silos with no staging area, no aggregation, and no warehouse. The report was slow not because there was more data, but because the data engineering was missing.

This guide explains what every developer needs to know about data engineering: how data flows from source systems into analytics, the difference between ETL and ELT, warehouses versus lakes, batch versus streaming, and the modeling patterns that make analytical queries fast. The goal is not to make you a data engineer — it is to make you a developer who understands why the analytics are slow and what infrastructure would fix it.

The Data Pipeline: Where Data Comes From and Where It Goes

Every organization has data sources — transactional databases, API logs, clickstream events, IoT sensors, payment processors. And every organization has data consumers — dashboards, machine learning models, business analysts, reporting tools. A data pipeline is the plumbing that moves data from sources to consumers.

A naive approach is to let consumers query the source systems directly. This fails for three reasons:

Performance. Analytical queries (aggregations, joins across millions of rows) compete with transactional queries (inserts, updates) for the same database resources. Production systems slow down.
Coupling. Every consumer queries the source schema directly. When the schema changes, every report breaks.
History. Transactional databases keep only current state. When an order is updated, the previous version is gone. Analytics needs history.

The pipeline solves all three: it extracts data from sources, transforms it into an analytical shape, and loads it into a dedicated store designed for queries.

ETL vs ELT: Where Transformation Happens

The two classic pipeline architectures differ in where the transformation step runs.

ETL (Extract, Transform, Load) transforms data before loading it into the target store. Data is extracted from sources, cleaned and reshaped in a staging area, then loaded into the warehouse in its final analytical form.

ELT (Extract, Load, Transform) loads raw data into the target store first, then transforms it in place. This became popular with cloud warehouses (BigQuery, Snowflake, Redshift) that can handle massive transformations with elastic compute.

Aspect	ETL	ELT
Where transformation runs	Staging server / pipeline	Inside the warehouse
Time to first query	Slow (transform before load)	Fast (raw data available immediately)
Storage cost	Low (only final data stored)	Higher (raw + transformed data stored)
Flexibility	Low (schema fixed at load)	High (transform on demand)
Best for	Legacy warehouses, strict schemas	Cloud warehouses with elastic compute

The trend is toward ELT because cloud warehouses make in-warehouse transformation cheap and flexible. But ETL still wins when you need to clean data before storing it — for example, masking PII or dropping malformed records before they enter the warehouse.

Warehouses vs Lakes: The Storage Decision

Data warehouses store structured, transformed data optimized for analytical queries. They use columnar storage (data stored by column, not row) which makes aggregations over millions of rows fast. Snowflake, BigQuery, Amazon Redshift, and ClickHouse are warehouses.

Data lakes store raw data in its native format — JSON, CSV, Parquet, images, even video — without transformation. They're cheap (object storage like S3) and flexible (you can store anything and figure out the schema later). The tradeoff: querying raw data is slower and messier.

A common modern architecture is the lakehouse : a data lake with warehouse-like query performance. Tools like Databricks and Dremio add a query engine on top of the lake, giving you the flexibility of a lake with the query speed of a warehouse.

Capability	Warehouse	Lake	Lakehouse
Query speed	Fast	Slow	Fast
Data flexibility	Structured only	Any format	Any format
Cost	Higher	Low	Medium
Schema enforcement	Strict	Schema-on-read	Optional
Typical users	Analysts, BI	Data scientists, ML	Both

The rule of thumb: store processed, queried data in a warehouse. Store raw, exploratory data in a lake. If you need both, consider a lakehouse.

Batch vs Streaming: When Data Arrives

Batch processing handles data in chunks — hourly, daily, or weekly. A nightly job that extracts the day's orders, transforms them, and loads them into the warehouse is a batch pipeline. Batch is simple, cost-efficient, and works well when data doesn't need to be current.

Streaming processes data continuously as it arrives — a click on the website, a sensor reading, a payment. Tools like Apache Kafka and Amazon Kinesis handle streams of events. Streaming is complex but enables real-time dashboards, fraud detection, and recommendation systems that react to events in milliseconds.

The classic pattern — the lambda architecture — runs both: a batch layer processes historical data (correct but slow), a speed layer processes real-time data (fast but approximate), and the results are merged. The more modern kappa architecture runs everything through streaming, using the stream as the single source of truth.

When to use each:

Batch: daily reports, monthly analytics, anything where 24-hour freshness is acceptable
Streaming: fraud detection, live dashboards, alerting, anything where latency matters
Lambda (both): systems that need both correctness (batch) and freshness (streaming)

Dimensional Modeling: Making Queries Fast

Once data is in a warehouse, the schema design determines whether queries are fast. Dimensional modeling (Kimball) is the dominant approach: separate facts from dimensions.

A fact table stores measurements — events, transactions, quantities. It's typically huge (millions of rows) and sparse: it has foreign keys to dimensions and a few numeric measures. A dimension table stores descriptive attributes — customers, products, dates, locations. It's smaller and denormalized: one row per entity with all its attributes.

Example — a sales fact table:

order_id	date_key	customer_key	product_key	quantity	revenue
1001	20260801	42	7	2	59.98
1002	20260801	7	3	1	19.99

The date_key, customer_key, and product_key link to dimension tables. The quantity and revenue are measures. This design makes the classic analytical query — "total revenue by product for July" — a simple JOIN + GROUP BY over a few tables, which columnar storage executes in milliseconds.

Star schema (the most common dimensional model) has one fact table in the center connected to multiple dimension tables — shaped like a star. It's denormalized for query speed, trading storage efficiency for fast reads. The SQL Index Selectivity Calculator helps you reason about whether the indexes on these join columns will actually be used as the fact table grows to millions of rows.

Slowly changing dimensions add a subtle but critical detail: customer attributes change over time, and analytics often needs to know what a customer's region was at the time of the sale, not today. The standard solutions are Type 1 (overwrite the attribute, losing history), Type 2 (add a new row with start/end dates, preserving history), and Type 3 (keep both current and original values). Most analytical teams converge on Type 2 for the dimensions that matter — the extra storage is worth the historical accuracy, and it is what makes "revenue by region as of the sale date" queries return correct answers months later when the sales team reorganizes territories.

The Data Pipeline in Practice

Let's trace a real pipeline for an e-commerce company:

Sources: the orders database (PostgreSQL), the website clickstream (Kafka events), the payment processor (API).
Extract: a nightly job reads new orders from PostgreSQL (batch); a Kafka consumer reads clickstream events in real time (streaming).
Load: raw order data lands in the lake as Parquet; clickstream events land in a raw Kafka topic.
Transform: the lake data is cleaned (remove test orders, normalize country codes), joined with dimension tables, and loaded into the warehouse.
Model: the warehouse schema uses dimensional modeling — a fact_sales table joined to dim_customer, dim_product, dim_date.
Consume: dashboards query the warehouse; ML models read aggregates from a feature store.

Where each calculator fits:

The Bandwidth Calculator sizes the network: if you're moving 50 GB of Parquet per night over a 1 Gbps connection, that's ~7 minutes of pure transfer — but with Kafka streaming at 10,000 events/second, the sustained throughput matters more.
The SQL Index Selectivity Calculator tells you whether the join columns in your warehouse queries are selective enough for indexes to help.
The Token Counter Calculator estimates the token cost if you're feeding warehouse data to an LLM for analysis — a growing pattern in modern data stacks.
The Big Number Calculator handles the astronomical counts in event volumes (billions of events per day) without overflow.
The Mean Median Mode Range Calculator and Standard Deviation Calculator power the basic descriptive analytics that dashboards show.

Practical Tips for Data Engineering

Never let consumers query source systems directly. Every analytical query against a production database is a slow query stealing resources from real users. Always stage data into a warehouse or lake.
Idempotent pipelines. A pipeline should be re-runnable without duplicating data. Use a watermark (last processed timestamp) or a key-based dedup (unique constraint on the natural key).
Backfill is inevitable. You will need to reprocess old data when a transformation bug is fixed or a new dimension is added. Design pipelines to be backfillable from raw source data — that's why storing raw data (lake/ELT) matters.
Monitor pipeline health. Track lag (how far behind the source the pipeline is), error rates per stage, and data volume. A silent pipeline that stops extracting for a week is worse than a pipeline that fails loudly.
Schema evolution is a contract. When you change a data schema, downstream consumers break. Version your schemas (Avro/Protobuf with compatibility rules) and communicate changes before shipping.
Store raw data forever. Even if you don't use it today, raw data is the insurance policy for future questions you can't yet imagine. Processed/aggregated data can be derived; raw data cannot be reconstructed.
Test with realistic data volumes. A pipeline that works with 1,000 rows may fail with 10 million. Test with a large sample and monitor the query plan with tools like BigQuery's query plan or EXPLAIN.

Limitations and When Data Engineering Is Overkill

Data engineering adds complexity: pipelines to run, storage to pay for, schemas to maintain. It's not always the right answer:

Small data. If your company has 100,000 rows and two analysts, a warehouse is overkill. A well-indexed database and a few SQL views handle it.
One-off analysis. A single report that needs data from two tables doesn't justify a pipeline. Export, join, analyze, and move on.
Low query frequency. If a report runs once a month, nightly batch is fine — you don't need streaming.
Tight coupling to a single source. If all your data comes from one system and one team consumes it, a direct connection with caching may suffice.

The tipping point is when multiple teams need analytical access to data or data volume exceeds what transactional databases can handle for queries. That's when the pipeline, warehouse, and modeling investment pays off.

Frequently Asked Questions

Q: What is the difference between ETL and ELT?

A: ETL transforms data before loading it into the target store. ELT loads raw data first and transforms it inside the warehouse. ELT is more popular with cloud warehouses (BigQuery, Snowflake) because elastic compute makes in-warehouse transformation cheap and flexible.

Q: What is a data warehouse vs a data lake?

A: A warehouse stores structured, transformed data optimized for analytical queries (columnar storage, fast aggregations). A lake stores raw data in any format (JSON, Parquet, video) cheaply. A lakehouse adds warehouse-style query performance on top of a lake.

Q: When should I use streaming instead of batch?

A: Use batch for anything where 24-hour freshness is acceptable (daily reports, monthly analytics). Use streaming for real-time needs: fraud detection, live dashboards, alerting. Kafka is the most common streaming platform.

Q: What is a star schema?

A: A dimensional model with one central fact table (measurements/events) connected to dimension tables (customers, products, dates) — shaped like a star. It's denormalized for query speed, making JOIN + GROUP BY queries fast on columnar warehouses.

Q: Why can't I query the production database directly?

A: Analytical queries compete with transactional queries for the same resources, slowing down production. Direct queries also couple consumers to source schemas (breaking on schema changes) and lose history (transactional databases keep only current state).

Q: What is Kafka?

A: Apache Kafka is a distributed event streaming platform. Applications publish events to topics; consumers subscribe and process them in real time. It's the standard for streaming data — clickstreams, logs, IoT events, and as the backbone of kappa architectures.

Q: How do I backfill a data pipeline?

A: Store raw source data permanently, then reprocess from the raw data when you need to fix a transformation bug or add a dimension. Design pipelines to be idempotent (re-runnable without duplicating data) using watermarks or key-based dedup.

Q: Do I need a data warehouse?

A: Not for small data or single-team analytics. The investment pays off when multiple teams need analytical access to data, or when query volume exceeds what transactional databases can handle. Start with well-indexed databases and SQL views; add a warehouse when queries slow down.

notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.

🔗 Original guide

HTTP & Networking Fundamentals: The Protocol Every Developer Works With

Adolfo Pedernera — Thu, 08 Oct 2026 05:42:20 +0000

HTTP's origin, methods, status codes, headers, caching, content negotiation, HTTP/2 vs HTTP/3, and how the web works.

The Protocol That Started With a Single Document

In March 1989, Tim Berners-Lee, a computer scientist at CERN, circulated a proposal titled "Information Management: A Proposal." It described a distributed hypertext system for sharing scientific papers across the laboratory's many incompatible computers. The idea was not new — hypertext had been discussed since the 1960s — but Berners-Lee built something nobody else had: a working implementation.

By 1991, the system was live. It had three parts: a markup language (HTML), a way to address documents (URLs), and a protocol to transfer them. That protocol was so simple it fit on a single page of notes: the client sent a request line ("GET the document at this path"), the server replied with a status line and the content. There was no request body, no headers beyond a couple of fields, no caching rules, no authentication. It was called HTTP — HyperText Transfer Protocol — and it transferred exactly one thing: a hypertext document.

Three decades later, HTTP is the foundation of everything on the web — not just documents, but APIs, streaming video, real-time chat, file uploads, and the billions of requests mobile apps make every second. The protocol that started as a single-line request is now a family of three major versions with request framing, multiplexing, compression, and encryption built in. This guide explains the protocol from its origin to its modern form: the request-response model, the methods and status codes that carry meaning, the headers that control behavior, and how the later versions fixed the problems of the earlier ones.

The Request-Response Model

Every HTTP interaction is a request followed by a response. The client (browser, mobile app, curl) sends a request; the server replies; the connection can then be reused or closed.

A minimal request:

GET /products/42 HTTP/1.1
Host: example.com
Accept: application/json

A minimal response:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 128

{"id": 42, "name": "T-shirt", "price": 19.99}

The first line of each carries the essential meaning. The request starts with a method (GET), a target (/products/42), and the protocol version (HTTP/1.1). The response starts with the version, a status code (200), and a reason phrase (OK).

Everything after the first line is headers — key-value pairs that describe the request or response. A blank line separates headers from the optional body (the payload). For GET, there is usually no body; for POST, PUT, and PATCH, the body carries the data.

The design is deliberately stateless: each request is independent, carrying whatever context it needs. This is what lets HTTP scale — any server can handle any request without remembering prior conversations. State, when needed, is layered on top via cookies, tokens, or session IDs (see the security guide for how those work without leaking data).

Methods: The Verbs of the Protocol

HTTP defines a set of methods, each with a specific meaning and specific safety properties:

Method	Purpose	Safe	Idempotent
`GET`	Retrieve a resource	Yes	Yes
`HEAD`	Retrieve headers only, no body	Yes	Yes
`OPTIONS`	Ask what methods are allowed	Yes	Yes
`POST`	Submit a new resource or action	No	No
`PUT`	Replace a resource entirely	No	Yes
`PATCH`	Partially update a resource	No	No
`DELETE`	Remove a resource	No	Yes

Safe means the request has no side effects — a GET must not change server state. Browsers and intermediaries rely on this: prefetching, link crawling, and browser caching all assume GET is harmless.

Idempotent means repeating the request produces the same server state as making it once. PUT and DELETE are idempotent: sending the same PUT /products/42 twice leaves the same state. POST is not: two POST /orders requests create two orders. This distinction drives how clients handle retries — an idempotent request can be safely retried after a timeout; a non-idempotent one needs an idempotency key (a client-generated token sent in a header) so the server can deduplicate.

Status Codes: The Language of Outcomes

Status codes tell the client what happened, in three digits that are structured for machine parsing and grouped for human learning:

1xx — Informational. The request is being processed. Rarely seen outside of 100 Continue and 103 Early Hints.
2xx — Success. The request worked. 200 OK, 201 Created (new resource), 204 No Content (success, no body).
3xx — Redirection. The resource moved or you need to do something else. 301 Moved Permanently, 302 Found (temporary), 304 Not Modified (cache is still valid — critical for HTTP caching).
4xx — Client error. The request was malformed or unauthorized. 400 Bad Request, 401 Unauthorized (not authenticated), 403 Forbidden (authenticated but not allowed), 404 Not Found, 429 Too Many Requests.
5xx — Server error. The server failed. 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout.

The 4xx/5xx boundary is the most useful distinction to internalize: 4xx means the client sent something wrong, 5xx means the server failed. When an API returns 4xx, the client can fix its request; when it returns 5xx, the client should retry with backoff, because the server is temporarily broken.

429 Too Many Requests deserves special attention: it signals the client exceeded a rate limit and should read the Retry-After header before trying again. The API Rate Limit & Cost Calculator models how retries on 429s affect both latency and monthly cost.

Headers: The Control Panel

Headers carry everything that is not the resource itself. They fall into a few families:

Representation headers describe the body: Content-Type (the format — application/json, text/html), Content-Length (bytes), Content-Encoding (compression — gzip, br), Content-Language (human language). These let the client interpret the payload correctly.

Negotiation headers let client and server agree on format: Accept (the client says what it understands — Accept: application/json), Accept-Language, Accept-Encoding. The server picks the best match and echoes its choice in Content-Type/Content-Encoding. This is content negotiation , and it is how the same URL serves different representations to different clients.

Caching headers control where and how long responses can be stored: Cache-Control (with directives like max-age=3600, no-cache, no-store, public, private), ETag (a version identifier for revalidation), and Expires (legacy). A 304 Not Modified response tells a cache that its stored copy is still fresh, so no body is sent.

Security headers harden the response: Strict-Transport-Security (force HTTPS), Content-Security-Policy (restrict what scripts run), X-Frame-Options (prevent clickjacking), X-Content-Type-Options: nosniff. These are covered in depth in the security fundamentals guide.

Request context headers tell the server who and what: Host (which site on a shared server), Authorization (credentials — API key or Bearer token), Cookie, User-Agent, Origin. When URL-encoding query parameters or request bodies, the URL Encode/Decode Calculator keeps the values valid inside URLs and headers.

HTTP/2: Fixing the Bottlenecks

HTTP/1.1, for all its longevity, had structural problems: one request per connection at a time (head-of-line blocking), text-based framing that was expensive to parse, and no way to prioritize resources. As pages grew from one document to dozens of assets (CSS, JS, images, fonts), those problems became bottlenecks.

HTTP/2, standardized in 2015 and adopted by all major browsers and servers, fixed the core issues:

Multiplexing: many concurrent requests share one connection. No more head-of-line blocking at the request level.
Binary framing: messages are split into binary frames, faster to parse than text.
Header compression (HPACK): repeated headers are compressed, reducing overhead.
Stream prioritization: the client can tell the server which resources matter most.
Server push (later deprecated in practice): the server could preemptively send resources it expected the client to need.

The key consequence for developers: HTTP/2 makes the "one connection per domain" model viable, and it changes the performance calculus. Merging many small files into one (a common HTTP/1.1 optimization) becomes counterproductive under HTTP/2, which multiplexes them efficiently. The Bundle Size Impact Calculator models the transfer-time side of that calculus.

HTTP/3: Moving to a New Transport

HTTP/2 fixed the protocol but still ran on top of TCP — and TCP has its own head-of-line blocking: a single lost packet stalls the entire connection until it is retransmitted. For multiplexed streams, that means one dropped packet delays all concurrent requests.

HTTP/3 solves this by running over QUIC , a transport protocol built on UDP. QUIC provides reliability (like TCP), encryption (TLS built in), and — critically — independent streams: a lost packet only affects the one stream it belongs to. It also reduces connection setup from multiple round-trips to one, which matters enormously on high-latency connections (mobile networks, cross-continental links).

The practical implications:

Faster connections: QUIC's single-round-trip handshake (with connection ID that survives IP changes) makes the first request faster and keeps connections alive across network switches (Wi-Fi to cellular).
Resilience: packet loss on one stream no longer stalls the whole page.
Adoption: HTTP/3 is enabled by default on major CDNs and browsers. Websites that measure a difference usually see it most on slow, lossy mobile connections.

For developers, the transport change is largely transparent — the request-response model, methods, status codes, and headers are identical across HTTP/1.1, 2, and 3. What changes is performance, and measuring it requires tracking protocol versions and connection metrics, not just response times.

How HTTP Fits the Network Stack

HTTP sits at the application layer of the TCP/IP stack: applications speak HTTP, which runs over TCP (or QUIC for HTTP/3), which runs over IP, which runs over the physical network.

Each layer has a job:

Application (HTTP): what the request means and how the response is interpreted.
Transport (TCP/QUIC): reliable, ordered delivery of bytes between two hosts.
Internet (IP): addressing and routing packets across networks.
Link: moving frames over actual hardware.

When you "load a page," the browser: resolves the domain to an IP via DNS, opens a TCP connection (or QUIC), sends an HTTP request, receives the response, parses the HTML, then issues additional HTTP requests for every referenced asset — reusing the connection when possible.

This is why bandwidth math is part of web performance. A page that weighs 2 MB served to a user on a 10 Mbps connection takes at least 1.6 seconds of pure transfer time, before latency and overhead. The Bandwidth Calculator converts payload sizes into transfer times across connection speeds — the same arithmetic that determines whether a page feels fast or slow.

Practical Tips for Working with HTTP

Learn to read a raw request and response. curl -v https://example.com shows every header and byte. Being able to see what your browser or client actually sends is the foundation of debugging.
Get the caching headers right. Cache-Control: max-age for static assets, no-store for anything with user-specific or sensitive data. A 304 saves the bandwidth of re-downloading unchanged resources.
Use the right status code. A 200 with an error body is a debugging nightmare — the client thinks everything worked. Use 400/404/422 for client errors and 429 when rate-limited, so clients can react correctly.
Understand Accept/Content-Type negotiation. Serve JSON to APIs and HTML to browsers from the same URL by honoring the client's Accept header.
Measure payload weight. Every byte you don't send is a byte the user doesn't download. Compress with Content-Encoding: gzip or br, and send only what's needed.
Use HTTPS everywhere. Strict-Transport-Security and modern TLS are not optional — HTTP/2 and HTTP/3 both assume encryption is the baseline.
Debug with the network tab, not guesswork. Response headers, timing breakdowns, and protocol columns (h2 vs h3) tell you exactly where the bottleneck is.

Limitations and Edge Cases

HTTP's simplicity has edge cases worth knowing. Requests and responses are asynchronous: a client can keep reading a response body while the server streams it, which is how Server-Sent Events and long downloads work. WebSockets, by contrast, upgrade an HTTP connection into a persistent bidirectional channel — useful for chat and live updates, but outside the request-response model (and harder to cache, load-balance, or retry). HTTP has no built-in security: it is a transfer protocol, not a security protocol — encryption and authentication are added by TLS and application logic. And the "stateless" design means anything that must persist across requests (sessions, cart contents) has to be managed explicitly by the application, which is where cookies, tokens, and server-side storage come in.

Frequently Asked Questions

Q: What is the difference between HTTP and HTTPS?

A: HTTPS is HTTP over TLS (Transport Layer Security). TLS encrypts the connection and verifies the server's identity, so a client knows it is talking to the real site and that no third party can read or alter the traffic. Modern HTTP/2 and HTTP/3 assume encryption is the baseline.

Q: What is the difference between GET and POST?

A: GET retrieves a resource and has no side effects (safe, idempotent). POST submits new data or triggers an action, may create resources, and is neither safe nor idempotent. Use GET for reading, POST for creating or acting.

Q: What does 401 vs 403 mean?

A: 401 Unauthorized means you are not authenticated — the server does not know who you are. 403 Forbidden means you are authenticated but not allowed to access the resource. Missing/invalid credentials get 401; insufficient permissions get 403.

Q: Why do I get a 304 Not Modified?

A: Your client already has a cached copy that is still valid. The server returns 304 with no body to confirm the cache is fresh, saving the bandwidth of re-downloading the resource. It is the core mechanism behind HTTP caching.

Q: What is the difference between HTTP/1.1, HTTP/2, and HTTP/3?

A: HTTP/1.1 is text-based, one request per connection. HTTP/2 adds multiplexing, binary framing, and header compression on the same TCP connection. HTTP/3 runs over QUIC (UDP-based), eliminating TCP head-of-line blocking and reducing connection setup to one round-trip.

Q: How does content negotiation work?

A: The client sends Accept headers describing what it understands (Accept: application/json, Accept-Language: en). The server picks the best matching representation and echoes the choice in Content-Type and Content-Encoding. This lets one URL serve different formats to different clients.

Q: What are the most important caching headers?

A: Cache-Control (max-age, no-cache, no-store, public, private), ETag (version identifier for revalidation), and Expires (legacy). Together they control where responses can be cached and for how long — the single biggest lever on repeat-visit performance.

Q: Should I use WebSockets or HTTP?

A: HTTP request-response is simpler, cacheable, and retryable — use it for most interactions, including polling. WebSockets provide a persistent bidirectional channel, useful for chat, live updates, and real-time collaboration, but they are harder to cache, load-balance, and debug. Choose the simpler option that meets the need.

notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.

🔗 Original guide

MLOps for Developers: Deploying, Monitoring, and Optimizing Machine Learning Models

Adolfo Pedernera — Thu, 03 Sep 2026 22:44:26 +0000

How to deploy, monitor, and optimize ML models in production: GPU selection, VRAM requirements, cloud vs local cost, model drift, and CI/CD pipelines for ML

The Model That Worked in Jupyter and Failed in Production

In 2020, a team at a financial services company trained a fraud detection model that achieved 99.2% accuracy in their Jupyter notebooks. They deployed it to production. Within a week, false positives had increased by 300%, and the company was blocking legitimate transactions from thousands of customers. The model wasn't broken — the data had changed. The pandemic had shifted spending patterns so dramatically that the model's training data no longer resembled reality.

This is the central lesson of MLOps: a model is not a feature you ship once. It's a system that degrades over time. Unlike traditional software, where a bug either exists or doesn't, ML models silently degrade as the world changes around them. The data drifts. The distribution shifts. The model that was accurate last month becomes inaccurate this month — and without monitoring, you won't know until customers complain.

MLOps (Machine Learning Operations) is the set of practices that keeps ML models working in production: deploying them reliably, monitoring them continuously, retraining them when they degrade, and doing all of this at scale. This guide covers the fundamentals: choosing the right hardware for inference, optimizing costs (cloud API vs self-hosting), detecting model and data drift, and building CI/CD pipelines that work for ML systems.

The ML Lifecycle: From Experiment to Production

The path from a Jupyter notebook to a production ML system has more stages than most developers expect:

Data collection & preparation — gathering, cleaning, labeling, and splitting data
Experimentation — training models, tuning hyperparameters, evaluating on validation sets
Model validation — testing on holdout sets, checking for bias, verifying performance metrics
Deployment — packaging the model, setting up inference infrastructure, exposing an API
Monitoring — tracking prediction latency, error rates, data drift, and model drift
Retraining — collecting new data, retraining the model, validating, redeploying

Traditional software has steps 1-4 and mostly skips 5-6. ML systems need all six — and steps 5-6 are where most teams fail. A model deployed without monitoring is a model that will fail silently.

The key insight: the experiment (steps 1-3) is often less than 20% of the total effort. The production infrastructure (steps 4-6) is where the real engineering happens.

Infrastructure: GPU Selection and VRAM Requirements

Deploying an ML model requires hardware — and for most modern models, that means GPUs. The central question: how much VRAM do I need?

VRAM (Video RAM) is the memory on a GPU where the model weights and activations live during inference. A 7B-parameter model in 16-bit precision requires roughly 14 GB just for the weights (7 billion parameters × 2 bytes each). Add activations, KV cache, and overhead, and a 7B model needs roughly 16-20 GB of VRAM for comfortable inference. A 70B model needs roughly 140-160 GB — which means multiple high-end GPUs.

The GPU landscape (2026):

GPU	VRAM	Approx. Price	Models it can run (16-bit)
NVIDIA RTX 5060	8 GB GDDR6	~$300	7B with quantization
NVIDIA RTX 5070	12 GB GDDR7	~$600	7B comfortably, 13B with quantization
NVIDIA RTX 5080	16 GB GDDR7	~$1,200	13B comfortably, 30B with quantization
NVIDIA RTX 5090	32 GB GDDR7	~$2,000	30B comfortably, 70B with quantization
NVIDIA A100	40/80 GB	~$10,000-15,000	70B comfortably, multiple models
NVIDIA H100	80 GB	~$30,000	70B+ with room for large batch sizes
Apple M5 Ultra	Up to 512 GB unified	~$5,000-15,000	70B+ with Apple's Metal optimization

Quantization reduces the precision of model weights to fit more into less VRAM. 8-bit quantization halves the memory requirement (7B model → ~7 GB). 4-bit quantization quarters it (7B model → ~3.5 GB). The cost: a small accuracy degradation that's often imperceptible for inference tasks.

The LLM Hardware Requirements Calculator estimates VRAM needs based on model size, precision (16-bit, 8-bit, 4-bit), and sequence length. Use it before buying hardware — the difference between a $300 RTX 5060 and a $15,000 H100 is whether your model fits in VRAM at all.

Cost Optimization: Cloud API vs Self-Hosting

The most consequential infrastructure decision for ML deployment is: should I call a cloud API or run the model myself?

Cloud APIs (OpenAI, Anthropic, Google) charge per token. Current flagship pricing (July 2026):

Model	Input	Output	Context
GPT-5.6 Sol	$5.00/M	$30.00/M	1.05M
GPT-5.4	$2.50/M	$15.00/M	400K
Claude Opus 4.8	$5.00/M	$25.00/M	1M
Claude Sonnet 5	$2.00/M	$10.00/M	1M
Gemini 3.1 Pro	$2.00/M	$12.00/M	-
DeepSeek V4 Flash	$0.12/M	$0.28/M	1M

For an application processing 10 million tokens per day using GPT-5.4 (mix of input/output), that's roughly $25-75 per day, or $750-2,250 per month. The benefit: zero infrastructure management, automatic scaling, and access to the latest models. The cost scales linearly with usage — there's no way to amortize hardware.

Self-hosting requires buying or renting GPUs but eliminates per-token costs. A $10,000 A100 amortized over 3 years costs roughly $275/month. Electricity and hosting add $100-200/month. Total: ~$400-500/month for unlimited inference. The break-even point depends on your token volume and the specific model — roughly 1-3 million tokens per day for current frontier models.

The LLM API Cost Calculator and Local LLM Break-Even Calculator compute the monthly cost for both approaches given your expected token volume. Use them to make the business case before committing to an architecture.

The hybrid approach (common in production): use cloud APIs for development, testing, and low-volume tasks; self-host for high-volume, latency-sensitive, or privacy-critical workloads. This gives you the flexibility of cloud APIs with the cost efficiency of self-hosting where it matters.

Monitoring: Detecting Silent Degradation

A model in production is like a sensor that slowly goes out of calibration. The inputs change (data drift), the relationship between inputs and outputs changes (concept drift), and the model's predictions become less accurate over time. Without monitoring, you won't notice until the business impact is severe.

Data Drift

Data drift occurs when the distribution of input data changes. A fraud detection model trained on 2019 spending patterns sees different data in 2024 — online shopping has increased, travel patterns have changed, new payment methods have emerged. The model's assumptions about "normal" behavior no longer hold.

Detection: Compare the statistical distribution of incoming features to the training data distribution. Common metrics:

Population Stability Index (PSI): measures how much a feature's distribution has shifted. PSI < 0.1 is stable; 0.1-0.25 is moderate drift; > 0.25 is significant drift
Kolmogorov-Smirnov test: a statistical test for whether two distributions differ significantly
Chi-square test: for categorical features, whether the category proportions have changed

Model Drift (Concept Drift)

Model drift occurs when the relationship between inputs and outputs changes — even if the input distribution stays the same. A spam filter trained to catch "Nigerian prince" emails becomes less effective when spammers switch to crypto scams. The inputs look similar (unsolicited emails), but the patterns have changed.

Detection: Monitor the model's prediction distribution over time. If the model was calibrated to predict 5% fraud but is now predicting 15% fraud, either fraud has genuinely increased (real change) or the model is degrading (drift). Ground truth labels (when available) let you compute actual accuracy; without labels, you monitor prediction distribution shifts.

Performance Metrics to Track

Metric	What it tells you	Alert threshold
Prediction latency	Is the model getting slower?	p95 > 2× baseline
Error rate	Are predictions failing?	> 1% for 5 minutes
Prediction distribution	Has the output distribution changed?	PSI > 0.25
Feature distribution	Has the input data changed?	PSI > 0.25 for any feature
Ground truth accuracy (when available)	Is the model still accurate?	Accuracy drops > 2% from baseline
Throughput	Can the system handle the load?	< 80% of target RPS

The retraining decision: when drift is detected, you need to decide whether to retrain. Retraining is expensive — it requires new labeled data, compute time, and redeployment. A common approach: retrain on a schedule (weekly/monthly) AND trigger additional retraining when drift exceeds a threshold.

CI/CD for ML: Pipelines That Handle Models

Traditional CI/CD (Continuous Integration/Continuous Deployment) automates building, testing, and deploying code. ML systems need additional steps because the "code" includes data, model weights, and hyperparameters — all of which can change independently.

A typical ML pipeline:

Data validation — check that new training data matches expected schema, distributions, and quality
Data preprocessing — clean, normalize, and split data
Model training — train the model with the current hyperparameters
Model evaluation — compare the new model's performance to the current production model
Model validation — check for bias, fairness, and edge cases
Model packaging — export to a deployable format (ONNX, TorchScript, TensorFlow SavedModel)
Deployment — roll out to production (canary, blue-green, or shadow deployment)
Monitoring — track the new model's performance in production

The key difference from traditional CI/CD: the model evaluation step compares the new model to the current model. If the new model doesn't improve performance (or degrades it), the pipeline should reject the deployment — even if all unit tests pass.

Deployment strategies:

Canary deployment: route 5% of traffic to the new model, monitor for errors, then gradually increase
Blue-green deployment: run two identical environments, switch traffic atomically
Shadow deployment: run the new model in parallel with the old, comparing outputs without affecting users

Rollbacks are critical. If the new model degrades in production, you need to revert to the previous version immediately. This requires keeping the previous model version available and having a fast rollback mechanism.

Practical Tips for MLOps

Version everything. Data, code, model weights, hyperparameters, and environment configurations should all be versioned. A model is not reproducible if you can't reconstruct the exact training environment.
Monitor from day one. Deploying a model without monitoring is shipping a bug you can't detect. Set up latency, error rate, and distribution monitoring before the model goes live.
Use feature stores. A feature store centralizes the features used for training and inference, ensuring consistency between the two. Without it, training uses one set of features and production uses another — a subtle but devastating bug.
Automate retraining. Manual retraining is retraining that doesn't happen. Set up pipelines that retrain on schedule and trigger on drift detection.
Test for edge cases. ML models fail on inputs that are rare in training data but common in production (unusual image formats, empty strings, extreme values). Test explicitly for these.
Plan for cold starts. A new deployment has no cache, no JIT compilation, no warm connections. The first requests will be slow. Pre-warm the model before routing production traffic to it.
Document model cards. A model card describes what the model does, what data it was trained on, its known limitations, and its performance characteristics. Future you (and your team) will thank you.

Limitations: MLOps Is Not a Silver Bullet

MLOps practices keep models working, but they don't solve fundamental problems:

Bad data → bad model. No amount of MLOps can compensate for biased, incomplete, or unrepresentative training data.
Wrong problem → wrong model. If you're predicting the wrong thing, a well-deployed model is a well-deployed mistake.
Regulatory compliance. MLOps doesn't automatically make your model GDPR-compliant or FDA-approved. Compliance requires additional processes.
Explainability. A model that works but can't be explained may be unacceptable in regulated industries (healthcare, finance, criminal justice).

MLOps is necessary but not sufficient. It keeps good models working — it doesn't make bad models good.

Frequently Asked Questions

Q: What is MLOps?

A: MLOps (Machine Learning Operations) is the set of practices for deploying, monitoring, and maintaining ML models in production. It extends traditional DevOps to handle the unique challenges of ML: data drift, model degradation, and experiment tracking.

Q: How much VRAM do I need for a 7B model?

A: A 7B-parameter model in 16-bit precision requires roughly 14 GB for weights alone, plus 2-6 GB for activations and KV cache. Total: 16-20 GB. With 8-bit quantization, you can fit it in 8-10 GB. With 4-bit, 4-6 GB.

Q: Should I use a cloud API or self-host?

A: Cloud APIs (OpenAI, Anthropic) are cheaper for low-volume usage and eliminate infrastructure management. Self-hosting is cheaper at high volumes (typically >1-3 million tokens/day). The break-even depends on your token volume and the specific model.

Q: What is data drift?

A: Data drift occurs when the distribution of input data changes over time. A model trained on last year's data may not work well on this year's data. Detection methods include PSI (Population Stability Index) and statistical tests like Kolmogorov-Smirnov.

Q: What is model drift?

A: Model drift (concept drift) occurs when the relationship between inputs and outputs changes. Even if input data looks the same, the model's predictions become less accurate. Detection requires monitoring prediction distributions and comparing to ground truth when available.

Q: How often should I retrain?

A: Retrain on a schedule (weekly or monthly) and additionally trigger retraining when drift exceeds a threshold. The right frequency depends on how fast your data changes — fraud models may need daily retraining; content recommendation models may need weekly.

Q: What is quantization?

A: Quantization reduces the precision of model weights to fit more into less VRAM. 8-bit halves the memory requirement; 4-bit quarters it. The cost is a small accuracy degradation that's often imperceptible for inference. Use it to run larger models on consumer hardware.

Q: What is a canary deployment?

A: A deployment strategy where you route a small percentage of traffic (e.g., 5%) to the new model, monitor for errors, then gradually increase. If something goes wrong, only a small fraction of users are affected.

notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.

🔗 Original guide

Caching Strategies Explained: From Browser to Database, and the Stale Data in Between

Adolfo Pedernera — Sun, 30 Aug 2026 11:10:34 +0000

HTTP caching, cache-aside, write-through, Redis patterns, and cache invalidation — plus the stale data problems.

The Request That Cost $100,000

In 2017, a major cloud provider experienced an outage that traced back to a single misconfigured cache. A caching layer that was supposed to serve data within 5 milliseconds had been configured with a TTL (time-to-live) of 24 hours. When the underlying database was updated at 2:00 PM, the cache continued serving the old data until 2:00 PM the next day. Customers saw stale pricing. Orders were placed at outdated rates. The company had to honor the incorrect prices for 12 hours of transactions — a six-figure loss caused by a single configuration value.

Caching is the most powerful performance tool in a developer's arsenal. It can make a system 100× faster, reduce database load by 95%, and turn a $10,000 database server into a $200 one. But caching is also the most dangerous tool, because it introduces a fundamental problem that no amount of engineering can eliminate: stale data. The cache and the database are two copies of the same truth, and keeping them in sync is one of the hardest problems in distributed systems.

This guide explains how to use caching effectively: the strategies that work, the patterns that prevent stale data, and the trade-offs that determine whether your cache is helping or hurting. Whether you're adding a simple HTTP cache header or designing a multi-layer caching architecture, the same principles apply.

The Two Hard Problems in Caching

Every caching decision comes down to two questions:

What do I cache? — The data that's read frequently but changes infrequently is the ideal candidate. User profiles, product catalogs, configuration settings, and rendered HTML fragments are all good candidates. Data that changes every second (stock prices, real-time sensor readings) is usually not worth caching.
How long do I keep it? — The TTL (time-to-live) determines the maximum staleness. A TTL of 5 seconds means a user might see data that's up to 5 seconds old. A TTL of 24 hours means they might see yesterday's data. The right TTL depends on the business: a news site might tolerate 5-minute-old headlines; a banking app must show real-time balances.

The tension is direct: longer TTL = better performance, worse freshness. A cache with a 1-hour TTL serves almost every request from memory (fast) but can show data that's up to an hour old (stale). A cache with a 1-second TTL is almost always fresh but provides almost no performance benefit because the cache expires before most users return.

The cache hit ratio measures how often the cache actually helps. A hit ratio of 95% means 95% of requests are served from the cache, and only 5% reach the database. This is the single most important metric for cache effectiveness. A cache with a 50% hit ratio is barely helping — half your requests still hit the database.

HTTP Caching: The First Line of Defense

The simplest and most effective caching happens at the HTTP level, before your application code even runs. Browsers, CDNs, and reverse proxies (NGINX, Varnish, Cloudflare) can cache HTTP responses based on headers your server sends.

The two most important headers:

Cache-Control tells caches how to handle the response:

max-age=3600 — cache for 1 hour (3600 seconds)
no-cache — always revalidate with the server before using the cached copy
no-store — never cache at all (sensitive data)
public — any cache can store this (CDNs, proxies)
private — only the user's browser can cache this (not shared caches)

ETag (entity tag) provides a version identifier. When the cache expires, the browser sends the ETag back to the server. If the data hasn't changed, the server responds with 304 Not Modified (no body), saving bandwidth. If it has changed, the server sends the new data with a new ETag.

Example — a product catalog page:

Cache-Control: public, max-age=300
ETag: "abc123"

This tells every cache between your server and the user: "Cache this for 5 minutes. After that, check with the server using ETag 'abc123' before using the cached version." The result: users see a page that's at most 5 minutes old, and your server only handles requests from users who haven't visited in the last 5 minutes.

The stale-while-revalidate pattern extends this: serve the cached version immediately (even if expired) while fetching a fresh version in the background. Users always get a fast response, and the cache stays fresh without blocking requests. This is how CDNs like Cloudflare and Fastly achieve both speed and freshness.

Application-Level Caching: Cache-Aside

HTTP caching handles static and semi-static content. For dynamic data — user sessions, personalized recommendations, computed results — you need application-level caching, usually with Redis or Memcached.

The most common pattern is cache-aside (also called lazy loading):

Application receives a request for user 42's profile
Application checks the cache: GET user:42
Cache hit → return the cached data (fast, ~1ms)
Cache miss → query the database (~50ms), store the result in cache with a TTL, return the data

The cache-aside pattern is simple and robust. The downside: the first request for any data is always a cache miss, and if the database is slow, that first request is slow. Also, if the data changes in the database, the cache still holds the old value until the TTL expires — the staleness problem again.

Cache-aside pseudocode:

function getUser(id):
    cached = redis.get("user:" + id)
    if cached != null:
        return cached
    user = db.query("SELECT * FROM users WHERE id = ?", id)
    redis.setex("user:" + id, 3600, user) // TTL 1 hour
    return user

The thundering herd problem. When a popular cache key expires, hundreds of simultaneous requests all miss the cache and hit the database at once. The database buckles. The fix: use a lock (mutex) so only one request fetches from the database while others wait and then read from the now-populated cache. Redis's SETNX (set-if-not-exists) is commonly used for this.

Write Strategies: When the Cache Updates

Cache-aside handles reads. For writes, there are three main strategies:

Write-Through

Every write goes to the cache and the database simultaneously (or the cache writes to the database synchronously). The cache is always up-to-date, but writes are slower because they wait for both systems.

Use when: consistency is more important than write speed (banking, inventory, booking systems).

Write-Behind (Write-Back)

Writes go to the cache immediately, and the cache asynchronously flushes changes to the database in batches. Writes are fast, but there's a window where the cache and database disagree. If the cache crashes before flushing, data is lost.

Use when: write speed matters more than perfect consistency (analytics, logging, metrics, social media likes).

Write-Around

Writes go directly to the database, bypassing the cache. The cache is only populated on the next read (cache-aside). This avoids polluting the cache with data that may not be read again soon.

Use when: most writes are not immediately re-read (user profile updates, settings changes).

Comparison:

Strategy	Read	Write	Consistency	Data loss risk
Cache-aside	Fast (hit) / Slow (miss)	Fast	Eventual	No
Write-through	Fast	Slow	Strong	No
Write-behind	Fast	Fast	Eventual	Yes (crash before flush)
Write-around	Fast (after first read)	Fast	Eventual	No

Cache Invalidation: The Hardest Problem

Phil Karlton famously said, "There are only two hard things in computer science: cache invalidation and naming things." The joke is that it's not a joke. Cache invalidation — ensuring the cache reflects the current truth — is genuinely difficult because the cache and the database are separate systems with no atomic coordination.

The three invalidation strategies:

TTL-based (time-to-live). Set an expiration time. Simple, but data can be stale for up to the TTL duration. Good for data that changes predictably (news feeds, trending lists).
Event-based (write-through). When the database changes, immediately update or delete the corresponding cache entry. More complex, but keeps the cache fresh. Requires application code to know which cache keys to invalidate when data changes.
Version-based. Include a version number in the cache key: user:42:v3. When the data changes, increment the version. Old versions expire naturally via TTL. This avoids the "delete the cache entry" race condition but requires the application to know the current version.

The race condition that breaks naive invalidation:

Application reads from database (value = A)
Database is updated by another process (value = B)
Application writes A to cache (stale!)
Application invalidates cache — but the invalidation arrives AFTER the stale write

The fix: use a cache invalidation queue (e.g., Redis pub/sub, Kafka) so invalidations are processed in order, or use version-based keys that make the race impossible.

Cache stampede (thundering herd). When a popular cache key expires, hundreds of simultaneous requests all miss the cache and hit the database at once. The database buckles under the sudden load. The fix: use a lock (mutex) so only one request fetches from the database while others wait and then read from the now-populated cache. Redis's SETNX (set-if-not-exists) is commonly used for this — the first request to acquire the lock fetches from the database; subsequent requests either wait or serve slightly stale data.

Multi-Layer Caching

Production systems rarely use a single cache. A typical architecture has multiple layers, each with different characteristics:

Layer	Technology	Latency	Scope	TTL
Browser cache	HTTP headers	0ms	Single user	Hours/days
CDN cache	Cloudflare, Fastly	~10ms	Global edge	Minutes/hours
Reverse proxy	NGINX, Varnish	~1ms	Per-datacenter	Seconds/minutes
Application cache	Redis, Memcached	~1ms	Per-instance	Minutes/hours
Database cache	PostgreSQL buffer pool	~0.1ms	Per-instance	Automatic

The key insight: each layer serves a different purpose. The browser cache eliminates network round-trips for returning users. The CDN cache serves static assets from edge locations close to users. The application cache reduces database load for dynamic data. The database cache keeps frequently accessed data in memory.

Cache hierarchy rule: a request should be served by the closest (fastest) layer that has the data. If the browser has it, don't hit the CDN. If the CDN has it, don't hit the origin. If the application cache has it, don't query the database. Each layer should fall through to the next only on a miss.

Practical Tips for Caching

Start with HTTP caching. Before adding Redis, make sure you're using Cache-Control and ETag headers correctly. This alone can reduce server load by 80% for read-heavy applications.
Measure your hit ratio. A cache with < 50% hit ratio is barely helping. Investigate why — maybe the TTL is too short, or the cache keys are too specific.
Use a small TTL for user-specific data. User sessions, preferences, and dashboards should have short TTLs (30 seconds to 5 minutes) to avoid showing stale personal data.
Use a large TTL for shared data. Product catalogs, configuration, and reference data can have long TTLs (1 hour to 1 day) because they change infrequently.
Implement cache warming. Pre-populate the cache on startup or after a deploy so the first users don't suffer cache misses. A simple script that queries the most popular items and stores them in Redis prevents the "cold start" problem.
Monitor cache memory usage. Redis and Memcached evict data when memory is full. If your hit ratio drops suddenly, check whether the cache is evicting keys due to memory pressure. Use allkeys-lru or volatile-lru eviction policies to keep the most recently used data.
Don't cache everything. Data that's written once and read once (event logs, audit trails) doesn't benefit from caching. Data that changes every second (real-time sensor data) can't be cached effectively. Focus on data that's read many times between writes.

Limitations and When Not to Cache

Caching is not a silver bullet. It adds complexity: another system to monitor, another failure mode, another source of bugs. Don't cache when:

Traffic is too low. If your site gets 100 requests per day, a cache adds complexity without meaningful benefit. The database can handle that load directly.
Data is highly personalized. If every user sees a completely different page, the cache hit ratio will be near zero. Cache the shared parts (header, footer, CSS) and fetch personalized parts from the database.
Consistency is critical. If showing stale data for even a second is unacceptable (financial transactions, medical records), caching may be the wrong tool. Use the database directly and optimize it instead (read replicas, connection pooling, query optimization).
Write-heavy workloads. If your application writes far more than it reads, the cache will be constantly invalidated, providing little benefit while consuming memory and CPU.

Frequently Asked Questions

Q: What is cache-aside?

A: The most common caching pattern: check the cache first, and on a miss, fetch from the database and store the result in the cache. Simple and robust, but the first request for any data is always a cache miss.

Q: What is the difference between write-through and write-behind?

A: Write-through writes to cache and database simultaneously (slow writes, strong consistency). Write-behind writes to cache first, then asynchronously flushes to the database (fast writes, eventual consistency, risk of data loss on crash).

Q: How do I choose a TTL?

A: It depends on how stale the data can be. User-specific data: 30 seconds to 5 minutes. Shared reference data: 1 hour to 1 day. Static assets: days to weeks. The shorter the TTL, the fresher the data but the lower the cache hit ratio.

Q: What is cache invalidation?

A: The process of ensuring the cache reflects the current truth. Three main strategies: TTL-based (expire after a time), event-based (update cache when data changes), and version-based (include a version in the cache key). All have trade-offs between consistency and complexity.

Q: What is a cache hit ratio?

A: The percentage of requests served from the cache rather than the database. A hit ratio of 95% means 95% of requests never reach the database. Below 50%, the cache is barely helping.

Q: What is the thundering herd problem?

A: When a popular cache key expires, hundreds of simultaneous requests all miss the cache and hit the database at once, potentially overwhelming it. Fix with a lock (mutex) so only one request fetches from the database while others wait.

Q: Should I use Redis or Memcached?

A: Redis is more versatile: it supports data structures (hashes, lists, sets), pub/sub, persistence, and Lua scripting. Memcached is simpler and slightly faster for pure key-value caching. For most applications, Redis is the better choice.

Q: How do I handle cache consistency in a microservices architecture?

A: Use event-driven invalidation: when a service updates data, it publishes an event (e.g., to Kafka or Redis pub/sub), and other services invalidate their cache entries in response. This keeps caches eventually consistent without tight coupling.

notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.

🔗 Original guide

Database Performance for Developers: Indexing, Query Plans, and the Queries That Don't Scale

Adolfo Pedernera — Fri, 28 Aug 2026 01:17:03 +0000

How database indexes actually work: B-trees, selectivity, composite indexes, covering indexes, and reading EXPLAIN plans so your queries scale past 10 million rows.

The Query That Worked Fine Until It Didn't

Every developer has a story about the query that worked perfectly in development and crawled to a halt in production. The table had 10,000 rows in your local database; in production it has 10 million. The query that took 5 milliseconds now takes 45 seconds. The dashboard times out. The support tickets arrive.

The culprit is almost always the same: a full table scan. The database engine reads every single row in the table, one by one, looking for the ones that match your WHERE clause. On a table with 10 million rows, that means reading millions of disk pages — work that could have been avoided with the right index.

An index is a separate data structure that the database maintains alongside your table. It works like the index at the back of a textbook: instead of reading every page to find every mention of "normalization," you look up the term in the index, which points you to pages 42, 87, and 156. Database indexes do the same thing, but with a twist that catches many developers off guard: an index is only useful if the query planner decides to use it. And the planner's decision depends on a single number: selectivity.

This guide explains how indexes actually work, when they help, when they hurt, and how to read an EXPLAIN plan to understand what your database is doing — so the next query that doesn't scale is one you catch before it reaches production.

How a B-Tree Index Works

Most database indexes are implemented as B-trees (or their variant, the B+-tree). A B-tree is a self-balancing tree structure that keeps data sorted and allows searches, insertions, and deletions in logarithmic time.

Imagine a table with 1 million rows and an index on the email column. Without an index, finding a specific email requires scanning all 1 million rows. With a B-tree index, the database starts at the root node, compares the target value to the keys in the node, and follows the appropriate branch — roughly 20 comparisons for 1 million rows (log₂ of 1,000,000 ≈ 20), each reading one disk page. The difference between reading 1 million pages and 20 pages is the difference between a query that takes seconds and one that takes milliseconds.

The B-tree structure also explains why indexes are sorted: range queries (WHERE age BETWEEN 25 AND 35) can find the starting point in the tree and then scan sequentially through the leaf nodes. A hash index, by contrast, supports only equality lookups — fast for WHERE email = 'x' but useless for ranges.

The cost of indexing is not free. Every index you add slows down INSERT, UPDATE, and DELETE operations because the database must update both the table and every affected index. A table with 5 indexes takes roughly 5× longer to insert into than a table with none. The art of indexing is balancing read speed against write speed — and knowing when an index helps so little that it isn't worth the cost.

Selectivity: The Number That Decides Everything

Selectivity is the fraction of rows a filter keeps. A filter WHERE email = 'a@b.com' on a unique column has selectivity ≈ 1/N (one row out of N). A filter WHERE status = 'active' on a column with two values has selectivity ≈ 0.5 (half the rows). The query planner uses this number to decide: is it cheaper to read the index and then fetch the matching rows, or just scan the whole table?.

The formula for selectivity is straightforward:

S = \frac{R}{N}

where R is the number of rows returned and N is the total rows in the table. A selectivity of 0.00001 (one row in 100,000) means the index is almost certainly used. A selectivity of 0.5 (half the rows) means the planner will likely ignore the index and scan the table directly — reading 500,000 rows via index lookups is more expensive than reading them sequentially.

The SQL Index Selectivity Calculator computes this from your table's row count, the number of distinct values in the indexed column, and the rows your query returns. It also reports the planner's likely choice (index scan vs sequential scan) and the estimated cost of each — useful for deciding whether an index on a given column will actually be used.

The cliff is steep. Going from 100 distinct values (1% selectivity, clearly indexed) to 10 distinct values (10% selectivity, borderline) multiplies selectivity by 10×. Below 100 distinct values on a large table, a single-column index is questionable; above 1,000 it is almost always worthwhile.

Composite Indexes: Order Matters

A composite index (also called a multi-column index) indexes two or more columns together. This is where most developers make their first indexing mistake: they assume the column order doesn't matter. It does.

A composite index on (last_name, first_name) can answer queries that filter on last_name alone, or on both last_name AND first_name. But it cannot answer a query that filters on first_name alone. The reason is the B-tree structure: the tree is sorted by last_name first, then by first_name within each last_name. Searching for first_name = 'John' without knowing the last_name is like searching for everyone named "John" in a phone book sorted by last name — you have to read the whole book.

The rule: put the most selective column first (the one that eliminates the most rows), and put columns used in equality filters before columns used in range filters. A query WHERE status = 'active' AND created_at > '2024-01-01' benefits from an index on (status, created_at) — equality first, range second. Reversing the order to (created_at, status) is less efficient because the planner can use the first column for range scanning but must filter the second column from the results.

Practical example: A users table with 5 million rows, 50,000 distinct cities, and 2 statuses (active/inactive). A query WHERE city = 'Berlin' AND status = 'active':

Index on (city, status): selectivity ≈ 1/50,000 × 0.5 = 0.00001 → index used, ~100 rows fetched
Index on (status, city): selectivity ≈ 0.5 × 1/50,000 = 0.00001 → same final result, but the planner may prefer the more selective column first

The difference becomes dramatic when the first column is not selective. An index on (status, city) forces the planner to scan all 2.5 million active users first, then filter by city. An index on (city, status) jumps directly to Berlin's ~100 users, then filters by status.

Covering Indexes: When the Index Is the Answer

A covering index is an index that contains all the columns a query needs, so the database can answer the query entirely from the index without ever touching the table. This is the fastest possible access path.

Consider a query SELECT user_id, email FROM users WHERE email = 'a@b.com'. With an index on (email), the database finds the row in the index, then fetches the corresponding table row to get user_id. With a covering index on (email, user_id), the database finds everything it needs in the index — no table access at all. On a large table, this can be 10-100× faster because the index is smaller and more cacheable than the table.

The tradeoff: covering indexes are wider (more columns = more bytes per entry), which means fewer entries per disk page and more storage. Use them for your most frequent queries, not for every query. A good candidate is a query that runs thousands of times per minute and returns few columns.

PostgreSQL's INCLUDE clause (available since PostgreSQL 11) lets you add non-key columns to an index without making them part of the sort key: CREATE INDEX ON users (email) INCLUDE (user_id). This gives you the covering benefit without the sorting overhead on the included columns.

Reading EXPLAIN: What the Planner Is Actually Doing

Every database provides an EXPLAIN command that shows the query plan — the step-by-step strategy the planner has chosen. Learning to read these plans is the single most useful skill for database performance.

A typical PostgreSQL EXPLAIN output for a query on a 10-million-row table:

Seq Scan on users (cost=0.00..183340.00 rows=1 width=36)
  Filter: (email = 'a@b.com'::text)

This tells you: the planner chose a sequential scan (reading every row), estimated cost is 183,340 (arbitrary units), expects to return 1 row, and applies the email filter to each row. The cost number is what the planner uses to compare strategies — lower is better.

After adding an index on email:

Index Scan using users_email_idx on users (cost=0.43..8.45 rows=1 width=36)
  Index Cond: (email = 'a@b.com'::text)

Now the planner uses the index, and the estimated cost dropped from 183,340 to 8.45 — a 20,000× improvement. The Index Cond shows what the index is filtering on.

Key terms in EXPLAIN output:

Term	Meaning	Action
`Seq Scan`	Full table scan	Add an index, or the table is small
`Index Scan`	Index used, then table fetch	Good for selective queries
`Index Only Scan`	Covering index, no table access	Best case for that query
`Bitmap Index Scan`	Multiple indexes combined	Common for AND/OR conditions
`Nested Loop`	Join: for each row in A, scan B	OK for small A, bad for large A
`Hash Join`	Build hash table on smaller table	Standard for equi-joins
`Sort`	Explicit sort step	Add an index on ORDER BY columns

The rows= estimate is critical. If the planner estimates 1 row but the query actually returns 100,000, it may have chosen a nested loop when a hash join would have been faster. Run ANALYZE after bulk loads to update statistics — stale statistics are the most common cause of bad plans.

When Indexes Don't Help

Indexes are not a performance panacea. They fail to help — or actively hurt — in several common scenarios:

Low cardinality columns. An index on a boolean column (2 distinct values) is almost never used. The planner would have to read half the table via index lookups, which is more expensive than a sequential scan. The same applies to any column where selectivity exceeds roughly 10-20%.

Functions on indexed columns. WHERE LOWER(email) = 'a@b.com' cannot use a regular index on email because the index stores the original values, not the lowercased versions. The fix is a functional index: CREATE INDEX ON users (LOWER(email)). Similarly, WHERE date(created_at) = '2024-01-01' cannot use an index on created_at; rewrite as WHERE created_at >= '2024-01-01' AND created_at < '2024-01-02' instead.

Implicit type casting. WHERE int_column = '42' (string literal) may prevent index use because the database must cast every row. Match types exactly: WHERE int_column = 42.

Leading wildcards. WHERE email LIKE '%@gmail.com' cannot use a B-tree index because the index is sorted from the left. WHERE email LIKE 'john%' can use the index; WHERE email LIKE '%john%' cannot. For substring search, consider full-text search indexes (GIN in PostgreSQL) instead.

Too many indexes. Each index slows writes. A table with 10 indexes takes roughly 10× longer to insert into than a table with none. If your workload is write-heavy (logging, event tracking), be stingy with indexes. If it's read-heavy (analytics, dashboards), be generous.

A Worked Example: The Slow Dashboard

A team reports that their admin dashboard takes 8 seconds to load. The query:

SELECT u.name, u.email, COUNT(o.id) as order_count
FROM users u
JOIN orders o ON o.user_id = u.id
WHERE u.status = 'active'
  AND u.created_at > '2024-01-01'
  AND o.total > 100
GROUP BY u.id
ORDER BY order_count DESC
LIMIT 20;

The users table has 5 million rows; orders has 50 million. EXPLAIN shows:

Sort (cost=452310.00..452310.50 rows=20 width=52)
  Sort Key: (count(o.id)) DESC
  -> HashAggregate (cost=452300.00..452305.00 rows=20 width=52)
        Group Key: u.id
        -> Hash Join (cost=125000.00..327300.00 rows=2500000 width=44)
              Hash Cond: (o.user_id = u.id)
              -> Seq Scan on orders (cost=0.00..150000.00 rows=10000000 width=12)
                    Filter: (total > 100)
              -> Hash (cost=100000.00..100000.00 rows=2500000 width=36)
                    -> Seq Scan on users (cost=0.00..100000.00 rows=2500000 width=36)
                          Filter: ((status = 'active') AND (created_at > '2024-01-01'))

Both tables are being fully scanned. The plan reads 2.5 million users and 10 million orders, joins them, groups, sorts, then returns 20 rows. The fix:

CREATE INDEX ON users (status, created_at) INCLUDE (name, email); — covering index for the user filter
CREATE INDEX ON orders (user_id) WHERE total > 100; — partial index for the order filter

After indexing, EXPLAIN shows:

Limit (cost=0.85..1250.50 rows=20 width=52)
  -> Sort (cost=0.85..1250.50 rows=5000 width=52)
        Sort Key: (count(o.id)) DESC
        -> GroupAggregate (cost=0.85..1000.00 rows=5000 width=52)
              Group Key: u.id
              -> Nested Loop (cost=0.85..800.00 rows=5000 width=44)
                    -> Index Only Scan using users_status_created_at_idx on users
                          Index Cond: ((status = 'active') AND (created_at > '2024-01-01'))
                    -> Index Scan on orders_user_id_idx on orders
                          Index Cond: (user_id = u.id)

Cost dropped from 452,310 to 1,250 — a 360× improvement. The dashboard loads in 20ms instead of 8 seconds. The key insight: the planner now reads only the ~5,000 active recent users (via the covering index) and joins only their orders (via the partial index), instead of scanning both tables entirely.

Practical Tips for Database Performance

Index columns in WHERE, JOIN, and ORDER BY clauses. These are the columns the planner needs to filter, join, or sort on.
Put the most selective column first in composite indexes. The column that eliminates the most rows should be leftmost.
Use covering indexes for hot queries. If a query runs thousands of times per minute, the extra storage is worth the speed.
Run ANALYZE after bulk loads. Stale statistics cause bad plans. PostgreSQL auto-analyzes, but large bulk loads may need a manual ANALYZE.
Use partial indexes for filtered queries. CREATE INDEX ... WHERE total > 100 is smaller and faster than indexing the whole table.
Don't index everything. Each index slows writes. A table with 10 indexes takes roughly 10× longer to insert into than a table with none.
Read EXPLAIN before deploying. If you see Seq Scan on a large table, investigate. The planner is telling you it has no better option.
Check selectivity before creating an index. If the column has fewer than ~100 distinct values on a large table, the index will likely be ignored. The SQL Index Selectivity Calculator tells you whether the planner will use it.

Limitations and When to Go Beyond Indexing

Indexing is necessary but not sufficient for database performance at scale. When indexes alone aren't enough:

Query rewriting. A query that uses a subquery may be slower than one that uses a join, even with identical indexes. EXPLAIN reveals the difference.
Denormalization. For read-heavy analytics, duplicating data (storing order_count on the user row instead of computing it) trades write complexity for read speed.
Partitioning. Splitting a 500-million-row table into partitions by date lets the planner skip entire partitions — a 100× reduction in data scanned for time-bounded queries.
Read replicas. For workloads with many more reads than writes, replicas distribute the read load across multiple servers.
Caching. Redis or application-level caching avoids the database entirely for frequently accessed, rarely changed data.

Indexing is the first line of defense, not the last. But it is the one that catches the most common performance problems — and the one that most developers underuse because they've never learned to read the planner's output.

Frequently Asked Questions

Q: What is a database index?

A: A separate data structure (usually a B-tree) that the database maintains alongside your table. It allows the database to find rows without scanning the entire table, similar to an index at the back of a textbook.

Q: Why isn't my index being used?

A: Most often because selectivity is too high — the filter keeps too many rows (roughly more than 10-20% of the table), so the planner decides a sequential scan is cheaper. Other causes: functions on the indexed column (WHERE LOWER(col) = ...), implicit type casting, or stale statistics.

Q: What is selectivity?

A: The fraction of rows a filter keeps: rows returned divided by total rows. Low selectivity (few rows) means the index is useful. High selectivity (many rows) means the planner will likely ignore the index.

Q: What is a composite index?

A: An index on two or more columns. Order matters: a composite index on (A, B) can answer queries filtering on A alone or A and B, but not B alone. Put the most selective column first.

Q: What is a covering index?

A: An index that contains all the columns a query needs, so the database can answer the query from the index alone without fetching table rows. This is the fastest access path for a query.

Q: How do I read an EXPLAIN plan?

A: Look for Seq Scan (full table scan — bad on large tables), Index Scan (index used — good), and Index Only Scan (covering index — best). The cost number lets you compare strategies; the rows estimate tells you how accurate the planner's guess is.

Q: Does every column need an index?

A: No. Each index slows down INSERT, UPDATE, and DELETE operations. Index columns used in WHERE, JOIN, and ORDER BY clauses, and be skeptical of columns with fewer than ~100 distinct values on large tables.

Q: What is a partial index?

A: An index with a WHERE clause: CREATE INDEX ... WHERE total > 100. It indexes only the rows matching the condition, making it smaller and faster than a full-table index. Useful for queries that always filter on the same condition.

notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.

🔗 Original guide

Security Fundamentals for Web Developers: Authentication, Authorization, and the Attacks You Need to Prevent

Adolfo Pedernera — Thu, 27 Aug 2026 14:37:19 +0000

Authentication vs authorization, JWT deep dive, OAuth 2.0, password hashing, XSS/CSRF/SQL injection prevention, and security headers.

The Developer Who Stored Passwords in Plaintext

In 2012, LinkedIn suffered a breach that exposed 6.5 million password hashes. The hashes were unsalted SHA-1 — a hashing algorithm so fast that an attacker with a modern GPU could crack billions of hashes per hour. Within days, 60% of the passwords were recovered in plaintext. The attackers didn't need sophisticated exploits; they just needed a database dump and a rainbow table.

The breach was preventable. Salting (adding random data to each password before hashing) and using a slow hashing algorithm (bcrypt, scrypt, or Argon2) would have made cracking infeasible. These weren't obscure techniques — they were well-documented best practices that LinkedIn's engineering team either didn't know or didn't prioritize. The cost: a $1.25 million settlement, a decade of reputational damage, and millions of users whose credentials were compromised.

This is the uncomfortable truth about web security: most breaches aren't caused by sophisticated attacks. They're caused by developers who didn't know the basics. SQL injection, cross-site scripting (XSS), and broken authentication have been on the OWASP Top 10 for over 20 years — and they're still the most common vulnerabilities in production applications.

This guide covers the security fundamentals that every web developer should know: the difference between authentication and authorization, how JWTs actually work (and how they're misused), how to hash passwords correctly, how to prevent the OWASP Top 10 vulnerabilities, and which security headers to include in every response. The goal isn't to make you a security engineer — it's to make you a developer who doesn't ship the vulnerabilities that attackers are counting on.

Authentication vs Authorization: Knowing Who vs Knowing What They Can Do

These two terms are confused so often that many developers use them interchangeably. They are fundamentally different questions:

Authentication answers: "Who are you?" — verifying identity (username/password, biometrics, API key)
Authorization answers: "Are you allowed to do this?" — checking permissions (is this user an admin? does this token have the read:users scope?)

A user who logs in with correct credentials is authenticated. Whether they can delete other users' accounts is an authorization decision. Confusing the two leads to vulnerabilities: a user who is authenticated but not authorized to access an admin endpoint can still access it if you only check "is logged in?".

The principle of least privilege: every user, service, and token should have exactly the permissions it needs — no more. A background job that only reads from a database should not have write access. A frontend API token should not have admin scopes. A user who can view their own profile should not be able to view others'.

JWT Deep Dive: What's Inside the Token

JSON Web Tokens (JWTs) are the most common mechanism for stateless authentication in modern web applications. A JWT is a self-contained token that encodes claims (user ID, permissions, expiration) in a JSON payload, signed by the server. The client sends the token in the Authorization: Bearer <token> header, and the server verifies the signature to trust the claims.

A JWT has three parts, each Base64URL-encoded and separated by dots:

eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjo0MiwiZXhwIjoxNjg3MzA1NjAwfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

Header (eyJhbGciOiJIUzI1NiJ9): algorithm (HS256) and token type
Payload (eyJ1c2VyX2lkIjo0MiwiZXhwIjoxNjg3MzA1NjAwfQ): claims — user_id, expiration, roles
Signature (SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c): HMAC of header+payload with server secret

The critical claim is exp (expiration). A JWT without an expiration is valid forever — if leaked, it's a permanent access token. Set short expirations (15 minutes to 1 hour) and use refresh tokens for longer sessions. The JWT Decoder parses tokens client-side, showing the header, payload, and signature — useful for debugging authentication flows and verifying that your tokens contain the expected claims.

Common JWT mistakes:

Storing sensitive data in the payload. JWTs are signed, not encrypted. Anyone can read them.
Not validating the signature on the server. If you trust the payload without verifying the signature, anyone can forge tokens.
Using alg: "none". Some JWT libraries accept tokens with no algorithm, bypassing signature verification entirely. Always pin the allowed algorithms server-side.
Long expirations without refresh. A token that's valid for 30 days is a 30-day window if leaked.

Password Hashing: Why Plaintext Is a Crime

Storing passwords in plaintext is the most basic security mistake — and it still happens. When a database is breached (and databases are breached), plaintext passwords are immediately usable. The fix is hashing : a one-way function that converts a password into a fixed-length string that cannot be reversed.

Not all hashing is equal. Fast hash functions (MD5, SHA-1, SHA-256) are designed for speed — which is exactly what you don't want for passwords. An attacker with a modern GPU can compute billions of SHA-256 hashes per second. A slow hash function (bcrypt, scrypt, Argon2) is designed to be computationally expensive, making brute-force attacks infeasible.

The algorithm comparison:

Algorithm	Hashes/second (GPU)	Recommended	Why
MD5	~180 billion	❌ Never	Broken, too fast
SHA-1	~6 billion	❌ Never	Broken, too fast
SHA-256	~2 billion	❌ Never	Too fast for passwords
bcrypt	~100,000	✅ Yes	Adaptive cost factor
scrypt	~10,000	✅ Yes	Memory-hard
Argon2	~1,000	✅ Yes (best)	Memory-hard, winner of Password Hashing Competition

Salting adds random data to each password before hashing, so two users with the same password have different hashes. This defeats rainbow tables (precomputed hash databases). Modern algorithms (bcrypt, Argon2) handle salting automatically.

The Password Generator creates strong random passwords with configurable length and character sets — useful for generating initial passwords, API keys, or test data. But remember: the generator creates the password; bcrypt/Argon2 protects it in storage.

The OWASP Top 10: Attacks Every Developer Should Prevent

The OWASP Top 10 lists the most critical web application security risks. The ones every developer should know how to prevent:

Broken Object-Level Authorization (BOLA)

The most common API vulnerability. If GET /users/42 returns user 42's data, but the server doesn't check that the authenticated user is user 42 (or an admin), then any authenticated user can access any other user's data by changing the ID in the URL. Every endpoint that takes an ID must check authorization.

Fix: Always verify that the authenticated user is authorized to access the specific resource they're requesting. Don't rely on clients to only request their own data.

Cross-Site Scripting (XSS)

An attacker injects malicious JavaScript into a page that other users view. The script runs in the victim's browser, stealing cookies, session tokens, or performing actions on their behalf. Three types:

Reflected XSS: the malicious script is part of the URL (e.g., ?search=<script>...</script>)
Stored XSS: the script is stored in the database (e.g., in a comment or post)
DOM-based XSS: the vulnerability is in client-side JavaScript that unsafely manipulates the DOM

Fix: Escape all user input before rendering it in HTML. Use a framework that auto-escapes by default (React, Vue, Angular). Set the Content-Security-Policy header to restrict which scripts can execute.

Cross-Site Request Forgery (CSRF)

An attacker tricks a user's browser into making a request to a site where the user is authenticated. If a bank transfer is triggered by POST /transfer?to=attacker&amount=1000, and the user visits a malicious page while logged in, the browser sends the request with the user's cookies — and the transfer happens.

Fix: Use CSRF tokens — a unique, unpredictable value embedded in every form and verified on the server. Modern frameworks (Rails, Django, Spring) include CSRF protection by default. For APIs, require a custom header (e.g., X-Requested-With) that cross-origin requests cannot set.

SQL Injection

An attacker injects SQL code through user input. If a query is built by string concatenation — SELECT * FROM users WHERE name = ' + userInput + ' — an attacker can input ' OR '1'='1 to bypass authentication, or '; DROP TABLE users; -- to destroy data.

Fix: Use parameterized queries (prepared statements) exclusively. Never concatenate user input into SQL. ORMs (Sequelize, SQLAlchemy, Entity Framework) use parameterized queries by default — but raw SQL queries are still vulnerable if not parameterized.

Security Misconfiguration

Default credentials, exposed error messages, unnecessary features enabled, missing security headers. The 2017 Equifax breach was caused by an unpatched Apache Struts vulnerability — a known exploit for which a patch had been available for months.

Fix: Harden your deployment: disable default accounts, remove unused features, keep dependencies updated, and include security headers in every response.

Security Headers: The Headers Every Response Should Include

HTTP security headers tell the browser how to handle your content. They're free protection — one line of configuration each:

Header	What it does	Recommended value
`Strict-Transport-Security`	Force HTTPS for all future requests	`max-age=31536000; includeSubDomains`
`X-Content-Type-Options`	Prevent MIME type sniffing	`nosniff`
`X-Frame-Options`	Prevent clickjacking (your page in an iframe)	`DENY` or `SAMEORIGIN`
`Content-Security-Policy`	Restrict which resources can load	`default-src 'self'; script-src 'self'`
`X-XSS-Protection`	Enable browser XSS filter (legacy)	`1; mode=block`
`Referrer-Policy`	Control how much referrer information is sent	`strict-origin-when-cross-origin`
`Cache-Control`	Prevent caching of sensitive data	`no-store` for authenticated responses

The most important: Strict-Transport-Security (HSTS) ensures that browsers always use HTTPS for your domain, preventing SSL stripping attacks. Once set, a user's browser will refuse to connect via HTTP for the duration of max-age.

Session Management: Stateful vs Stateless

Stateful sessions store session data on the server (in memory, Redis, or a database) and send the client a session ID cookie. The server looks up the session on every request. Pros: easy to invalidate, can store large data. Cons: requires shared session storage in a multi-server setup, memory overhead.

Stateless sessions (JWTs) encode all session data in the token itself. The server doesn't store anything. Pros: no shared storage needed, works across services. Cons: cannot invalidate individual tokens before expiration, token size grows with claims.

The hybrid approach (most common in production): use a short-lived JWT (15-60 minutes) for stateless authentication, plus a long-lived refresh token (stored server-side) to issue new JWTs. This gives you the scalability of stateless tokens with the ability to revoke access via the refresh token.

Practical Tips for Secure Development

Never roll your own crypto. Use established libraries (bcrypt, Argon2, libsodium). Cryptography is easy to get wrong in ways that are invisible until you're breached.
Use HTTPS everywhere. No exceptions. Even internal APIs should use TLS. Use Let's Encrypt for free certificates.
Validate input on the server. Client-side validation is for UX; server-side validation is for security. Never trust client input.
Keep dependencies updated. Use npm audit, Snyk, or Dependabot to flag known vulnerabilities in your dependencies.
Log security events. Failed logins, permission denied errors, and unusual patterns should be logged and monitored.
Use environment variables for secrets. Never commit API keys, database passwords, or JWT secrets to version control.
Implement rate limiting on authentication endpoints. Brute-force attacks try thousands of passwords per second. Rate limit to 5-10 attempts per minute per IP.

Limitations: What This Guide Doesn't Cover

This guide covers the fundamentals — the vulnerabilities that every developer should prevent. It does not cover:

Penetration testing — actively trying to break your own application
Threat modeling — systematically identifying attack vectors
Compliance — GDPR, HIPAA, PCI-DSS requirements
Infrastructure security — network segmentation, firewalls, intrusion detection
Cryptography engineering — designing custom protocols (don't)

For production applications handling sensitive data, hire a security professional. The fundamentals prevent the most common attacks, but determined attackers with resources require defense in depth.

Frequently Asked Questions

Q: What is the difference between authentication and authorization?

A: Authentication answers 'who are you?' (verifying identity). Authorization answers 'are you allowed to do this?' (checking permissions). A user can be authenticated (logged in) but not authorized (not an admin).

Q: What is a JWT?

A: A JSON Web Token is a self-contained token that encodes claims (user ID, permissions, expiration) in a JSON payload, signed by the server. JWTs are signed, not encrypted — never put secrets in them.

Q: How should I store passwords?

A: Hash them with a slow algorithm: bcrypt, scrypt, or Argon2. Never use fast hashes (MD5, SHA-1, SHA-256) for passwords. Never store plaintext.

Q: What is the most common web vulnerability?

A: Broken Object-Level Authorization (BOLA) — every endpoint that takes an ID must check that the authenticated user is authorized to access that specific resource.

Q: What is XSS?

A: Cross-Site Scripting — an attacker injects malicious JavaScript into a page that other users view. Fix by escaping all user input and setting Content-Security-Policy headers.

Q: What is CSRF?

A: Cross-Site Request Forgery — an attacker tricks a user's browser into making a request to a site where they're authenticated. Fix with CSRF tokens and SameSite cookies.

Q: What is SQL injection?

A: An attacker injects SQL code through user input. Fix by using parameterized queries (prepared statements) exclusively — never concatenate user input into SQL.

Q: What security headers should I include?

A: At minimum: Strict-Transport-Security (force HTTPS), X-Content-Type-Options: nosniff, X-Frame-Options: DENY, and Content-Security-Policy to restrict resource loading.

notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.

🔗 Original guide

A/B Testing & Experimentation: How to Run Tests That Actually Teach You Something

Adolfo Pedernera — Thu, 27 Aug 2026 04:48:07 +0000

Hypothesis testing, p-values, confidence intervals, statistical power, and the traps that fool even experienced teams.

The Tasting Illusion

In the 1970s, Pepsi ran a marketing campaign that seemed to prove something remarkable: in blind taste tests, more people preferred Pepsi over Coke. The campaign was a sensation — millions of consumers sipped from unmarked cups and pointed to the sweeter drink. Pepsi's stock rose. Coke panicked, reformulated, and launched "New Coke" in 1985, one of the most infamous product launches in history.

The problem wasn't the taste. It was the test. A single sip from a small cup measures immediate sweetness preference, not which beverage someone drinks for decades. The blind test had statistical significance — the difference was real and repeatable — but it lacked practical significance. It measured the wrong thing.

This is the central trap of A/B testing and experimentation: statistical significance does not mean business significance. A test can prove, with 99% confidence, that a button color change increases clicks by 0.3% — a finding that is simultaneously true and useless. This guide explains how to design and interpret experiments that avoid that trap: how to set up hypotheses, calculate whether your sample is large enough, interpret p-values without fooling yourself, and distinguish signals that matter from noise that merely looks convincing.

The Logic of Experimentation

Every controlled experiment follows the same skeleton, whether you're testing a new cancer drug, a website headline, or a pricing page. You start with a hypothesis — a specific, testable prediction. You split your subjects randomly into two groups: the control (which sees the current version) and the treatment (which sees the change). You measure an outcome in both groups. Then you ask a single question: is the difference I see larger than what random chance would typically produce?

That last step is where statistics enters. The logic is counterintuitive on purpose: you begin by assuming the boring explanation (your change did nothing) and then check whether the data is incompatible with that assumption. This starting assumption is called the null hypothesis , written $H_{0}$ . The alternative — your actual prediction — is the alternative hypothesis , written $H_{1}$ .

The engine that drives the answer is the p-value : the probability of observing a difference at least as large as the one you measured, assuming the null hypothesis is true.

p = P (observed difference ∣ H_{0} is true)

A p-value of 0.03 means: if your change actually did nothing, you would still see a difference this large 3% of the time by pure chance. It does NOT mean there is a 97% chance your change works. This misinterpretation is so common that the American Statistical Association published a formal statement warning against it.

The Two Mistakes You Can Make

Whenever you draw conclusions from an experiment, you can go wrong in two distinct ways:

Error	What happens	Probability	Cost
Type I (false positive)	You conclude the change works when it doesn't	p-value (α)	Ship a dud, waste engineering time
Type II (false negative)	You conclude the change does nothing when it actually works	β	Miss a real improvement

The p-value threshold you choose — the significance level , denoted α — is the Type I error rate you're willing to tolerate. The conventional 0.05 means "I accept a 5% chance of shipping something that does nothing." In high-stakes domains (medicine, aviation), researchers demand α = 0.001 or lower. In fast-moving consumer apps, teams sometimes accept 0.10 to move faster.

The Type II error rate, β, is the flip side. If β = 0.20, you have a 20% chance of missing a real effect. The complement, 1 − β, is called statistical power : the probability that your test will detect an effect if one truly exists. A test with 80% power and 5% significance is the standard minimum in most fields — and many published studies fall short of even that.

Power is the probability you will detect a real effect. Most teams ignore it — and run tests that were doomed to be inconclusive before the first user arrived.

Sample Size: The Number That Decides Everything

Here is the uncomfortable truth about A/B testing: a test with too few users cannot detect anything useful, no matter how good the change is. If you run a test on 50 users and see no significant result, you have learned nothing — the test may simply have been too small to detect the effect. This is an underpowered test, and underpowered tests produce more noise than signal.

The required sample size depends on four numbers:

Baseline conversion rate (e.g., 5% of visitors sign up)
Minimum detectable effect (e.g., you want to catch a 10% relative lift → from 5% to 5.5%)
Significance level α (typically 0.05)
Power 1 − β (typically 0.80)

The formula for a two-proportion z-test (the standard for conversion rate comparisons) is approximately:

n \approx \frac{( z _{α /2} + z _{β} ) ^{2} \cdot 2 p ( 1 - p )}{( p _{1} - p _{2} ) ^{2}}

where p is the pooled proportion, $p_{1} - p_{2}$ is the absolute effect size, and $z_{α /2}$ , $z_{β}$ are the critical z-values (1.96 and 0.84 for α=0.05, power=0.80).

For a baseline of 5% and a minimum detectable effect of 10% relative (0.5 percentage points absolute), this gives n ≈ 15,700 users per variation. For a 5% relative lift — a more realistic goal — you need roughly 10× more: ~157,000 users per variation. This is why small teams with modest traffic often conclude that A/B testing "doesn't work" — they were running experiments that needed ten times their monthly traffic to detect anything. The Sample Size Calculator computes this directly for your baseline and desired power, so you can plan your test duration before launching.

Users per variation needed for 80% power, α=0.05, two-sided test. Smaller effects and lower baselines explode the sample size.

Practical rule of thumb: If your site gets 1,000 visitors per day and you need 15,700 per variation, the test runs for ~31 days. If your test takes longer than 4 weeks, reconsider whether the minimum detectable effect is realistic — or whether A/B testing is the right tool for your traffic level.

Confidence Intervals: The Number That Tells the Whole Story

A p-value tells you whether the effect is "significant." A confidence interval tells you the range of plausible effect sizes — which is usually what you actually want to know.

A 95% confidence interval of [+0.2%, +1.8%] for a conversion rate lift means: we are 95% confident the true effect lies between 0.2 and 1.8 percentage points. Crucially, this interval does NOT include zero — which is why the p-value is below 0.05. But it also tells you something the p-value hides: the effect could be as small as 0.2%, which may not justify the engineering cost.

The formula for a confidence interval on the difference between two proportions:

(p_{1} - p_{2}) \pm z_{α /2} \frac{p _{1} ( 1 - p _{1} )}{n _{1}} + \frac{p _{2} ( 1 - p _{2} )}{n _{2}}

When the confidence interval includes zero, the result is not statistically significant. When it does not include zero, you have a significant result — and the interval's width tells you how precisely you've estimated the effect. A wide interval ([−0.1%, +2.1%]) means "something is happening, but we're not sure exactly what." A narrow interval [+0.8%, +1.2%] means "the effect is almost certainly in this tight range". If you have the observed counts from a finished test, the Confidence Interval Calculator returns the interval directly — useful for reporting results to stakeholders who want to know the range, not just the p-value.

Use the confidence interval as your primary result, not the p-value. It answers the business question ("how much lift can I expect?") rather than the statistical one ("can I reject the null?").

Chi-Square and t-Tests: Choosing the Right Tool

Not all experiments compare conversion rates. Sometimes you're comparing averages (revenue per user, time on page, order value), and sometimes you're comparing distributions across categories (did the change shift users between plan tiers?). The right test depends on the data type.

For conversion rates and proportions (click/no click, signup/no signup): use a chi-square test or a two-proportion z-test. The chi-square test checks whether the observed frequencies in your 2×2 table (control vs treatment × converted vs not) differ from what the null hypothesis predicts.

χ^{2} = \sum \frac{( O _{i} - E _{i} ) ^{2}}{E _{i}}

where $O_{i}$ is the observed count and $E_{i}$ is the expected count under the null. The result is compared to a chi-square distribution with 1 degree of freedom to obtain a p-value. The Chi-Square Calculator does this computation from your observed counts, returning both the statistic and the p-value — useful when you have the raw numbers from a finished test and want to verify significance without building the table by hand.

For continuous outcomes (revenue, time, score): use a t-test. The t-test compares the means of two groups while accounting for the spread (standard deviation) within each groups:

t = \frac{x ˉ _{1} - x ˉ _{2}}{\frac{s _{12}}{n _{1}} + \frac{s _{22}}{n _{2}}}

where $x ˉ$ is the sample mean, $s$ is the standard deviation, and $n$ is the sample size. The denominator is the standard error of the difference — a quantity that combines the uncertainty from both groups. The t-Test Calculator computes this from your two samples, handling both the equal-variance and unequal-variance (Welch's) cases.

The standard deviation matters enormously here. If revenue per user swings wildly (high variance), you need far more users to detect the same mean difference than if revenue is consistent. This is why the confidence interval on the mean difference is so useful: it shows whether the signal (the mean difference) is large relative to the noise (the standard deviation).

Practical decision rule:

Binary outcome (clicked/didn't, signed up/didn't) → chi-square or z-test for proportions
Continuous outcome (revenue, time, score) → t-test
Comparing more than two variants → ANOVA (not covered here)

The Traps That Fool Experienced Teams

Peeking at Results

The p-value formula assumes you look at the data exactly once, at a predetermined sample size. If you peek at your test on day 3, day 7, day 10 — and stop as soon as p < 0.05 — you are dramatically inflating your false positive rate. With 5 peeks, the true false positive rate is not 5% but roughly 14%.

Fix: Decide your sample size in advance, run the test to that sample size, then analyze once. If you must peek, use sequential testing methods (always-valid p-values, alpha spending) that adjust the threshold for each peek — but these require roughly 30% more users for the same power.

P-Hacking (Multiple Comparisons)

If you test 20 different metrics (conversion rate, revenue, time on page, scroll depth, bounce rate...) and report the one that came back significant at p < 0.05, you are p-hacking. With 20 independent tests at α = 0.05, the probability of at least one false positive is 1 − (1−0.05)^20 ≈ 64%.

Fix: Pre-specify a single primary metric before the test begins. Treat everything else as exploratory. If you must test multiple metrics, apply a Bonferroni correction: divide α by the number of tests (so 0.05 / 20 = 0.0025 for 20 metrics).

Simpson's Paradox

A treatment can win in every subgroup (mobile users, desktop users, new visitors, returning visitors) and lose in aggregate — or vice versa. This happens when the subgroups have very different sizes and very different baseline rates.

Fix: Always segment your results. If the treatment effect reverses between mobile and desktop, the aggregate number is misleading. Report the per-segment results alongside the overall.

Novelty and Primacy Effects

A new feature may initially perform worse (users are confused) or better (users are curious) than it will long-term. A 2-week test captures the novelty effect, not the steady-state effect.

Fix: Run tests for at least 2 full weeks (to capture day-of-week effects) and ideally 4 weeks for features with learning curves. Be skeptical of results that appear in week 1 and vanish in week 2.

A Worked Example: Button Color

A team tests whether a green signup button outperforms the current blue one. Baseline conversion rate: 4.0%. They want to detect a 10% relative lift (to 4.4%) with 80% power and α = 0.05.

Step 1 — sample size. Using the formula above with p₁ = 0.04, p₂ = 0.044, α = 0.05, power = 0.80: n ≈ 39,000 users per variation. With 20,000 daily visitors split 50/50, the test runs for ~4 days.

Step 2 — run the test. Control (blue): 780 conversions out of 39,000 (2.00%). Treatment (green): 936 conversions out of 39,000 (2.40%).

Step 3 — chi-square test. The observed counts form a 2×2 table. The chi-square statistic is 15.2, which gives p ≈ 0.0001 — well below 0.05. The result is statistically significant.

Step 4 — confidence interval. The difference is 0.40 percentage points, with a 95% CI of [+0.22%, +0.58%]. This interval excludes zero (confirming significance) and tells the business: "the lift is somewhere between 0.22 and 0.58 percentage points, or 5.5% to 14.5% relative."

Step 5 — decision. Is a 0.4 percentage point lift worth the engineering cost and the design inconsistency? That is a business question, not a statistical one. The statistics only say: the lift is real and almost certainly between 0.22% and 0.58%.

When A/B Testing Is the Wrong Tool

A/B testing is powerful, but it is not always the right method. Consider alternatives when:

Your traffic is too low. If you need 100,000 users per variation and get 5,000 per month, a properly powered test would take years. Use qualitative research, user interviews, or heuristic evaluation instead.
You're testing a fundamental redesign. A/B testing measures incremental changes. A complete redesign has too many variables changing at once to attribute causality. Use before/after analysis with caution.
Network effects exist. If users in the treatment group interact with users in the control group (social networks, marketplaces), the groups are no longer independent. The test is invalid.
You need to understand WHY. A/B testing tells you whether something works, not why. Pair experiments with qualitative research (session recordings, user interviews) to understand the mechanism.

Practical Tips for Running Experiments

Pre-register your test. Write down the hypothesis, primary metric, sample size, and analysis plan before collecting data. This prevents post-hoc rationalization.
Run an A/A test occasionally. Show both groups the same experience. If you get a "significant" result, your testing infrastructure is broken.
Segment after, not during. Pre-specify one primary metric. Segment (by device, by region, by new/returning) only as exploratory analysis.
Track guardrail metrics. A treatment might increase clicks but hurt revenue. Define guardrail metrics (revenue per user, unsubscribe rate, page load time) that must not degrade.
Account for multiple variants. If you run an A/B/C/D test, apply a Bonferroni correction (α / number of comparisons) or use a false discovery rate method.
Report confidence intervals, not just p-values."Lift of 0.4 pp, 95% CI [+0.22, +0.58]" is more actionable than "p = 0.0001."

Limitations and Edge Cases

A/B testing has honest limitations. It is not good at detecting small effects unless you have enormous traffic. It assumes random assignment — if your randomization is broken (e.g., more mobile users end up in treatment by chance), the results are biased. It measures short-term effects — a headline that clicks well today may fatigue users after a month. And it cannot detect interactions unless you specifically design for them (a treatment may help new users but hurt returning users, which cancels out in aggregate).

Finally, A/B testing is a tool for optimization, not for strategy. It can tell you which of two headlines works better, but it cannot tell you what your content strategy should be. Use it to refine, not to think.

Frequently Asked Questions

Q: What is a p-value?

A: The probability of observing a result at least as extreme as yours, assuming the null hypothesis (that your change did nothing) is true. A p-value of 0.05 means: if your change did nothing, you would still see a result this large 5% of the time by chance. It is NOT the probability that your change works.

Q: What is the difference between statistical significance and practical significance?

A: Statistical significance means the observed difference is unlikely due to chance (low p-value). Practical significance means the difference is large enough to matter for your business. A result can be statistically significant (real) but practically insignificant (too small to care about).

Q: How many users do I need for an A/B test?

A: It depends on your baseline rate and the minimum effect you want to detect. For a 5% baseline and a 10% relative lift (to 5.5%), you need roughly 15,700 users per variation for 80% power at α=0.05. Smaller effects require dramatically more users.

Q: What is statistical power?

A: The probability that your test will detect a real effect if one exists. Typically set to 80% (β=0.20). An underpowered test — one with too few users — will often conclude 'no significant difference' even when a real effect exists.

Q: Why shouldn't I peek at my test results every day?

A: Because each peek inflates your false positive rate. With 5 peeks at α=0.05, your true false positive rate is roughly 14%. Decide your sample size in advance and analyze once, or use sequential testing methods that adjust for peeking.

Q: What is the difference between a t-test and a chi-square test?

A: Use a t-test when comparing means of continuous outcomes (revenue, time, score). Use a chi-square test when comparing proportions or frequencies (conversion rates, click/no-click). The t-test assumes continuous data; the chi-square test works with counts in categories.

Q: What is a confidence interval?

A: A range of plausible values for the true effect. A 95% confidence interval of [+0.2%, +1.8%] means we are 95% confident the true lift is between 0.2 and 1.8 percentage points. If the interval excludes zero, the result is statistically significant.

Q: What is Simpson's Paradox?

A: A phenomenon where a trend appears in every subgroup but reverses in aggregate (or vice versa). It happens when subgroups have very different sizes and baselines. Always segment your results — the aggregate can hide or reverse real effects.

notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.

🔗 Original guide

Statistics for Data-Driven Developers: Making Decisions With Numbers

Adolfo Pedernera — Wed, 26 Aug 2026 16:37:47 +0000

Descriptive stats, probability distributions, confidence intervals, hypothesis testing, and the mistakes that fool experienced engineers.

The Alert That Cried Wolf

A team at a mid-size SaaS company set up monitoring for their API response times. They configured an alert: "Trigger if average response time exceeds 500 ms over a 5-minute window." The alert fired constantly. Engineers sprinted to investigate, found nothing wrong, and started ignoring it. Two weeks later, a genuine outage went undetected for 20 minutes because everyone had learned to dismiss the alert.

The problem was the word "average." The average response time was 350 ms, but the distribution was heavily skewed: 95% of requests completed in under 200 ms, while 5% took over 2 seconds (timeouts and retries). The average hid the tail — the 5% of users having a terrible experience. When the team switched to alerting on the 95th percentile (p95) of response time instead of the mean, the false alarms stopped. The p95 was 450 ms — still healthy — and the tail latency that had been dragging up the average became visible in a separate dashboard.

This is the fundamental lesson of statistics for developers: the summary statistic you choose determines the story you see. Mean, median, and percentile each reveal different truths. Standard deviation tells you whether your system is consistent or erratic. Confidence intervals tell you whether a change is real or noise. Hypothesis testing tells you whether your "improvement" actually improved anything or whether you're just seeing random variation.

This guide explains the statistics that matter for developers who work with data — whether that data is response times, conversion rates, error rates, or business metrics. It covers descriptive statistics (how to summarize data), probability (how to model uncertainty), and inferential statistics (how to draw conclusions from samples) — with the goal of making better decisions and avoiding the traps that fool even experienced engineers.

Descriptive Statistics: Summarizing Without Lying

Descriptive statistics compress a dataset into a few numbers. The art is choosing numbers that tell the truth — and not choosing numbers that hide it.

Central Tendency: Mean, Median, Mode

The mean (average) is the most common summary — and the most misleading when data is skewed. If 9 users pay $10/month and 1 user pays $1,000/month, the mean revenue per user is $109, which describes no one's actual experience.

The median (middle value) is robust to outliers. In the same dataset, the median is $10 — a more honest summary of the typical user. Use median when data is skewed (revenue, response times, house prices) and mean when data is symmetric (heights, test scores, measurement errors).

The mode (most frequent value) matters for categorical data: the most common error code, the most popular feature, the most frequent user country.

Spread: Variance and Standard Deviation

Central tendency tells you where the middle is. Spread tells you how far the data ranges around that middle. Two APIs can have the same mean response time (350 ms) but wildly different consistency: one with σ = 50 ms (predictable) and one with σ = 400 ms (erratic).

Variance is the average squared deviation from the mean:

s^{2} = \frac{\sum ( x _{i} - x ˉ ) ^{2}}{n - 1}

Standard deviation is the square root of variance, bringing the units back to the original data:

s = s^{2}

For the normal distribution, roughly 68% of values fall within ±1 standard deviation of the mean, 95% within ±2, and 99.7% within ±3 — the empirical rule. The Standard Deviation Calculator and Variance Calculator compute these from your raw data, including the crucial sample-vs-population distinction (divide by n-1 for samples, n for populations).

Percentiles: The Tail Tells the Story

For skewed data (response times, revenue, file sizes), percentiles are more honest than mean + standard deviation. The p50 is the median. The p95 is the value below which 95% of observations fall. The p99 captures the worst 1%.

Why percentiles matter for monitoring:

Mean hides the tail (the alert that cried wolf)
p95 captures the experience of the slowest 5% without being dominated by outliers
p99 captures the worst 1% — the users who are most likely to churn

Percentiles are not additive. The p95 of a sum is not the sum of the p95s. If API A has p95 = 200 ms and API B has p95 = 300 ms, the end-to-end p95 of calling both is not 500 ms — it's typically less, because the slowest 5% of A's calls don't always coincide with the slowest 5% of B's calls. This is why distributed tracing matters: you need to measure the end-to-end percentile, not sum individual ones.

Response time percentiles for a typical API. The median (180ms) looks healthy, but p99 reveals 1% of users wait over 1 second. The mean (350ms) is pulled up by the tail.

Probability: Modeling Uncertainty

Probability is the language of uncertainty. For developers, it appears in reliability engineering (what's the chance of outage?), capacity planning (what's the chance we exceed capacity?), and A/B testing (what's the chance this result is real?).

Distributions: The Shapes of Data

Every dataset has a shape — a distribution. The most important distributions for developers:

Normal (Gaussian) distribution is the bell curve: symmetric, mean = median = mode. Heights, measurement errors, and sums of many independent variables tend to be normal (thanks to the Central Limit Theorem, discussed below).

Log-normal distribution is skewed right: the log of the values is normal. Response times, file sizes, and revenue per user tend to be log-normal. The mean is much higher than the median. Use geometric mean or percentiles, not arithmetic mean.

Exponential distribution models time between events: time between requests, time between failures. It has a "memoryless" property: the probability of a failure in the next minute is the same regardless of how long it's been since the last failure.

Poisson distribution models counts of events in a fixed interval: requests per second, errors per hour, support tickets per day. If events arrive independently at rate λ, the count in any interval is Poisson-distributed with mean λ.

The Central Limit Theorem

The Central Limit Theorem (CLT) is one of the most powerful results in statistics: if you take sufficiently large random samples from ANY distribution (normal, skewed, it doesn't matter), the distribution of the sample means will be approximately normal.

This is why the normal distribution appears everywhere: any metric that's the sum or average of many independent contributions (response times averaged over a window, conversion rates averaged over days) tends toward normal, even if the underlying data is wildly skewed. The CLT is also why you can use normal-based methods (z-tests, t-tests) even when your data isn't normal — as long as your sample size is large enough (typically n > 30).

Correlation: The Number That Gets Misused

Correlation measures the linear relationship between two variables, ranging from -1 (perfect inverse) to +1 (perfect direct). A correlation of 0 means no linear relationship.

The Pearson correlation coefficient:

r = \frac{\sum ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum ( x _{i} - x ˉ ) ^{2} \sum ( y _{i} - y ˉ ) ^{2}}

Correlation ≠ causation is the most violated principle in data analysis. Ice cream sales and drowning deaths are correlated — not because ice cream causes drowning, but because both increase in summer (a confounding variable ). In software: server load and error rates are correlated, but the cause is a third factor (a deployment that introduced both a memory leak and increased CPU usage).

Spurious correlations appear by chance when you test enough pairs. If you check 100 unrelated metrics, about 5 will show "significant" correlation at p < 0.05 purely by chance. This is why pre-specifying hypotheses matters — and why data mining without theory produces nonsense.

Inferential Statistics: Drawing Conclusions from Samples

Descriptive statistics summarize what you have. Inferential statistics let you draw conclusions about a population from a sample — which is essential when you can't measure everything.

Confidence Intervals: The Range of Plausible Values

A confidence interval is a range that likely contains the true population parameter. A 95% confidence interval of [2.1%, 2.5%] for a conversion rate means: "if we repeated this experiment many times, 95% of the intervals we'd compute would contain the true conversion rate".

The formula for a confidence interval on a mean:

x ˉ \pm z_{α /2} \frac{s}{n}

The key insight: the width shrinks with $\frac{1}{n}$ — to halve the margin of error, you need 4× more data. The Confidence Interval Calculator computes this from your sample mean, standard deviation, and sample size, for any confidence level.

Why confidence intervals beat point estimates. Reporting "conversion rate is 2.3%" hides the uncertainty. Reporting "conversion rate is 2.3% ± 0.2% (95% CI)" tells you the precision of your estimate. If the CI for the difference between two variants includes zero, the difference is not statistically significant.

Hypothesis Testing: Is This Real or Noise?

Hypothesis testing is the formal framework for deciding whether an observed effect is real or due to chance. The logic: assume the boring explanation (null hypothesis), compute the probability of seeing your data under that assumption (p-value), and reject the null if that probability is low enough.

The framework:

Null hypothesis (H₀): the default assumption (e.g., "the new feature does nothing")
Alternative hypothesis (H₁): what you want to prove (e.g., "the new feature increases conversion")
Test statistic: a number computed from your data (t-statistic, chi-square statistic, z-score)
p-value: the probability of seeing a test statistic at least as extreme as yours, assuming H₀ is true
Decision: if p < α (typically 0.05), reject H₀

The p-value is not the probability that H₀ is true. It is the probability of the data given H₀ — not the probability of H₀ given the data. This confusion is so pervasive that the American Statistical Association published a formal statement clarifying it.

The t-Test Calculator and Chi-Square Calculator compute test statistics and p-values for the two most common tests: t-test for comparing means (revenue, time, score) and chi-square for comparing proportions (conversion rates, click/no-click).

Statistical Power: The Probability You'll Detect a Real Effect

Statistical power is the probability that your test will detect an effect if one truly exists. A test with 80% power has a 20% chance of missing a real effect (Type II error).

Power depends on three things:

Effect size: larger effects are easier to detect
Sample size: more data → more power
Significance level: lower α → less power (stricter threshold)

The Sample Size Calculator computes how many observations you need for a desired power, given your expected effect size and α. Use it before running any experiment — an underpowered test is worse than no test at all, because it produces false confidence in a "no difference" result.

Practical Applications for Developers

Monitoring and Alerting

The most common statistical mistake in monitoring is alerting on the mean. For skewed distributions (response times, queue lengths, file sizes), the mean hides the tail. Alert on percentiles instead:

p95 for "most users are having a good experience"
p99 for "the worst 1% are having a bad experience"
Mean only for symmetric metrics (CPU utilization, memory usage).

Set thresholds using baselines, not absolutes. A response time of 500 ms might be normal for a database query but catastrophic for a cache lookup. Establish per-endpoint baselines using historical percentiles, and alert on deviations from the baseline (e.g., "p95 is 3× higher than the 7-day rolling average").

A/B Testing and Experimentation

A/B testing is hypothesis testing applied to product decisions. The same principles apply: pre-specify your primary metric, compute sample size in advance, run the test to completion, and report confidence intervals — not just p-values.

Common mistakes:

Peeking: checking results daily and stopping when p < 0.05 inflates false positives
Multiple comparisons: testing 20 metrics and reporting the one that's significant is p-hacking
Underpowered tests: running on too few users produces inconclusive results
Ignoring practical significance: a 0.1% lift may be statistically significant with 1M users but not worth shipping

Data-Driven Development

"Data-driven" means letting evidence guide decisions — but evidence can be misleading if you don't understand the statistics. Before trusting any metric:

Check the distribution. Is it normal or skewed? Use mean for normal, median/percentiles for skewed.
Look at the spread. A mean of 350 ms with σ = 50 ms is very different from a mean of 350 ms with σ = 400 ms.
Compute confidence intervals. A conversion rate of "2.3%" is less useful than "2.3% ± 0.2%."
Beware of confounding. Correlation is not causation. A metric may move because of a third factor you're not measuring.
Pre-specify hypotheses. Decide what you're testing before you look at the data. Post-hoc rationalization produces false discoveries.

Capacity Planning and Forecasting

Statistics also powers capacity planning: predicting how much infrastructure you'll need. If your traffic grows 10% per month, you can extrapolate when you'll need to scale. But extrapolation requires understanding variance — if traffic is highly variable (high standard deviation), you need more headroom than the average suggests.

The rule of thumb: plan for the p95 of expected traffic, not the mean. If your mean projected traffic is 10,000 requests per second, but the 95th percentile is 15,000, provision for 15,000. Otherwise, you'll be overloaded 5% of the time — which is roughly 3.6 hours per day.

Regression for forecasting. Simple linear regression fits a line to historical data (traffic over time) and extrapolates. The regression calculator on this site fits a line to your data and reports the slope (growth rate) and R² (how well the line fits). Use it to answer: "At current growth rate, when will we hit our limit?" But be cautious — regression assumes the future resembles the past. A viral post, a product launch, or a pandemic breaks that assumption.

Error Budgets and Reliability

Site Reliability Engineering (SRE) uses statistics to define error budgets. If your SLO (Service Level Objective) is 99.9% uptime, your error budget is 0.1% — roughly 43 minutes of downtime per month. Statistics tells you whether you're on track to meet that budget or whether you've already blown it.

The math: if you've had 30 minutes of downtime in the first 15 days of a 30-day month, you're on pace for 60 minutes — double your budget. But if your downtime events are random (Poisson-distributed), the variance matters. A month with 30 minutes of downtime could be normal variation or the start of a trend. Statistical process control charts (which plot metrics with control limits at ±3σ) distinguish between normal variation and a genuine shift.

Common Mistakes That Fool Experienced Engineers

Confusing correlation with causation. Two metrics moving together doesn't mean one causes the other. Look for confounding variables.
Ignoring sample size. A 5% conversion rate from 20 users (1 conversion) is not a reliable estimate. Compute confidence intervals.
Using mean for skewed data. Response times, revenue, and file sizes are almost always skewed. Use percentiles.
Not checking assumptions. t-tests assume approximately normal data (or large samples). Chi-square tests assume sufficient expected counts. Violating assumptions produces unreliable p-values.
Treating p < 0.05 as truth. A p-value of 0.04 means there's a 4% chance of this result under the null — not that the alternative is true. Replicate findings before acting on them.
Forgetting about practical significance. With enough data, any tiny effect becomes statistically significant. Ask: "Is this effect large enough to matter?"

Frequently Asked Questions

Q: What is the difference between mean and median?

A: Mean is the average (sum divided by count). Median is the middle value when data is sorted. Mean is sensitive to outliers; median is robust. Use mean for symmetric data, median for skewed data (response times, revenue).

Q: What is standard deviation?

A: A measure of spread: how far data typically sits from the mean. A small standard deviation means data is tightly clustered; a large one means data is widely scattered. For normal data, 68% falls within ±1σ, 95% within ±2σ.

Q: What is a percentile?

A: The value below which a given percentage of observations fall. p50 is the median. p95 is the value below which 95% of observations fall. Percentiles are robust to outliers and ideal for skewed data like response times.

Q: What is a confidence interval?

A: A range that likely contains the true population parameter. A 95% CI of [2.1%, 2.5%] means: if we repeated this experiment many times, 95% of the intervals would contain the true value. If the CI for a difference includes zero, the result is not statistically significant.

Q: What is a p-value?

A: The probability of observing a result at least as extreme as yours, assuming the null hypothesis (no effect) is true. A p-value of 0.05 means: if there were no real effect, you'd see a result this large 5% of the time by chance. It is NOT the probability that your hypothesis is true.

Q: What is statistical power?

Q: What is the Central Limit Theorem?

A: The principle that the distribution of sample means approaches a normal distribution as sample size increases, regardless of the underlying data distribution. This is why normal-based methods work even with non-normal data — as long as your sample is large enough (typically n > 30).

Q: What is the difference between correlation and causation?

A: Correlation means two variables move together. Causation means one causes the other. Correlation does not imply causation — both variables may be influenced by a third confounding factor. Ice cream sales and drowning are correlated (both increase in summer) but not causally related.

notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.

🔗 Original guide

API Design & Rate Limiting: Building APIs That Scale Without Breaking

Adolfo Pedernera — Wed, 26 Aug 2026 14:34:20 +0000

How to design REST APIs that scale: HTTP methods, status codes, pagination, rate limiting (token bucket, sliding window), JWT auth, and security best practices.

The API That Became a Weapon

In 2021, a developer at a major cloud provider wrote a blog post about an outage that cost the company an estimated $10 million in lost revenue. The root cause wasn't a hardware failure or a DDoS attack. It was a single internal API that was designed without rate limits. A misconfigured internal service began calling the API in a retry loop — 10,000 requests per second, each timing out and retrying immediately. Within minutes, the API was down. Cascading failures took out three dependent services. By the time engineers identified the source, customers had been unable to access their data for 4 hours.

The API had no rate limiting, no circuit breaker, and no Retry-After header to tell the client to back off. It was, in retrospect, an accident waiting to happen — and it happened.

This guide explains how to design APIs that scale gracefully: how to use HTTP methods and status codes correctly, how to implement rate limiting that protects your infrastructure without punishing legitimate users, how to handle authentication with JWTs, and how to avoid the security mistakes that turn APIs into attack surfaces. Whether you're building your first REST API or hardening a production service that handles millions of requests, the same principles apply.

REST API Design: Resources, Not Actions

The most common mistake in API design is treating endpoints like function calls. A poorly designed API looks like a list of actions: /getUser, /createOrder, /deleteProduct. A well-designed API looks like a map of resources: /users, /orders, /products. The difference matters because HTTP already provides the verbs.

REST (Representational State Transfer) models your application as a set of resources identified by URLs, and uses HTTP methods to perform operations on those resources. The four methods you'll use 95% of the time:

Method	Purpose	Idempotent	Example
`GET`	Retrieve a resource	Yes	`GET /users/42` returns user 42
`POST`	Create a new resource	No	`POST /users` creates a new user
`PUT`	Replace a resource entirely	Yes	`PUT /users/42` replaces user 42
`PATCH`	Update part of a resource	No	`PATCH /users/42` updates some fields
`DELETE`	Remove a resource	Yes	`DELETE /users/42` removes user 42

Idempotency means making the same request multiple times produces the same result as making it once. GET, PUT, and DELETE are idempotent: deleting user 42 five times leaves the database in the same state as deleting it once. POST is not: five POST /users requests create five different users.

This matters for reliability. Networks fail. Clients timeout. If a POST request times out, the client doesn't know whether the server processed it — retrying might create a duplicate. The fix is an idempotency key : a unique client-generated token (a UUID works well) sent in a header like Idempotency-Key: <uuid>. The server stores the key with the result and returns the cached result for duplicate keys.

URL design guidelines:

Use nouns, not verbs: /users not /getUsers
Use plurals for collections: /orders not /order
Nest for relationships: /users/42/orders for user 42's orders
Use query parameters for filtering: /orders?status=active&limit=20
Keep URLs lowercase with hyphens: /order-items not /orderItems

HTTP Status Codes: The Language Clients Speak

HTTP status codes are how your API communicates what happened. Using them correctly means clients can respond appropriately without parsing response bodies. The categories:

Range	Category	When to use
2xx	Success	The request succeeded
3xx	Redirection	The resource moved (rare in APIs)
4xx	Client error	The request was malformed or unauthorized
5xx	Server error	Something went wrong on your end

The codes you'll use most often:

200 OK — standard success for GET, PUT, PATCH, DELETE
201 Created — POST that created a new resource (include a Location header pointing to the new resource)
204 No Content — success with no response body (common for DELETE)
400 Bad Request — the request was malformed (missing required field, invalid type)
401 Unauthorized — authentication failed (missing or invalid credentials)
403 Forbidden — authenticated but not allowed to access this resource
404 Not Found — the resource doesn't exist
409 Conflict — the request conflicts with the current state (duplicate email, version mismatch)
422 Unprocessable Entity — the request was well-formed but semantically invalid (validation errors)
429 Too Many Requests — rate limit exceeded (include Retry-After header)
500 Internal Server Error — something went wrong on your end (don't expose details)

The 401 vs 403 distinction trips up many developers. 401 means "I don't know who you are" (authentication). 403 means "I know who you are, but you can't do this" (authorization). A user without a valid token gets 401. A regular user trying to access an admin endpoint gets 403.

429 Too Many Requests deserves its own section — it's the rate limit response, and how you implement it determines whether your API survives a traffic spike or a misbehaving client.

Rate Limiting: Protecting Your Infrastructure

Rate limiting is the practice of controlling how many requests a client can make in a given time window. It protects your API from abuse, prevents a single tenant from monopolizing resources, and ensures fair access across all users.

The challenge is doing this without frustrating legitimate users. A developer integrating with your API will hit rate limits during testing. A batch job running at midnight will hit rate limits. A user with a flaky network that retries failed requests will hit rate limits. Good rate limiting distinguishes between a client that's slightly over its limit and one that's attacking your API.

The Three Algorithms

Fixed window is the simplest approach: allow N requests per window (e.g., 1,000 per hour), reset the counter at the start of each window. The implementation is trivial — a counter per client that resets on a timer:

counter[client_id] += 1
if counter[client_id] > LIMIT:
    return 429

The weakness is the edge effect : a client that sends 1,000 requests at 10:59 and another 1,000 at 11:01 has sent 2,000 requests in 2 minutes, despite the "1,000 per hour" limit. The fixed window doesn't see that the two bursts overlap.

Sliding window log stores every request timestamp for every client and counts how many fall within the current window. It's precise but expensive: storing millions of timestamps per client is memory-intensive. A common optimization is the sliding window counter , which combines the current window's count with the previous window's count, weighted by how far into the current window you are:

co u n t = c u rre n t_{c} o u n t \times (1 - \frac{t im e _{i} n t o _{w} in d o w}{w in d o w _{s} i ze}) + p re v i o u s_{c} o u n t

This approximates the sliding window without storing every timestamp, at the cost of allowing slightly more than the limit at window boundaries.

Token bucket is the most flexible algorithm and the one used by most production APIs (including GitHub and Stripe). Imagine a bucket that holds a maximum of B tokens. Tokens are added at a constant rate r per second (e.g., 10 tokens per second). Each request consumes one token. If the bucket is empty, the request is rejected with 429.

The token bucket handles bursts naturally: a client that's been idle for a minute has accumulated 60 tokens and can burst to 60 requests immediately. But sustained traffic cannot exceed the refill rate. The algorithm is:

t o k e n s = min (B, t o k e n s + r \times Δ t) - 1

If tokens >= 0, the request is allowed. If tokens < 0, it's rejected.

The API Rate Limit & Cost Calculator computes utilization, throttled requests, and monthly cost for any of these algorithms given your limit, load, and price per request. Use it to estimate whether your current limits can handle a 10× traffic spike, or to plan the cost of a rate-limited batch job.

Communicating Limits to Clients

A rate limit without communication is just an error. Good APIs tell clients three things in every response:

What the limit is : X-RateLimit-Limit: 1000
How many requests remain : X-RateLimit-Remaining: 847
When the limit resets : X-RateLimit-Reset: 1687305600 (Unix timestamp)

When the limit is exceeded, return 429 Too Many Requests with a Retry-After header that tells the client how long to wait before retrying:

HTTP/1.1 429 Too Many Requests
Retry-After: 60
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1687305600

{
  "error": "rate_limit_exceeded",
  "message": "You have exceeded the rate limit of 1000 requests per hour.",
  "retry_after": 60
}

The Retry-After header is in seconds (or an HTTP date). A well-behaved client reads this header and waits before retrying — instead of retrying immediately and making the problem worse.

Practical implementation: Most production APIs implement rate limiting at the API gateway or load balancer level (Kong, NGINX, AWS API Gateway, Cloudflare) rather than in application code. This offloads the work and applies limits before requests reach your application servers.

Authentication: Knowing Who's Calling

Every API needs to answer two questions: who is this? (authentication) and are they allowed to do this? (authorization). The most common approaches:

API Keys

The simplest approach: give each client a unique key, and require it in every request (usually via header: Authorization: Bearer <key> or X-API-Key: <key>). The server looks up the key to identify the client and check permissions.

API keys are easy to implement but have a weakness: they don't expire. If a key is leaked (committed to a GitHub repo, logged in an error message, intercepted over an unencrypted connection), anyone who has it can impersonate that client. Mitigations: require HTTPS, allow clients to rotate keys, and set expiration dates on keys.

JWT (JSON Web Tokens)

A JWT is a self-contained token that encodes claims (user ID, permissions, expiration) in a JSON payload, signed by the server. The client sends the token in the Authorization: Bearer <token> header. The server verifies the signature and extracts the claims — no database lookup required.

A JWT has three parts, each Base64URL-encoded and separated by dots:

eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjo0MiwiZXhwIjoxNjg3MzA1NjAwfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

Header (eyJhbGciOiJIUzI1NiJ9): algorithm and token type
Payload (eyJ1c2VyX2lkIjo0MiwiZXhwIjoxNjg3MzA1NjAwfQ): claims — user_id, expiration, roles
Signature (SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c): HMAC of header+payload with server secret

The critical claim is exp (expiration). A JWT without an expiration is valid forever — if leaked, it's a permanent access token. Set short expirations (15 minutes to 1 hour) and use refresh tokens for longer sessions.

JWT ≠ encryption. The payload is signed, not encrypted — anyone who intercepts the token can read the claims (that's why you should never put secrets in a JWT). The signature only guarantees that the token was issued by someone with the server secret and hasn't been tampered with. The JWT Decoder parses tokens client-side, showing the header, payload, and signature — useful for debugging authentication flows.

OAuth 2.0

OAuth 2.0 is a delegation protocol: it lets a user grant a third-party application limited access to their resources without sharing their password. The flow: the user clicks "Log in with Google," Google asks for permission, and issues an access token that the third-party app can use on the user's behalf.

For machine-to-machine APIs (where there's no user to click "Log in"), use the client credentials flow: the client exchanges its client ID and secret for an access token. This is what backend services use to communicate with each other.

Security: The Mistakes That Get You Hacked

APIs are the most common attack surface for web applications. The OWASP API Security Top 10 lists the most critical risks. The ones every developer should know:

Broken Object-Level Authorization (BOLA)

Excessive Data Exposure

APIs often return more data than the client needs. If GET /users/42 returns the user's email, phone, address, and hashed password, the client may only display the name — but the sensitive data is now in the browser's network tab, in logs, and in any proxy between client and server. Return only the fields the client needs, and use different response schemas for different roles.

Lack of Resources & Rate Limiting

Without rate limits, an attacker can brute-force passwords, scrape data, or overwhelm your API with requests. This is the failure mode that caused the $10 million outage described earlier. Rate limit authentication endpoints aggressively (5 attempts per minute per IP) and API endpoints per-client.

Mass Assignment

If your API accepts a JSON body and binds it directly to a database model without filtering, an attacker can set fields they shouldn't control. A PATCH /users/42 endpoint that accepts { "role": "admin" } because the request body is bound directly to the model is a mass assignment vulnerability. Whitelist the fields that can be updated.

Security Headers

Every API response should include:

Strict-Transport-Security: max-age=31536000 — force HTTPS
X-Content-Type-Options: nosniff — prevent MIME sniffing
Cache-Control: no-store — don't cache responses with sensitive data

Practical Tips for API Development

Always version your API. Use URL versioning (/v1/users) or header versioning (Accept: application/vnd.api.v1+json). Never release a breaking change without a version bump.
Use pagination for collections. Returning 100,000 records in one response is a memory bomb. Use cursor-based pagination (?cursor=<token>&limit=20) for large datasets, or offset-based (?offset=0&limit=20) for smaller ones.
Return consistent error formats. Every error response should have the same structure: { "error": { "code": "...", "message": "...", "details": [...] } }.
Log requests, not bodies. Log the method, URL, status code, and response time. Never log request bodies — they may contain passwords or tokens.
Use HTTPS everywhere. No exceptions. Even internal APIs should use TLS.
Set request size limits. A POST endpoint without a body size limit is vulnerable to memory exhaustion attacks. Limit to what you actually need (1 MB for most APIs).
Implement circuit breakers. If a dependent service is failing, stop calling it after N consecutive failures and return a fallback response. This prevents cascading failures.

Frequently Asked Questions

Q: What is the difference between REST and GraphQL?

A: REST models your API as resources identified by URLs, with HTTP methods as verbs. GraphQL uses a single endpoint and lets clients specify exactly what data they need in the query. REST is simpler and cacheable; GraphQL is more flexible for complex data requirements.

Q: What is idempotency and why does it matter?

A: Idempotency means making the same request multiple times produces the same result as making it once. GET, PUT, and DELETE are idempotent; POST is not. For non-idempotent operations, use an idempotency key (a UUID) so retries don't create duplicates.

Q: What is the difference between 401 and 403?

A: 401 Unauthorized means authentication failed (missing or invalid credentials) — I don't know who you are. 403 Forbidden means you're authenticated but not allowed to access this resource — I know who you are, but you can't do this.

Q: What is a JWT?

A: A JSON Web Token is a self-contained token that encodes claims (user ID, permissions, expiration) in a JSON payload, signed by the server. The server verifies the signature and extracts the claims without a database lookup. JWTs are signed, not encrypted — never put secrets in them.

Q: What is the best rate limiting algorithm?

A: Token bucket is the most common in production (used by GitHub and Stripe) because it handles bursts naturally while capping sustained traffic. Fixed window is simpler but allows 2× bursts at window boundaries. Sliding window is more precise but more expensive to implement.

Q: What headers should a rate-limited API return?

A: X-RateLimit-Limit (the limit), X-RateLimit-Remaining (requests left), and X-RateLimit-Reset (when the limit resets). On 429 responses, include Retry-After (seconds to wait before retrying).

Q: How do I protect against BOLA?

A: Broken Object-Level Authorization — the most common API vulnerability. Every endpoint that takes an ID must check that the authenticated user is authorized to access that specific resource. Don't rely on clients to only request their own data.

Q: Should API keys expire?

A: Yes. Keys that never expire are a permanent liability if leaked. Set expiration dates (90 days is common) and allow clients to rotate keys. Require HTTPS to prevent interception.

notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.

🔗 Original guide

Web Performance on a Budget: Bundle Size, API Limits, and Database Indexes

Adolfo Pedernera — Tue, 25 Aug 2026 05:09:22 +0000

The three budgets every web app hits: bundle size, API rate limits, and database index selectivity. Learn the formulas, thresholds, and PR checklist.

Introduction

Every web app has three budgets, and only one of them lives in your code. The bundle budget lives in the browser — 244 kB gzip is where Lighthouse starts penalizing Time to Interactive. The API budget lives at the provider — 5,000 requests per hour on GitHub, 500 per minute on OpenAI Tier 1, 100 writes per second on Stripe. The database budget lives in the planner — an index with 0.5 selectivity will be ignored and the query will seq scan 50,000 rows. Exceed any of the three and the user pays: slower paint, 429 Too Many Requests, or a query that times out.

The budgets are linked by the same habit that breaks them: averaging. A bundle that is 71 kB lodash full on average is 7.2 kB lodash/get when imported correctly — the average hides the choice. An API that is 20 requests per minute on average is 800 per minute during a burst deploy — the average hides the spike. A table where status has two values has selectivity 0.5 on average but 0.95 for active and 0.05 for pending — the average hides the skew.

This guide makes the three budgets explicit, with the formulas, the thresholds, and a single PR checklist that covers frontend, backend, and database. Every number is reproducible with the three calculators that accompany it — the Bundle Size Impact Calculator, the API Rate Limit & Cost Calculator, and the SQL Index Selectivity Calculator. For the network that connects them, the Bandwidth Calculator sizes the pipe.

1. Bundle Size — The Frontend Budget

The smallest budget is the one the user downloads. A parsed package of 71.0 kB lodash full is 24.4 kB gzip and 20.1 kB brotli. On a 1.6 Mbps 3G link that is 122 ms of transfer before first-paint; on 9 Mbps 4G it is 22 ms. The same utility as lodash/get is 7.2 kB parsed, 2.9 kB gzip, 15 ms on 3G — an 8× saving from a one-line import change.

The formula is linear:

S_{gzip} = S_{parsed} \times r_{gzip}

T = \frac{S _{gzip} \times 8}{B \times 1000} + RTT

where B is Mbps and RTT is round-trip in seconds. The budget share is S_gzip / 244 × 100. The Bundle Size Impact Calculator evaluates all three plus brotli.

Code — the 8× mistake and the fix:

import _ from 'lodash'; // 71.0 kB parsed
import get from 'lodash/get'; // 7.2 kB parsed
import { get } from 'lodash-es'; // tree-shaken, 7.2 kB

Bundlers like webpack only tree-shake lodash-es and named ESM imports — the default lodash import is not shaken.

Reference — popular packages (minified+gzipped):

Package	Parsed (kB)	Gzip (kB)	% of 244 kB
lodash (full)	71.0	24.4	10.0%
lodash/get	7.2	2.9	1.2%
moment	66.5	19.8	8.1%
date-fns	19.0	5.8	2.4%
react-dom	130.0	42.0	17.2%

Gzip sizes — single import choice moves 21.5 kB (107 ms on 3G). Replace moment with date-fns and save another 14 kB (70 ms).

Tip: set a CI budget of 244 kB gzip total, 130 kB per route, and paste the calculator's Budget% into the PR description — future reviewers see the cost without rebuilding.

2. API Rate Limits — The Backend Budget

If the bundle budget is about bytes, the API budget is about tokens per minute. GitHub's 5,000 per hour is 83.3 per minute; OpenAI's Tier 1 is 500 per minute; Stripe's writes are 100 per second. All are token buckets: refill at a fixed rate up to a burst capacity. You can burst to capacity instantly, but sustained throughput cannot exceed refill.

Let L = limit per minute, D = demand per minute, r = retry fraction, p = price per request:

U = D / L

T = max (0, D - L)

R = T \times r

Monthly cost (30-day month, 43,200 minutes):

C_{month} = (D + R) \times 43200 \times p

At $2 per 1K ($0.002 per request), 500 per minute limit with 800 per minute burst and 30% retry gives U = 160%, T = 300 throttled/min, R = 90 extra/min, and Deff = 890/min during the burst — $3.56 for two minutes, or $76k per month if sustained. The fix is not more retries but a queue that smooths the burst.

Code — queue instead of retry:

import pLimit from 'p-limit'; const limit = pLimit(10); // 10/min
await Promise.all(urls.map(url > limit(() => fetch(url))));

Batch where the API allows it — GitHub GraphQL and Stripe batches turn N requests into 1.

The API Rate Limit & Cost Calculator evaluates U, T, R, and C_month for any per-second/minute/hour limit.

3. SQL Index Selectivity — The Database Budget

The database budget is the most misread. Selectivity S = R / N (rows returned / total rows) decides if the planner uses the index. At S = 0.5 (50% of rows) the index is ignored; at S = 0.001% (1 row) it is always used. The threshold is around 5–20% with default costs.

S = R / N

C_{seq} = N \times 1.0, C_{idx} = R \times 4.0

If C_idx < C_seq the planner chooses index scan. For status = 'active' on 100k rows where 95k are active, S = 0.95, C_idx = 380k vs C_seq = 100k — seq scan wins. For email = 'a@b.com' with one row, S = 0.00001, C_idx = 4 vs 100k — index wins 25,000×.

Code — check before indexing:

SELECT COUNT(DISTINCT status) FROM users; -- C
EXPLAIN SELECT * FROM users WHERE status='active'; -- rows, cost

If C < 100 on 100k rows, a single-column index on that column will be ignored for most values — use a composite (status, created_at) instead.

Distinct (C)	Rows/value	Selectivity	Planner
100,000 (unique)	1	0.001%	Index
1,000	100	0.10%	Index
10	10,000	10%	Borderline
2	50,000	50%	Seq Scan

Selectivity % (log) — unique is 0.001% (always indexed), 2 distinct 50% (never alone).

The SQL Index Selectivity Calculator reports S, efficiency 1−S, and the verdict with cost.

4. Putting It Together — The PR Checklist

A single PR that adds a dependency, a new API call, and a migration can bust all three budgets at once. Check them together:

Budget	Check	Tool	Gate
Bundle	`S_gzip / 244 < 5%` per new dep	Bundle Size Impact	CI 244 kB
API	`U = D/L < 80%` sustained	API Rate Limit	Alert at 80%
Database	`S = R/N < 10%` for indexed filter	SQL Index Selectivity	EXPLAIN rows

Begginer: paste the three calculator outputs into the PR description — reviewers see the numbers without pulling the branch. Senior: add the three gates to CI — bundle via Lighthouse CI, API via X-RateLimit-Remaining header check, index via EXPLAIN in migration tests. Sensei: make the checklist a required GitHub PR template so every service that ships JS, calls an API, and migrates a table is measured the same way.

For the network that connects them, the Bandwidth Calculator converts S_gzip to wall-clock time on 3G/4G.

5. Case Study — One PR, Three Budgets

A real pull request that adds a dashboard with a chart, an analytics API call, and a new filter illustrates how the budgets interact.

The PR: Add recharts for a new RevenueChart component, fetch /api/revenue?range=30d on page load, and add WHERE status = 'active' AND region = 'EU' to the revenue query.

Bundle check: recharts is 130 kB parsed, 42 kB gzip — 17.2% of the 244 kB budget. The existing bundle is 180 kB gzip, so the new total is 222 kB (91%). The Bundle Size Impact Calculator shows 42 kB + 3G 210 ms. The fix is code-splitting: move RevenueChart to React.lazy so the initial route stays at 180 kB and the chart chunk loads on demand.

API check: The dashboard polls every 30 seconds, so 2 requests per minute per user. With 500 concurrent users, D = 1,000 per minute. The API limit is 500 per minute, so U = 200%, T = 500 throttled/min. With 30% retry, R = 150, Deff = 1,150. The API Rate Limit & Cost Calculator shows the cost and the need for a queue. The fix is caching the GET for 60 seconds and batching the poll to 1 per minute via SWR.

Database check: The new filter WHERE status='active' on 500k rows where 80% are active has S = 0.80, efficiency 20% — the planner will seq scan. Adding AND region='EU' where EU is 10% of rows makes the composite S = 0.80 × 0.10 = 0.08 (8%) — now selective enough for a composite index (region, status) or (status, region) depending on which column is more selective. The SQL Index Selectivity Calculator confirms S = 8% → Index Scan, cost 32k vs seq 500k.

The PR ships when all three checks are green: bundle under 244 kB initial, API U < 80%, and EXPLAIN shows Index Scan. One budget in the red is a revert.

6. Monitoring After Ship

Budgets are not one-time checks; they drift. Bundle size grows with every npm install, API load grows with every new replica, and selectivity shifts as the table fills. Track the three on the same dashboard: Lighthouse CI for bundle (S_gzip trend), X-RateLimit-Remaining header sampling for API (U over time), and pg_stat_user_indexes scans for database (S per index). When any of the three crosses 80% of its budget, the next feature PR should include a budget-reducing change — a lazy chunk, a cache, or a composite index — before new functionality.

7. Practical Tips

Measure with the same tool you ship with. esbuild --metafile vs Bundlephobia differ by minifier — pick one.
Import the file, not the barrel. lodash/get vs lodash is 8× — the same holds for date-fns/format vs date-fns.
Queue bursts, don't retry bursts. A p-limit queue at L eliminates throttling without retries.
Check cardinality before indexing. COUNT(DISTINCT col) < 100 on 100k rows → composite, not single-column.
Read EXPLAIN rows, not just the index list. rows = S×N tells you the selectivity the planner actually used.
Version your budgets today. Keep the three thresholds (244 kB, 80% API, 10% selectivity) in a budgets.json or lighthouse-budget.json so CI and humans share the same numbers — drift happens when the budget lives only in a comment.

Frequently Asked Questions

Q: What is the 244 kB bundle budget?

A: A common Lighthouse threshold where total gzip over ~244 kB starts penalizing Time to Interactive. Use it as a reference; your product's budget may be 170 or 500 kB.

Q: Gzip or brotli for budgeting?

A: Gzip for the conservative gate (all CDNs support it), brotli for the optimistic check (most modern CDNs serve brotli, ~15 percent smaller).

Q: When should I retry a 429?

A: Retry 20–30 percent once with exponential backoff starting from the Retry-After header. Retrying 100 percent on an overloaded bucket just re-throttles.

Q: When does an index get ignored?

A: When selectivity is high — roughly above 10 percent with default costs. On 100k rows, keeping 10k rows (10 percent) is borderline; keeping 50k (50 percent) is always a seq scan.

Q: Can I fix low selectivity with an index?

A: Alone, no — a single-column index on a 2-value column is 50 percent selective. As part of a composite (status, created_at) that raises cardinality, yes.

Q: How do I know my API's limit?

A: Check the docs (GitHub 5,000/hour, OpenAI 500/minute, Stripe 100 writes/second) and the response headers X-RateLimit-Limit/Remaining/Reset. The calculator handles per second/minute/hour.

Q: Do I need all three calculators for every PR?

A: No — check the budget you touch. New dep → bundle, new API call → rate limit, new WHERE → selectivity. If a PR touches all three, check all three.

notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.

🔗 Original guide

Design Tools Guide - Contrast, Golden Ratio and Aspect Ratio

Adolfo Pedernera — Sun, 23 Aug 2026 21:24:27 +0000

WCAG color contrast, the golden ratio, and aspect ratios — how to combine readability and proportion tools for better layouts.

Introduction

Design decisions that look subjective usually reduce to two measurable questions: can people read it, and does the layout feel balanced? The first is answered by color contrast — the numerical relationship between text and background that determines legibility for everyone, and especially for people with low vision. The second is answered by proportion — how the dimensions of a frame, a block of type, or a page relate to one another. Both questions have precise answers, and both are what make the difference between a component that merely exists and one that communicates.

This guide brings the two questions together because working designers solve them together. You do not choose a contrast ratio and a frame size in isolation: a button's color must pass WCAG against the panel behind it while the panel itself sits at a proportion that fits the canvas. The Contrast Checker measures readability, the Golden Ratio Calculator measures balance, the Aspect Ratio Calculator measures shape, and the Color Converter and Ratio Calculator fill in the conversion work those three rely on.

By the end you will know what each tool computes, when it matters, where the numbers come from, and — just as important — where each number stops being useful. The goal is not to replace design judgment with formulas, but to give judgment a numerical backbone.

The Two Design Decisions: Readability and Proportion

Every visual decision can be filed under one of two headings: contrast or proportion. Readability is a contrast problem. Typography, color, and luminance all converge on one question — is there enough difference between the text and what is behind it for the eye to separate them? Proportion is a shape problem. It asks how the parts of a composition relate: how wide a block of type should be relative to its height, how a poster's panels divide, how an image frames a subject.

Contrast is enforced by a standard. The Web Content Accessibility Guidelines (WCAG) set numeric thresholds, tested by browsers, tools, and increasingly by legal requirement, so a designer can compute pass or fail with certainty. Proportion is guided, not enforced. There is no WCAG for a golden rectangle, because balance is cultural and contextual — but there are geometric relationships, like the golden ratio, that recur across centuries of design because they are consistently pleasant to look at.

The distinction matters because it changes how you use the numbers. A contrast ratio below the WCAG threshold is objectively wrong for body text — no amount of aesthetic judgment justifies it. A proportion that ignores the golden ratio is not wrong, merely one choice among many. Designers who treat every numeric tool as a hard rule end up with rigid work; designers who treat every tool as optional end up with unreadable work. The skill is knowing which number is a gate and which is a guide.

Color Contrast and WCAG Accessibility

Color contrast is the luminance difference between two colors, computed by comparing their relative luminances and expressed as a ratio from 1:1 (identical colors) to 21:1 (black on white). The ratio is standardized by WCAG and is the same ratio every contrast checker on the web reports, so a value computed here matches the value your testing tool reports.

The relative luminance of a color is a weighted sum of its red, green, and blue channels after each channel is linearized:

L = 0.2126 \cdot R^{'} + 0.7152 \cdot G^{'} + 0.0722 \cdot B^{'}

where each channel value is converted from the sRGB 8-bit range (0–255) using the piecewise curve defined in the WCAG specification. Two colors are compared by dividing the lighter luminance plus 0.05 by the darker luminance plus 0.05:

contrast ratio = \frac{L _{lighter} + 0.05}{L _{darker} + 0.05}

WCAG sets the following pass thresholds for normal text, large text, and user interface components:

Requirement	Minimum ratio	Applies to
AA normal text	4.5:1	All body copy, buttons, form labels
AA large text	3:1	Text at least 24px, or 19px bold
AAA normal text	7:1	Extended reading, long-form content
AAA large text	4.5:1	Large text under the strictest standard
UI components	3:1	Focus indicators, icons, input borders

The thresholds are not arbitrary. Four and a half to one is calibrated so that people with typical low-vision contrast sensitivity — roughly the level of someone in their eighties or with moderate visual impairment — can still read body text. AAA pushes that further to cover people with more severe loss. Horton and Quesenbery make the accessibility argument concrete in A Web for Everyone: contrast problems are the single most common reason users with low vision abandon a site, and fixing them costs nothing but a choice of color.

Practical steps when a color pair fails: darken the text, lighten the background, or widen the type. The Color Converter helps here — converting the failing color to HSL lets you drop the lightness channel by small increments and convert back, which keeps the hue intact while moving the ratio. A brand color that fails at 12px can often pass at large heading sizes, because large text only needs 3:1. That is why the ratio, not the color, is the design constraint.

The Golden Ratio and Proportion

The golden ratio, φ (phi), is the number 1.6180339887…, and it has a distinctive property: the ratio of the whole to the larger part equals the ratio of the larger part to the smaller part. In a line divided at the golden point, the larger segment is about 61.8% of the whole and the smaller about 38.2%. The value satisfies φ = 1 + 1/φ, and it is the positive root of the equation:

φ = \frac{1 + 5}{2} \approx 1.618

The Fibonacci sequence produces the same number as a limit: each term is the sum of the previous two, and the ratio of successive terms oscillates around φ, converging on it exactly. This convergence is why designers reach for Fibonacci numbers (1, 1, 2, 3, 5, 8, 13…) as a spacing scale — 8, 13, 21, 34 is close enough to a golden scale that a spacing system built from it inherits the balance without using decimals.

Famously, φ is used to build balanced rectangles in layout: a golden rectangle's sides are in the ratio 1:φ, and a range of other classic proportions sit nearby.

Proportions	Ratio	Typical use
Golden (1:φ)	1.618	Editorial grids, poster panels, image crops
Square (1:1)	1.000	Avatars, galleries, product tiles
√2 paper (1:1.414)	1.414	ISO A-series print, book margins
Fibonacci 3:5	1.667	Magazine column pairs
Fibonacci 5:8	1.600	Spacing and type scales

Classic layout proportions plotted by ratio. The golden ratio sits between the square and the √2 paper ratio, near the Fibonacci-derived 5:8 and 3:5 pairs that form practical integer spacing scales.

The golden ratio is best treated as a starting point for composition rather than a requirement. Livio's history of the number cautions against the mythologizing that claims every great work of art uses it — the evidence that the Parthenon or Leonardo's paintings deliberately follow φ is weak, and the number's power is that it appears wherever people have sought balanced proportions, not that it was universally applied. Use it to generate a width from a height (or vice versa) when you need a pleasant rectangle fast; the Golden Ratio Calculator splits a line or scales a value by φ in either direction.

Aspect Ratios and Delivery Formats

An aspect ratio is the proportional relationship between an image's width and height, written as two numbers reduced by their greatest common divisor: 1920 × 1080 reduces to 16:9. It answers a delivery question the golden ratio does not — not "does this feel balanced" but "what shape is this frame, and what else will it fit in."

Delivery formats are strict. A 16:9 video shown on a 21:9 monitor is letterboxed; a 21:9 film on a 16:9 screen is pillarboxed. Photographers crop for print sizes (4:3, 3:2), video and displays standardized on 16:9, and ultrawide monitors push 21:9. Each one is a hard constraint on the canvas, and knowing the target ratio before you start composing saves the most work. RED's filmmaking reference is blunt about this: choose the frame first, then light and compose for it — a ratio is a contract with the viewer.

The Aspect Ratio Calculator handles both directions of the problem: it reduces any width and height to its simplest ratio, and it finds the missing dimension when you know the target ratio and one side. It shares DNA with the Ratio Calculator, which expresses the same relationship as a fraction for pure math work — the aspect calculator adds the display framing (common names, decimals, percentages) that layout work needs.

Design Tools at a Glance

Tool	What it computes	Typical question
Contrast Checker	WCAG contrast ratio, AA/AAA pass/fail	"Is this text readable on this background?"
Golden Ratio Calculator	φ split and scale of any length	"What width feels balanced for this height?"
Aspect Ratio Calculator	Reduced W:H, missing dimension, common name	"What height fits 1920 px wide at 16:9?"
Color Converter	Hex, RGB, and HSL conversions	"What lightness keeps this hue accessible?"

The four tools form a small workflow. Convert a brand color with the Color Converter, check the pair with the Contrast Checker, size the frame with the Aspect Ratio Calculator, and refine the balance with the Golden Ratio Calculator. Each tool hands its output to the next, and none of them alone decides whether a layout works.

A Worked Example: A Call-to-Action Card

A concrete walkthrough shows how the four tools cooperate. Suppose you are designing a call-to-action card for a landing page: a headline, a body paragraph, and a button, on a solid background.

Start with the shape. The card will sit beside a 16:9 hero image, but the card itself should feel calmer and more editorial — a 3:2 frame is a classic choice. You want the card to be 600 px wide. The Aspect Ratio Calculator gives the matching height: 600 ÷ 3 × 2 = 400 px. That frame now composes cleanly with the hero, sharing its 16:9 language through the common factor of the 3:2 crop.

Next, the colors. Your brand gray, #767676, on the white card background is a common "muted" choice. The Contrast Checker reports a ratio of 4.54:1 — just above the 4.5:1 AA threshold for normal text, but below the 7:1 AAA threshold. That is a defensible result for a landing page: AA passes for body copy, and the requirement is met. If this were a long-form article where AAA is the target, you would darken the gray toward #555555 (7.46:1) to clear the stricter bar.

Then the spacing. You set the padding and the gap between headline, paragraph, and button using a Fibonacci scale: 8, 13, 21, 34. The Golden Ratio Calculator can refine the choice — split the 600 px width in the golden ratio and you get 371 px for the larger part and 229 px for the smaller. A two-column card with the text block at 371 px and the visual at 229 px inherits the golden proportion, while the integer Fibonacci gaps keep the vertical rhythm mechanical and consistent.

Finally, the button. Its background must clear the 3:1 threshold against the white card because it is a UI component. The Color Converter converts your brand color into HSL so you can raise its lightness without drifting hue, and the Contrast Checker re-verifies the new pair. The card is now measurable at every step: a 3:2 frame, a 4.54:1 text ratio, a golden horizontal split, and a 3:1+ component ratio. None of these numbers replaced design judgment — they made the judgment auditable.

Common Mistakes

Treating a pass/fail contrast check as the whole accessibility story. Contrast is the measurable core, but readability also depends on type size, line length, spacing, and target size for touch. A 4.5:1 pass with 10px gray text is still hard to read.
Chasing AAA everywhere. AAA on body text is a strong goal for long-form reading, but it is not required everywhere. A badge or a disabled button at 2.5:1 is acceptable in many designs; spending hours forcing every element to 7:1 usually produces flat, low-contrast-looking pages.
Applying the golden ratio as a law. Compositional rules are heuristics. If a 1.618 ratio makes a panel too wide for its content, a 4:3 or 5:8 frame is a legitimate choice. The ratio suggests, it does not mandate.
Checking contrast on the design canvas, not the delivery frame. A hero image cropped to 4:5 on social media may show the text over a different background region than the 16:9 desktop mockup. Verify the pair against the actual export.
Assuming all ratios are alike. 21:9 ultrawide panels are often physically 43:18 or 64:27. Enter the real resolution, not the marketing label, when you compute the ratio.
Skipping the browser check. Every contrast tool computes the same ratio, but rendering differences, fonts, and anti-aliasing affect perceived contrast. Spot-check pass/fail pairs in a real browser before shipping.

Frequently Asked Questions

Q: What is the difference between color contrast and an aspect ratio?

A: Color contrast is a luminance relationship between two colors, expressed as a ratio from 1:1 to 21:1, that determines text readability. An aspect ratio is a width-to-height relationship of a frame, like 16:9. Contrast answers whether something can be read; aspect ratio answers what shape the frame is.

Q: What does WCAG contrast ratio 4.5:1 mean?

A: It is the minimum contrast ratio WCAG AA requires for normal-size text. It means the lighter color is at least 4.5 times brighter (luminance plus 0.05) than the darker color. Text meeting 4.5:1 is readable for the majority of people with low vision.

Q: How do I compute the contrast ratio between two colors?

A: Convert each color to its relative luminance L by linearizing the sRGB channels and weighting them 0.2126R + 0.7152G + 0.0722B. Then divide the lighter luminance plus 0.05 by the darker luminance plus 0.05. The Contrast Checker does all of this automatically.

Q: Is the golden ratio the same as the Fibonacci sequence?

A: No, but they are related. The ratio of consecutive Fibonacci numbers (1, 1, 2, 3, 5, 8, 13…) approaches the golden ratio 1.618 as the sequence grows. The sequence gives an integer approximation of φ that designers use for spacing scales.

Q: How is the golden ratio used in layout design?

A: Most often to generate a balanced rectangle: multiply one side of a frame by 1.618 (or by its reciprocal 0.618) to get the other. It is also the basis of Fibonacci spacing scales and classic editorial grids. It is a guide for balance, not a compliance requirement.

Q: What is the most common aspect ratio for displays?

A: 16:9 is the global standard for HDTV, most online video platforms, smartphones, and computer monitors. Other common ratios are 4:3 (older monitors and photography), 3:2 (camera sensors), 21:9 (ultrawide monitors), and 1:1 (square social media images).

Q: How do I find the height given a width and a ratio?

A: Divide the width by the ratio's width component, then multiply by the ratio's height component. For example, 1920 px wide at 16:9 is 1920 ÷ 16 × 9 = 1080 px. The Aspect Ratio Calculator does this for any ratio.

Q: Do WCAG contrast rules apply to images and logos?

A: Large logos and incidental imagery are exempt from most WCAG contrast requirements, but text overlays on images are not. Any text that carries information — including on hero images, badges, and video captions — must meet the ratio for its size.

Q: Can a design pass contrast but still be inaccessible?

A: Yes. Contrast is necessary but not sufficient. Type size, line height, spacing, color-blindness-safe palettes, focus indicators, and motion reduction all affect accessibility. A 4.5:1 pass with a cramped line height and tiny type is still a poor experience.

Q: What is the fastest way to fix a failing contrast ratio?

A: Convert the color to HSL and lower the lightness of the background or raise the lightness of the text — or simply darken text and lighten background a few steps at a time, rechecking after each change. Moving to a bolder or larger type also lowers the required threshold from 4.5:1 to 3:1 for large text.

notAcalculator provides free online calculators and educational guides covering finance, fitness, mathematics, and everyday calculations.

🔗 Original guide