DEV Community: Krishna

The Python Default Argument Bug That Looks Completely Normal

Krishna — Sun, 01 Nov 2026 19:03:01 +0000

Here's a Python function that looks completely normal:

def add_tag(tag, tags=[]):
    tags.append(tag)
    return tags

Give it a tag and, optionally, an existing list of tags.

If you don't pass a list, it looks like the function should start with an empty one.

Right?

Let's try it.

print(add_tag("urgent"))
print(add_tag("billing"))
print(add_tag("refund"))

Output:

['urgent']
['urgent', 'billing']
['urgent', 'billing', 'refund']

Wait. What?

Each call was supposed to start with an empty list. Instead, every call remembers the tags from the previous ones.

The function isn't broken. Python is doing exactly what it's supposed to.

The problem is how we understand default arguments.

Here's the one sentence that explains it:

Python evaluates a default argument once, when the def statement runs, not every time the function is called.

There's only one empty list, and every call that doesn't provide its own list shares it.

Let me show you how to prove that, why it can cause real problems, and how to fix it.

Step 1: Watch Python reuse the same list

Python stores a function's positional default arguments in an attribute called __defaults__.

We can inspect it directly.

def add_tag(tag, tags=[]):
    tags.append(tag)
    return tags

print(add_tag.__defaults__)

add_tag("urgent")
add_tag("billing")

print(add_tag.__defaults__)

Output:

([],)
(['urgent', 'billing'],)

Look at that.

The default started as an empty list.

After two calls, the function's default itself contains "urgent" and "billing".

The default isn't an instruction to create a new empty list. It's a reference to one specific list object.

We can even prove both calls return that same object:

first = add_tag("a")
second = add_tag("b")

print(first is second)  # True

is checks whether two variables refer to the same object.

And they do.

That's why changes made during one call show up in later calls.

Step 2: It's not really a list problem

Most explanations call this the "mutable default argument bug."

That's accurate, but there's something more important underneath.

The actual rule is about when Python evaluates the default expression.

Lists make the problem obvious because you can modify them. But the timing also matters for values that aren't mutable.

Consider this:

import time
from datetime import datetime

def log(message, at=datetime.now()):
    print(f"[{at:%H:%M:%S}] {message}")

log("job started")

time.sleep(2)

log("job finished")

You might expect two different timestamps.

But the output looks like this:

[10:15:03] job started
[10:15:03] job finished

Your actual timestamp will differ, but both messages will show the same time.

Why?

Because datetime.now() ran when Python executed the def log(...) statement.

It didn't run again when we called log().

So both messages reused the same timestamp, even though we waited two seconds between calls.

Once you understand this, the list problem stops looking like some strange Python exception.

It's the same rule in both cases.

The default expression runs once when the function is defined, and its result is reused.

Step 3: Where this becomes a real problem

The tag example is annoying, but mostly harmless.

Now imagine a function that validates batches of orders.

It collects warnings for orders with missing amounts.

def validate_orders(orders, warnings=[]):
    for order in orders:
        if order["amount"] is None:
            warnings.append(
                f"order {order['id']}: missing amount"
            )

    return warnings

Let's process Monday's orders:

monday = [
    {"id": 1, "amount": 90},
    {"id": 2, "amount": None}
]

print("Monday:", validate_orders(monday))

Output:

Monday: ['order 2: missing amount']

That's correct.

Order 2 has a missing amount.

Now let's process Tuesday's orders:

tuesday = [
    {"id": 3, "amount": 40}
]

print("Tuesday:", validate_orders(tuesday))

Output:

Tuesday: ['order 2: missing amount']

Tuesday's data is completely valid.

But its validation report still contains Monday's warning.

Nothing about Tuesday's input caused that warning.

It was carried over through the shared default list.

Now imagine this function inside a long-running application.

A scheduled worker processing different jobs. A notebook that's been open all afternoon. Or a web server handling requests from multiple users.

If the same function object handles those calls, state stored in a mutable default can carry across them.

Depending on what you're storing, that can lead to incorrect results or even expose one request's data in another response.

And a test that only calls the function once might never catch it.

The bug becomes obvious when you test repeated calls, not just isolated ones.

Step 4: Fix it with `None`

The usual fix is simple.

Instead of using an empty list as the default, use None.

Then create the list inside the function.

def add_tag(tag, tags=None):
    if tags is None:
        tags = []

    tags.append(tag)
    return tags

print(add_tag("urgent"))
print(add_tag("billing"))

Output:

['urgent']
['billing']

Now each call that omits tags gets a fresh list.

Why does this work?

Because None is the shared default, but the list gets created inside the function body.

That code runs every time we call the function without supplying a list.

And if someone passes an existing list, we still use it:

my_tags = ["important"]

add_tag("review", my_tags)

print(my_tags)

Output:

['important', 'review']

So we get the behavior we actually wanted.

No list provided? Create one.

List provided? Use that one.

The timestamp example needs the same treatment:

def log(message, at=None):
    if at is None:
        at = datetime.now()

    print(f"[{at:%H:%M:%S}] {message}")

Now the timestamp is generated when log() runs, unless the caller explicitly provides one.

Step 5: Be careful with `tags or []`

There's a shorter version of this fix that you'll see in Python code.

It looks convenient:

def add_tag(tag, tags=None):
    tags = tags or []

    tags.append(tag)
    return tags

And it appears to work.

Until someone passes an empty list intentionally.

my_tags = []

add_tag("urgent", my_tags)

print(my_tags)

Output:

[]

Wait, where did "urgent" go?

The function did append it, but not to my_tags.

Here's the problem:

tags = tags or []

An empty list is falsy in Python.

So when tags is [], the expression chooses a new list instead.

The caller's original list gets ignored.

That's why this version is safer:

if tags is None:
    tags = []

It creates a new list only when the argument is actually None.

An empty list passed by the caller remains an empty list that the function can modify.

Small difference. Very different behavior.

Step 6: Python dataclasses have a similar safeguard

There's another place where this issue shows up.

Suppose you're using a dataclass to represent a support ticket.

You might try:

from dataclasses import dataclass

@dataclass
class Ticket:
    tags: list[str] = []

But Python raises a ValueError because a mutable list is being used as a dataclass field default.

Instead, use field(default_factory=list).

from dataclasses import dataclass, field

@dataclass
class Ticket:
    tags: list[str] = field(default_factory=list)

Now let's create two tickets.

first = Ticket()
second = Ticket()

first.tags.append("urgent")

print(first.tags)
print(second.tags)

Output:

['urgent']
[]

Each ticket gets its own list.

default_factory=list tells the dataclass to call list() when creating a new instance, rather than sharing one list across all instances.

It's the same underlying idea as our None fix: create the mutable object when it's actually needed.

Step 7: Let your linter catch this

You don't have to rely entirely on remembering this rule.

Python linters can flag suspicious default arguments before the code runs.

For example:

Ruff / flake8-bugbear B006 catches mutable argument defaults like tags=[].
Ruff B008 flags function calls in default arguments, such as datetime.now().
Pylint W0102 warns about dangerous mutable defaults.

If you're using Ruff, you can check for both of these patterns with:

ruff check --select B006,B008 .

That doesn't replace understanding the behavior, but it's a useful way to catch mistakes during development.

And one important distinction: not every default argument is dangerous.

def greet(name, punctuation="!"):
    return f"Hello, {name}{punctuation}"

Strings, numbers, and other immutable values are generally fine to reuse.

Even an empty tuple () is a reasonable default when you only need a read-only sequence.

Just remember that an immutable container can still hold mutable objects.

And if you genuinely want state shared across calls, make that intention explicit. For caching, something like functools.lru_cache is usually clearer than hiding a dictionary in a default argument.

The mental model that finally makes sense

I used to read this:

def add_tag(tag, tags=[]):

as:

"If no list is passed, create an empty list."

But Python actually treats it more like:

"When this function is defined, create one empty list. Whenever a caller doesn't supply a list, use that same one."

That's a very different thing.

Once you understand that default expressions run when the def statement executes, the shared list and frozen timestamp make sense.

And the fix becomes obvious.

If a default value needs to be fresh for every call, create it inside the function.

Try it yourself

Take the validate_orders() example and run it with three different batches.

Then inspect:

print(validate_orders.__defaults__)

You'll see the old warnings sitting inside the default list.

Change the function to use warnings=None, create a fresh list inside the function, and try again.

This time, __defaults__ should remain (None,), no matter how many batches you process.

I'm building Runtime, a free hands-on learning platform for Python and SQL where you can practice concepts by writing and running code directly in your browser.

If you're learning Python, this is a good bug to reproduce yourself. It's much easier to remember once you've seen it happen.

Have you ever run into this bug, or caught it during a code review?

My SQL JOIN Ran Perfectly and Doubled the Refunds

Krishna — Wed, 28 Oct 2026 14:03:24 +0000

The SQL query ran without errors. The columns looked right. The totals seemed reasonable.

Except Ava's refund total was $60.

She'd only been refunded $30.

No missing data. No syntax error. Nothing obviously wrong with the query.

SQL had done exactly what the query asked it to do. The problem was that the query wasn't asking the right question.

Here's the idea that explains it:

A JOIN doesn't guarantee one result row per row in your first table. One row can match multiple rows, and those matches can multiply your results.

That might sound harmless until you start calculating totals.

Let's reproduce the problem with a dataset small enough to check every number ourselves.

Step 1: Four customers, six orders, three refunds

Imagine we're building a report for an online store.

We have three tables: customers, orders, and refunds.

Let's create them.

CREATE TABLE customers (
    customer_id INTEGER PRIMARY KEY,
    name        TEXT
);

CREATE TABLE orders (
    order_id    INTEGER PRIMARY KEY,
    customer_id INTEGER,
    amount      INTEGER
);

CREATE TABLE refunds (
    refund_id   INTEGER PRIMARY KEY,
    customer_id INTEGER,
    amount      INTEGER
);

INSERT INTO customers VALUES
    (1, 'Ava'),
    (2, 'Ben'),
    (3, 'Cara'),
    (4, 'Dan');

INSERT INTO orders VALUES
    (101, 1, 90),
    (102, 1, 40),
    (103, 2, 75),
    (104, 2, 60),
    (105, 2, 60),
    (106, 3, 120);

INSERT INTO refunds VALUES
    (1, 1, 20),
    (2, 1, 10),
    (3, 2, 15);

For this example, orders and refunds are recorded separately against each customer.

Here's what our actual numbers look like:

Customer	Orders	Total spent	Total refunded
Ava	2	130	30
Ben	3	195	15
Cara	1	120	0
Dan	0	0	0
Total	6	445	45

These are the correct numbers.

Keep them in mind. We're about to make SQL return something very different.

Step 2: One JOIN, more rows than customers

Let's start with something simple.

SELECT
    c.name,
    o.order_id,
    o.amount
FROM customers AS c
JOIN orders AS o
    ON o.customer_id = c.customer_id
ORDER BY o.order_id;

Result:

name	order_id	amount
Ava	101	90
Ava	102	40
Ben	103	75
Ben	104	60
Ben	105	60
Cara	106	120

Four customers. Six joined rows.

Ava appears twice because she has two orders. Ben appears three times because he has three.

Dan disappears because an inner JOIN only returns matching rows, and Dan has no orders.

SQL hasn't accidentally duplicated anything.

Each result row represents one matching customer-order pair.

And that brings us to a concept that's extremely useful in analytics.

Grain.

Grain simply means what one row represents.

In our example:

customers has one row per customer.
orders has one row per order.
The joined result has one row per matching order, with customer information attached.

The relationship between customers and orders is one-to-many.

One customer can have multiple orders.

You'll see the same pattern everywhere. Departments have employees. Accounts have transactions. Projects have tasks.

Joining to the many side can change the grain of your result.

That's not inherently a problem. Sometimes that's exactly what you want.

The trouble starts when you forget it happened.

Step 3: The report that's only half right

Now imagine we want a customer report showing how much each customer spent and how much they received in refunds.

This query looks reasonable:

SELECT
    c.name,
    SUM(o.amount) AS total_spent,
    SUM(r.amount) AS total_refunded
FROM customers AS c
LEFT JOIN orders AS o
    ON o.customer_id = c.customer_id
LEFT JOIN refunds AS r
    ON r.customer_id = c.customer_id
GROUP BY
    c.customer_id,
    c.name
ORDER BY c.customer_id;

It runs successfully.

Here's what we get:

name	total_spent	total_refunded
Ava	260	60
Ben	195	45
Cara	120	NULL
Dan	NULL	NULL

Something is wrong.

Ava actually spent $130, but the report says $260. Her refunds doubled from $30 to $60.

Ben's spending is correct at $195, but his $15 refund has somehow become $45.

Cara looks fine, apart from the missing refund being represented as NULL.

And Dan has no activity.

Across the whole report, the spending total has grown from $445 to $575, and refunds have jumped from $45 to $105.

The dangerous part?

Some values are correct. Others are wrong. And every incorrect number looks believable.

That's what makes this kind of SQL bug difficult to catch.

Step 4: Why it happens

Look at our joins again.

LEFT JOIN orders AS o
    ON o.customer_id = c.customer_id

LEFT JOIN refunds AS r
    ON r.customer_id = c.customer_id

Both tables are joined using customer_id.

But neither orders nor refunds necessarily contains only one row per customer.

Ava has two orders and two refunds.

So SQL produces every matching order-refund combination for Ava.

order_id	order_amount	refund_id	refund_amount
101	90	1	20
101	90	2	10
102	40	1	20
102	40	2	10

Two orders multiplied by two refunds gives us four rows.

Now look at what happens when we aggregate:

Order total:
90 + 90 + 40 + 40 = 260

Refund total:
20 + 10 + 20 + 10 = 60

Every order was counted twice because there were two matching refunds.

Every refund was counted twice because there were two matching orders.

This is called join fan-out.

Now consider Ben.

He has three orders but only one refund.

Three orders multiplied by one refund gives us three rows.

His order amounts each appear once, so his spending total stays correct.

But his single $15 refund appears three times.

That's why the query returns $45.

The important thing to understand is:

When you join two one-to-many relationships through the same key, matching rows from each side can combine with every matching row from the other side.

And a LEFT JOIN doesn't prevent that.

A left join preserves unmatched rows from the left side. It doesn't guarantee only one match for each row.

Step 5: How to catch this before trusting the report

When a JOIN produces unexpected totals, the first thing I'd check isn't the SUM().

I'd check what the rows represent.

Start with two questions:

What should one row in the final report represent?

In this case, one customer.

Can the join key match multiple rows in either table?

Let's find out.

Check 1: Count the rows before aggregation

SELECT
    c.name,
    COUNT(*) AS joined_rows
FROM customers AS c
LEFT JOIN orders AS o
    ON o.customer_id = c.customer_id
LEFT JOIN refunds AS r
    ON r.customer_id = c.customer_id
GROUP BY
    c.customer_id,
    c.name
ORDER BY c.customer_id;

Result:

name	joined_rows
Ava	4
Ben	3
Cara	1
Dan	1

This tells us how many joined rows feed into each customer's calculations.

Ava's totals are being calculated from four rows, not two orders or two refunds independently.

Notice that the original report still returned one final row per customer because of GROUP BY.

Having the correct number of final rows doesn't mean your aggregated values are correct.

Check 2: Look for repeated join keys

For example, let's check the refunds table.

SELECT
    customer_id,
    COUNT(*) AS rows_per_key
FROM refunds
GROUP BY customer_id
HAVING COUNT(*) > 1;

Result:

customer_id	rows_per_key
1	2

Customer 1 has two refund rows.

That means a customer-level JOIN can multiply rows for Ava.

We should perform the same uniqueness check on orders, because that table also has multiple rows per customer.

Check 3: Reconcile totals with the source

This is one of the simplest checks you can make.

SELECT
    SUM(amount) AS actual_refunds
FROM refunds;

Result:

actual_refunds
--------------
45

But our customer report adds up to 105.

If both reports cover the same records and use the same definitions, those totals should agree.

They don't.

That tells us the aggregation logic needs attention.

Step 6: Fix the grain before joining

Here's the solution.

We want one row per customer.

So rather than joining individual orders and refunds directly, let's aggregate each table to one row per customer first.

WITH order_totals AS (
    SELECT
        customer_id,
        COUNT(*) AS order_count,
        SUM(amount) AS total_spent
    FROM orders
    GROUP BY customer_id
),
refund_totals AS (
    SELECT
        customer_id,
        SUM(amount) AS total_refunded
    FROM refunds
    GROUP BY customer_id
)
SELECT
    c.name,
    COALESCE(o.order_count, 0) AS order_count,
    COALESCE(o.total_spent, 0) AS total_spent,
    COALESCE(r.total_refunded, 0) AS total_refunded
FROM customers AS c
LEFT JOIN order_totals AS o
    ON o.customer_id = c.customer_id
LEFT JOIN refund_totals AS r
    ON r.customer_id = c.customer_id
ORDER BY c.customer_id;

Now the results are correct:

name	order_count	total_spent	total_refunded
Ava	2	130	30
Ben	3	195	15
Cara	1	120	0
Dan	0	0	0

Every number matches our original data.

Why does this work?

Because each CTE has at most one row per customer_id.

The order_totals CTE summarizes orders independently.

The refund_totals CTE summarizes refunds independently.

Then we join those customer-level summaries to the customer table.

There's no order-refund multiplication left to inflate our totals.

We also keep LEFT JOIN so Dan still appears, even without orders or refunds.

And COALESCE turns missing aggregate values into zero for this report.

The key takeaway:

If your report needs one row per customer, make sure the data you're joining is at the customer grain before calculating customer-level totals.

That's often much safer than joining raw transactions and trying to repair the result afterward.

The tempting fix: DISTINCT

Here's where things get interesting.

When people see repeated rows, a common reaction is to use DISTINCT.

So what if we try this?

SELECT
    c.name,
    SUM(DISTINCT o.amount) AS total_spent,
    SUM(DISTINCT r.amount) AS total_refunded
FROM customers AS c
LEFT JOIN orders AS o
    ON o.customer_id = c.customer_id
LEFT JOIN refunds AS r
    ON r.customer_id = c.customer_id
GROUP BY
    c.customer_id,
    c.name
ORDER BY c.customer_id;

Result:

name	total_spent	total_refunded
Ava	130	30
Ben	135	15
Cara	120	NULL
Dan	NULL	NULL

Ava is fixed.

Ben is now wrong in a completely different way.

His actual spending was $195, but the query says $135.

Why?

Look at Ben's orders:

Order 103: $75
Order 104: $60
Order 105: $60

Actual total: $195

Ben placed two separate $60 orders.

But SUM(DISTINCT o.amount) doesn't know that those are different orders.

It only sees the values.

So it calculates:

75 + 60 = 135

One legitimate $60 order disappears from the calculation.

This is why SUM(DISTINCT ...) is not a general fix for JOIN fan-out.

DISTINCT removes duplicate values, not repeated business entities.

There's nothing wrong with using DISTINCT when you genuinely need unique values.

But using it to cover up unexpected join multiplication can replace one reporting error with another.

Fix the grain, not just the symptom.

Two more traps worth knowing

COUNT(*) can mislead you after a LEFT JOIN

Remember Dan?

He has no orders.

Now run this:

SELECT
    c.name,
    COUNT(*) AS joined_rows,
    COUNT(o.order_id) AS order_count
FROM customers AS c
LEFT JOIN orders AS o
    ON o.customer_id = c.customer_id
GROUP BY
    c.customer_id,
    c.name
ORDER BY c.customer_id;

Result:

name	joined_rows	order_count
Ava	2	2
Ben	3	3
Cara	1	1
Dan	1	0

Dan has zero orders.

But COUNT(*) returns 1 because the LEFT JOIN preserves an unmatched customer row.

The order columns in that row are NULL.

COUNT(o.order_id) ignores those NULLs and correctly returns zero.

One caution: when you add other one-to-many joins, even COUNT(o.order_id) can become inflated by fan-out. Count at the correct grain or count distinct identifiers when that matches the question.

Sometimes EXISTS is better than JOIN

Suppose we only want to answer:

"Which customers have received at least one refund?"

We don't need every refund row.

We just need to know whether one exists.

SELECT
    c.name
FROM customers AS c
WHERE EXISTS (
    SELECT 1
    FROM refunds AS r
    WHERE r.customer_id = c.customer_id
)
ORDER BY c.customer_id;

Result:

Ava
Ben

Each customer appears once.

An ordinary inner JOIN to refunds would return Ava twice because she has two refunds.

EXISTS answers the existence question without bringing every matching refund into the result.

That's a useful distinction.

The checklist I use before trusting a JOIN

Before trusting a query that calculates counts, totals, or averages, these are the questions worth asking:

What's the grain of each table? What does one source row represent?
What grain should my result have? One row per customer, order, transaction, or something else?
Is the join key unique on either side? If not, how many matches can each row produce?
Am I combining multiple one-to-many relationships? If so, should I aggregate each source separately before joining?
Do the totals reconcile with the source data? Can I explain any differences?

Most JOIN tutorials focus on INNER JOIN, LEFT JOIN, and RIGHT JOIN.

Those are important.

But there's another question that's just as useful:

How many rows can each row match?

Once you start asking that, unexpected JOIN results become much easier to understand.

And you stop assuming that a query is correct just because it runs successfully.

Try it yourself

Take the example tables from this article and deliberately break the report.

Add another refund for Ava. Predict how many joined rows she'll have and how much her spending total will increase.

Then add a second order for Cara and compare the results.

Finally, try adding a customer with no orders or refunds and see how INNER JOIN, LEFT JOIN, and EXISTS behave.

I'm building Runtime, a free, hands-on learning platform for SQL and Python where you can learn concepts by writing and running actual queries.

Working with small datasets like these is one of the best ways to understand what SQL is really doing before you run the same logic against millions of records.

What's the most confusing JOIN bug you've encountered?

Python Generators Finally Made Sense When I Stopped Thinking About Lists

Krishna — Mon, 26 Oct 2026 18:39:59 +0000

For a long time, I thought Python generators were just a complicated way to create lists.

I knew yield made something "lazy", but I didn't really understand what that meant. Most examples showed a function yielding a few numbers, and I couldn't see why I'd use that instead of returning a list.

Then two things finally clicked.

A list stores values. A generator produces values when you ask for them.

And more importantly:

A generator is a function that pauses.

When you call a generator function, Python doesn't immediately run its body. It gives you a generator object. That object runs the function a little at a time, pausing at each yield until you ask for another value.

That's the part I was missing.

Let me show you what actually happens.

Step 1: Calling a generator doesn't run it

Here's a simple countdown:

def countdown(n):
    print("starting")

    while n > 0:
        yield n
        n -= 1

    print("done")

gen = countdown(3)

print("generator created")

Output:

generator created

Notice something?

There's no "starting" in the output.

Even though we called countdown(3), Python hasn't executed the function body yet.

It created a generator object and handed it back.

Now we have to ask it for a value.

Step 2: `next()` runs it until the next `yield`

Let's call next() a few times:

print(next(gen))
print(next(gen))
print(next(gen))

Output:

starting
3
2
1

Here's what happened:

The first next() starts executing the function, prints "starting", and runs until yield n. It gives us 3.
The second next() resumes after that yield, decrements n, and gives us 2.
The third does the same thing and gives us 1.

The important part is that the function doesn't start over each time.

It remembers where it stopped, including the values of its local variables.

That's what yield does. It hands back a value without finishing the function.

What happens if we call next() again?

try:
    next(gen)
except StopIteration:
    print("no more values")

Output:

done
no more values

The function finishes, and Python raises StopIteration because there are no more values to produce.

That's the basic generator mechanism.

Step 3: A `for` loop does all of this for you

In everyday Python code, you rarely need to call next() manually.

A for loop handles it:

for number in countdown(3):
    print(number)

Output:

starting
3
2
1
done

The loop keeps requesting values and automatically stops when the generator is exhausted.

From the outside, it looks almost identical to looping over a list.

But there's a big difference in when the work happens.

Step 4: Stop early, and the remaining work never happens

This is where generators started feeling genuinely useful to me.

Imagine we're converting strings into integers.

I want to find the first number greater than 10.

Here's a generator that prints whenever it processes something:

def parse(values):
    for value in values:
        print(f"  parsing {value!r}")
        yield int(value)

data = ["4", "8", "15", "16", "23", "42"]

for number in parse(data):
    if number > 10:
        print("found", number)
        break

Output:

  parsing '4'
  parsing '8'
  parsing '15'
found 15

Look at that output.

We had six strings, but Python only converted three.

The moment we found 15, we stopped asking for more values.

The remaining three strings were never processed.

Now compare that with building a list first:

numbers = [int(value) for value in data]

for number in numbers:
    if number > 10:
        print("found", number)
        break

This produces the same result, but the list comprehension converts all six strings before the loop begins.

With six values, that difference barely matters.

With six million records, an expensive calculation, or an API you're paging through, avoiding unnecessary work becomes much more interesting.

And this is also where the memory benefit comes in.

The generator doesn't need to store every converted value in a new list.

It produces them as they're requested.

Step 5: Something realistic — processing application logs

Let's use a more practical example.

Suppose we have a large application log and want to extract only the error messages.

First, let's create a tiny log file so you can run the entire example yourself:

with open("app.log", "w", encoding="utf-8") as file:
    file.write(
        "09:00:01 INFO server started\n"
        "09:00:05 ERROR database timeout\n"
        "09:00:09 INFO request ok\n"
        "09:01:12 ERROR disk almost full\n"
        "09:02:30 ERROR database timeout\n"
    )

Now we'll build three generators, each with one responsibility.

One reads lines, one parses them into records, and one filters by log level.

def read_lines(source):
    for line in source:
        yield line.rstrip("\n")


def parse_log(lines):
    for line in lines:
        time, level, message = line.split(" ", 2)

        yield {
            "time": time,
            "level": level,
            "message": message,
        }


def only(level, records):
    for record in records:
        if record["level"] == level:
            yield record

Now connect them:

with open("app.log", encoding="utf-8") as file:
    errors = only(
        "ERROR",
        parse_log(read_lines(file))
    )

    for record in errors:
        print(record["time"], record["message"])

Output:

09:00:05 database timeout
09:01:12 disk almost full
09:02:30 database timeout

The interesting part is how these generators work together.

The for loop asks only() for a record.

only() asks parse_log() for a parsed record.

parse_log() asks read_lines() for a line.

And read_lines() gets that line from the file.

The line passes through the pipeline before another is requested.

We're not loading the entire file into a list, parsing every record upfront, and then building another list containing errors.

Each stage processes data incrementally.

That's useful whether the log contains five lines or millions.

What if we only want the first two errors?

Python's itertools.islice lets us stop after a certain number of results.

from itertools import islice

with open("app.log", encoding="utf-8") as file:
    errors = only(
        "ERROR",
        parse_log(read_lines(file))
    )

    for record in islice(errors, 2):
        print(record["message"])

Output:

database timeout
disk almost full

Once the second matching record is produced, iteration stops. The remaining lines aren't read by our pipeline.

The with open(...) block also makes sure the file is closed when we're done, including when we stop early.

This is the kind of problem where generators really earn their place.

And notice how each stage is just a small, reusable function.

parse_log() doesn't care whether its lines came from a file, a list, or another generator.

Step 6: Generator expressions — the one-line version

You've probably used a list comprehension like this:

squares = [x * x for x in range(5)]

That builds a list containing all five results.

Change the square brackets to parentheses:

squares = (x * x for x in range(5))

Now it's a generator expression.

The values are produced when you ask for them.

print(next(squares))  # 0
print(next(squares))  # 1
print(next(squares))  # 4

Here's something interesting.

What happens if we turn that same generator into a list now?

print(list(squares))

Output:

[9, 16]

Not [0, 1, 4, 9, 16].

Why?

Because we already consumed the first three values.

The generator continues from where it left off.

That little example explains one of the biggest differences between generators and lists.

Generator expressions are also handy with functions like sum() and any():

print(sum(x * x for x in range(5)))
# 30

print(any(x > 10 for x in range(20)))
# True

sum() consumes all the generated values.

any() stops as soon as it finds a truthy result, so it doesn't need to check all 20 numbers.

The gotchas, so you don't learn them the hard way

Generators are useful, but there are a few behaviors that can catch you off guard.

1. You can't reuse an exhausted generator

numbers = (x * x for x in range(3))

print(list(numbers))
print(list(numbers))

Output:

[0, 1, 4]
[]

The second list is empty because the generator has already finished.

If you need another pass, create a new generator or store the results in a list.

2. An empty generator is still truthy

This one is surprisingly easy to miss:

numbers = (x for x in [])

if numbers:
    print("this still prints")

Output:

this still prints

A generator object itself is truthy, regardless of whether it will produce any values.

So if numbers: doesn't check whether the generator has data.

If you need to check for at least one item, you can use next(numbers, None) when None isn't a valid value in your data.

Just remember that checking this way also consumes the first value, if one exists.

3. No indexing or `len()`

This doesn't work:

numbers = (x for x in range(5))

print(numbers[0])  # TypeError
print(len(numbers))  # TypeError

Generators don't support random-access indexing or len().

If you need those operations, a list is usually a better fit.

4. Errors can appear later than you expect

Remember our parse() generator?

numbers = parse(["4", "oops"])

No error yet.

Now:

print(next(numbers))  # 4
print(next(numbers))  # ValueError

The second call fails because "oops" can't be converted to an integer.

The error didn't happen when we created the generator. It happened when execution reached the invalid value.

This also matters when testing generators.

If your test only creates a generator without iterating over it, you haven't actually tested its body.

5. Turning everything back into lists

numbers = list(x * x for x in range(1_000_000))

This is valid Python.

But now you've stored all one million values.

Sometimes that's exactly what you need. Other times, you've lost the memory benefit you were trying to get.

The important question isn't whether you can use a generator.

It's whether you need all the results stored at once.

So, list or generator?

I don't think generators should replace lists everywhere.

They solve different problems.

List	Generator
Stores its elements	Produces values on demand
Can be iterated repeatedly	Gets consumed as you iterate
Supports indexing	No direct indexing
Supports `len()`	No built-in length
Convenient when you need all results	Useful when processing values one at a time
Holds its elements in memory	Avoids storing the full generated sequence

Use a list when you need to access elements repeatedly, index into them, sort them, or keep the results around.

Use a generator when you're dealing with large data, might stop early, or want to connect processing steps without building intermediate lists.

And one thing worth remembering: generators aren't automatically faster.

Producing values one at a time has overhead too. Their advantage is often reduced memory usage and avoiding work you never needed to do.

The thing that finally made it click

I used to think of a generator as a different kind of list.

Now I think of it as a paused computation that knows how to produce the next value.

That's why yield makes sense.

That's why next() resumes execution instead of starting over.

That's why a generator can be exhausted.

And that's why generators work so well for reading files, processing records, and chaining transformations.

A list is a collection of values you already have.

A generator is a way to get the next value when you need it.

Once I understood that difference, generators stopped feeling complicated.

Go deeper, hands-on

Generators are one of those Python features that make much more sense when you actually write a few and watch what happens.

I'm building Runtime, a free hands-on learning platform for Python and SQL.

There's an interactive lesson on generator functions where you can experiment with yield, write your own generators, and work through coding challenges in your browser.

Practice Python generators on Runtime

The lessons that follow explore yield from, lazy processing pipelines, and measuring memory usage with tracemalloc.

If you've been learning generators, try the stop-early example from this post. Change the condition, add a few more values, and see exactly when Python does the work.

When did Python generators finally click for you?

Python Decorators Finally Clicked When I Saw This One Line

Krishna — Fri, 23 Oct 2026 14:44:29 +0000

Decorators confused me for an embarrassingly long time. Every tutorial jumped straight to @app.route or @lru_cache, and I nodded along without actually getting it.

Then someone showed me the one line that makes it click.

This:

@deco
def f():
    pass

is equivalent to:

def f():
    pass

f = deco(f)

That's it.

A decorator takes a function and returns a replacement. Usually, that replacement is another function that adds behavior around the original.

Everything else builds on that idea. Let me prove it by building one from scratch.

Step 1: Functions are values

In Python, a function is just another value. You can pass it into another function:

def add_exclamation(text):
    return text + "!"

def apply_twice(func, value):
    return func(func(value))

print(apply_twice(add_exclamation, "hello"))
# hello!!

We passed add_exclamation into apply_twice without calling it ourselves. Then apply_twice called it twice.

A decorator uses exactly this ability: it takes a function, builds a wrapper around it, and returns the wrapper.

Step 2: Build a decorator by hand

Say I want to count how many times a function gets called:

def count_calls(func):
    def wrapper(*args, **kwargs):
        wrapper.calls += 1
        return func(*args, **kwargs)

    wrapper.calls = 0
    return wrapper

def parse_magnitude(text):
    return float(text)

parse_magnitude = count_calls(parse_magnitude)

print(parse_magnitude("5.5"))    # 5.5
print(parse_magnitude(" 4.7 "))  # 4.7
print(parse_magnitude.calls)     # 2

What actually happened?

count_calls received the original parse_magnitude function.
It created a new function called wrapper.
The wrapper increments a counter, calls the original function, and returns its result.
count_calls returns the wrapper, which replaces the original function under the name parse_magnitude.

Three things to notice:

wrapper accepts *args, **kwargs and passes them through unchanged, so it can handle different function signatures.
State (the count) lives on the wrapper itself.
The wrapper returns the original's result. Forget that return and every call silently returns None.

One subtle detail: the counter increments before calling the original function. That means failed calls count too.

Step 3: The @ shorthand

That last assignment is so common Python gave it its own syntax:

@count_calls
def place_name(feature):
    return feature["properties"]["place"]

This is the same thing as:

def place_name(feature):
    return feature["properties"]["place"]

place_name = count_calls(place_name)

The name now refers to the wrapper.

There's no special execution engine hiding behind @. Python defines the function, passes it to the decorator, and assigns the returned object to the name.

Once I understood that, decorators stopped looking like magic.

Step 4: Don't lose the original function's identity

There's one catch.

Because the decorated name now points at the wrapper, inspecting the function can give you the wrapper's metadata instead of the original's.

Using our first decorator:

print(place_name.__name__)  # wrapper

That gets confusing when you're debugging, logging, or inspecting functions.

The standard library gives us functools.wraps to fix this:

import functools

def count_calls(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        wrapper.calls += 1
        return func(*args, **kwargs)

    wrapper.calls = 0
    return wrapper

@count_calls
def place_name(feature):
    """Return the place name from a feature."""
    return feature["properties"]["place"]

print(place_name.__name__)  # place_name
print(place_name.__doc__)
# Return the place name from a feature.

@functools.wraps(func) preserves useful metadata from the original function, including its name and docstring. It also keeps a reference to the wrapped function.

Make it a habit when writing wrappers.

And yes, @wraps is itself a decorator.

Step 5: A realistic one — logging with failures

Wrappers can run code before and after a function call, and they can see exceptions too.

Imagine you're processing incoming data and want to know which functions were called, what arguments they received, and whether they succeeded.

Instead of adding logging to every function, we can write one reusable decorator.

import functools

log = []

def log_calls(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        parts = [repr(a) for a in args]
        parts += [
            f"{k}={v!r}"
            for k, v in kwargs.items()
        ]

        call = f"{func.__name__}({', '.join(parts)})"

        try:
            result = func(*args, **kwargs)
        except Exception as err:
            log.append(
                f"{call} raised {type(err).__name__}"
            )
            raise

        log.append(f"{call} -> {result!r}")
        return result

    return wrapper

Now let's apply it to a parser that converts depth measurements into kilometers:

@log_calls
def parse_depth(text, unit="km"):
    value = float(text)
    return value / 1000 if unit == "m" else value

parse_depth("22")
parse_depth("10", unit="m")

try:
    parse_depth("n/a")
except ValueError:
    pass

for line in log:
    print(line)

Output:

parse_depth('22') -> 22.0
parse_depth('10', unit='m') -> 0.01
parse_depth('n/a') raised ValueError

Notice what happened.

Our decorator:

Recorded the function name and arguments, including keyword arguments.
Logged the result when the call succeeded.
Captured the exception type when parsing failed.
Re-raised the exception so the caller still knew something went wrong.

The raise matters. Without it, our wrapper would hide errors and make failed operations appear successful.

And the actual parse_depth function knows nothing about logging.

That's the useful part of decorators: we added behavior without modifying the function's main job.

When to actually use them

Decorators are useful for behavior that multiple functions need in the same way, but that isn't their main responsibility.

Think logging, timing, caching, retrying, validating arguments, or checking permissions.

If you find yourself copy-pasting the same five lines into every function, that's a decorator worth considering.

And a few classic gotchas, so you don't learn them the hard way:

Forgetting return wrapper — the decorated name becomes None.
Calling the original without returning its result — callers receive None.
Using fixed parameters instead of *args, **kwargs — your wrapper only supports certain function signatures.
Swallowing exceptions — failures disappear when callers may need to handle them.
Skipping @functools.wraps — the decorated function exposes the wrapper's name and other metadata.

You don't need a decorator for everything. Sometimes a normal function is simpler.

But when the same supporting behavior belongs around many functions, decorators are a clean way to reuse it.

The one line to remember

If you forget everything else from this post, remember:

@decorator
def my_function():
    pass

means:

def my_function():
    pass

my_function = decorator(my_function)

A function goes in. A replacement comes out.

Once that clicked, the rest of decorators started making sense.

Go deeper, hands-on

Reading about decorators is one thing. Writing them yourself and seeing exactly what breaks is another.

I built a free interactive lesson on this. You can run Python directly in your browser and work through two coding challenges that test the concepts we covered.

Practice Python decorators interactively on Runtime

It's part of Runtime, a free, hands-on learning platform I'm building for Python and SQL.

The idea is simple: learn a concept, write real code, and test whether you actually understand it.

If decorators finally make sense, the next lesson covers decorators that take their own settings, like:

@retry_on(TimeoutError, attempts=3)
def fetch_data():
    ...

That's where things get even more interesting.

What was the moment Python decorators finally clicked for you?

DEV Community: Krishna

The Python Default Argument Bug That Looks Completely Normal

Step 1: Watch Python reuse the same list

Step 2: It's not really a list problem

Step 3: Where this becomes a real problem

Step 4: Fix it with None

Step 5: Be careful with tags or []

Step 6: Python dataclasses have a similar safeguard

Step 7: Let your linter catch this

The mental model that finally makes sense

Try it yourself

My SQL JOIN Ran Perfectly and Doubled the Refunds

Step 1: Four customers, six orders, three refunds

Step 2: One JOIN, more rows than customers

Step 3: The report that's only half right

Step 4: Why it happens

Step 5: How to catch this before trusting the report

Check 1: Count the rows before aggregation

Check 2: Look for repeated join keys

Check 3: Reconcile totals with the source

Step 6: Fix the grain before joining

The tempting fix: DISTINCT

Two more traps worth knowing

COUNT(*) can mislead you after a LEFT JOIN

Sometimes EXISTS is better than JOIN

The checklist I use before trusting a JOIN

Try it yourself

Python Generators Finally Made Sense When I Stopped Thinking About Lists

Step 1: Calling a generator doesn't run it

Step 2: next() runs it until the next yield

Step 3: A for loop does all of this for you

Step 4: Stop early, and the remaining work never happens

Step 5: Something realistic — processing application logs

What if we only want the first two errors?

Step 6: Generator expressions — the one-line version

The gotchas, so you don't learn them the hard way

1. You can't reuse an exhausted generator

2. An empty generator is still truthy

3. No indexing or len()

4. Errors can appear later than you expect

5. Turning everything back into lists

So, list or generator?

The thing that finally made it click

Go deeper, hands-on

Python Decorators Finally Clicked When I Saw This One Line

Step 1: Functions are values

Step 2: Build a decorator by hand

Step 3: The @ shorthand

Step 4: Don't lose the original function's identity

Step 5: A realistic one — logging with failures

When to actually use them

The one line to remember

Go deeper, hands-on

Step 4: Fix it with `None`

Step 5: Be careful with `tags or []`

Step 2: `next()` runs it until the next `yield`

Step 3: A `for` loop does all of this for you

3. No indexing or `len()`