How I build web scrapers a business can actually rely on

Mojtaba Karimi — Mon, 26 Oct 2026 15:49:27 +0000

The problem. A business wants to track prices across many online stores — competitors, suppliers, its own catalog. Checking by hand is slow and drifts out of date, and a naive scraper makes it worse: it gets blocked, it breaks the moment a page changes, and it hands back half-parsed HTML nobody can trust. The real question is never "can you scrape it." It is "can I rely on this data tomorrow, and next month."

What I built. A price-intelligence platform that scrapes products across stores, keeps a full price history, detects every change, and shows it on a dashboard and over an API. You can open it right now: https://price-intelligence-demo.onrender.com (free instance, the first request takes about 40 seconds to wake). The code is public, with the test suite and CI badges next to it.

The decisions that make it reliable — the part most scrapers skip:

Polite by default. It reads robots.txt, holds one request per second per host, and an SSRF guard resolves every URL and refuses private and internal addresses, with no switch to turn that off. This keeps the target site, and you, out of trouble.
Append-only history. Every product carries its full trend line, its lowest-ever price, and a log of every detected change. Nothing is overwritten, so "when did this move, and by how much" always has an answer.
One file per store. Adding a new shop is a single new plugin, with no change to the pipeline, the API or the database. The system grows without rewrites.
Honest data. Tests run against fixtures, offline and deterministic, so the suite proves the parsing is correct instead of depending on a live shop that changes under it.

Proof, not promises. An 80% test-coverage floor is enforced in CI — the build fails if it drops. ruff, mypy and bandit run on every push. And there is a live demo you can click. I do not ask you to take my word for any of it: the badges are live and the code is open.

What this means if you hire me. You get clean, structured data — CSV, JSON, or straight into a database — plus the script and a short guide, so you can run it again yourself. It extends to new sources without a rewrite, and it tells you when something breaks instead of failing silently. Competitor prices, B2B lead lists, or any public dataset: the same discipline applies — sourced, checked, and something you can still trust next month.

Want this for your data? Tell me the site and the fields you need, and I will tell you honestly what is possible. Hire me on Upwork, Freelancer or Fiverr, or email mojtaba.python@gmail.com.

Python's smtplib doesn't verify TLS certificates by default

Mojtaba Karimi — Fri, 04 Sep 2026 09:51:31 +0000

I found this while writing the e-mail alert channel for a monitoring tool. The
code looked fine, passed review, and was quietly insecure:

with smtplib.SMTP(host, 587) as server:
    server.starttls()
    # credentials, over an unauthenticated channel:
    server.login(username, password)
    server.sendmail(...)

That connection is encrypted but not authenticated. Anyone who can
intercept the traffic can present any certificate they like, and Python will
accept it — then send your SMTP username and password through the tunnel they
control.

The proof

Here is a local SMTP server presenting a self-signed certificate issued for
totally-not-your-mail-server.example, while the client connects to
127.0.0.1. The hostname doesn't match and no CA signed it — two independent
reasons to reject it:

1) client.starttls()                      # the default
   RESULT: handshake ACCEPTED
   -> the client trusted this certificate

2) client.starttls(context=ssl.create_default_context())
   RESULT: handshake REFUSED
   -> [SSL: CERTIFICATE_VERIFY_FAILED] self-signed certificate

The default accepts a certificate that every browser on earth would refuse.

Why

From CPython's smtplib:

def starttls(self, *, context=None):
    ...
    if context is None:
        context = ssl._create_stdlib_context()
    self.sock = context.wrap_socket(self.sock, server_hostname=self._host)

And what is _create_stdlib_context? In ssl.py:

>>> ssl._create_stdlib_context is ssl._create_unverified_context
True

It is an alias for _create_unverified_context. The name says it.

>>> ctx = ssl._create_stdlib_context()
>>> ctx.check_hostname, ctx.verify_mode
(False, <VerifyMode.CERT_NONE: 0>)
>>> d = ssl.create_default_context()
>>> d.check_hostname, d.verify_mode
(True, <VerifyMode.CERT_REQUIRED: 2>)

create_default_context() — the one you have to ask for by name — is the safe
one. The one you get by doing nothing is not.

It is not just STARTTLS

I assumed implicit TLS would be fine. It isn't. SMTP_SSL.__init__ has the
same default:

def __init__(self, host='', port=0, ..., context=None):
    if context is None:
        context = ssl._create_stdlib_context()
    self.context = context

So SMTP_SSL(host, 465) — the "just use SMTPS, it's secure" option — is also
unverified unless you pass a context. Both roads lead to the same place.

The fix

One argument:

import ssl, smtplib

context = ssl.create_default_context()      # verifies chain AND hostname

with smtplib.SMTP(host, 587) as server:
    server.starttls(context=context)
    server.login(username, password)

For implicit TLS:

smtplib.SMTP_SSL(host, 465, context=ssl.create_default_context())

While you're there, one more guard worth having — refuse to send credentials at
all if the session never got encrypted:

if username and password:
    if not (use_tls or use_ssl):
        raise RuntimeError("refusing to send SMTP credentials over cleartext")
    server.login(username, password)

A misconfiguration should cost you a failed alert, not a leaked password.

Check your own code

grep -rn "starttls()" --include="*.py" .
grep -rn "SMTP_SSL(" --include="*.py" .

Any starttls() with empty parentheses, or any SMTP_SSL(...) without
context=, is unverified.

This shows up a lot in the wild: password-reset mailers, CI notifiers, cron
scripts, "send me an alert when the job fails" glue code. All of them
authenticate to an SMTP server, which means all of them hand over a credential.

Is this a bug?

Not exactly, and that's the interesting part.

The docstring does hint at it — "If you provide the context parameter, the
identity of the SMTP server and client can be checked" — which, read
carefully, tells you that without it, the identity is not checked. But it
is phrased as an added capability rather than a warning, and the safe behaviour
is the one you have to opt into.

The default is what it is for backward compatibility. urllib and httplib
were fixed to verify by default back in Python 3.4.3 (PEP 476); smtplib was
left alone, largely because a great many mail servers ran self-signed or
mismatched certificates and changing it would have broken them. That reasoning
made sense in 2015. It has aged.

The practical consequence today: TLS in Python's SMTP client protects you
against a passive eavesdropper, and not at all against an active one. If your
threat model includes someone who can redirect traffic — a hostile network, a
compromised router, DNS poisoning — then the default gives you a comfortable
feeling and no security.

Takeaway

Encrypted is not the same as authenticated. If a TLS API lets you connect
without naming a context, find out which context it picked for you.

ssl.create_default_context()   # the one you want
ssl._create_stdlib_context()   # the one you get

Verified on CPython 3.12.5; the same code is present in current CPython. I ran into this while building an open-source monitoring tool whose alerting sends e-mail — the fix and its regression tests are in src/webmon/alerting/channels.py.

DEV Community: Mojtaba Karimi