DEV Community: seotto.com

How to check hreflang tags for errors (with a script)

seotto.com — Sat, 31 Oct 2026 22:42:16 +0000

If your site has several language or country versions, hreflang tags tell search engines which version to show to which audience. They're also easy to get wrong in ways that don't show up anywhere visible: the page looks fine, but the tags are ignored. Here are the common mistakes and a script to find them.

What correct hreflang looks like

Each language version lists every version, including itself:

<!-- on https://example.com/en/pricing -->
<link rel="alternate" hreflang="en" href="https://example.com/en/pricing" />
<link rel="alternate" hreflang="de" href="https://example.com/de/preise" />
<link rel="alternate" hreflang="x-default" href="https://example.com/en/pricing" />

And /de/preise must carry the same set, pointing back.

The common mistakes

Missing return links. Page A says B is its German version, but B doesn't list A. Search engines may ignore the pair.
Invalid codes. en-UK instead of en-GB, jp instead of ja. Language codes are ISO 639-1; regions are ISO 3166-1 alpha-2.
Pointing to redirects or 404s. Every href should return 200 directly.
Pointing to non-canonical URLs. If the target page's canonical points elsewhere, the signals conflict.
Relative URLs. Use absolute URLs including the protocol.
No self-reference. Each page should list itself.

A checker script

#!/usr/bin/env python3
import sys, requests
from html.parser import HTMLParser

UA = {"User-Agent": "Mozilla/5.0 (hreflang check)"}

class Alts(HTMLParser):
    def __init__(self):
        super().__init__(); self.alts, self.canonical = {}, None
    def handle_starttag(self, tag, attrs):
        if tag != "link": return
        a = dict(attrs); rel = (a.get("rel") or "").lower()
        if rel == "alternate" and a.get("hreflang"):
            self.alts[a["hreflang"]] = a.get("href", "")
        if rel == "canonical":
            self.canonical = a.get("href")

def read(url):
    r = requests.get(url, headers=UA, timeout=20, allow_redirects=False)
    p = Alts()
    if r.status_code == 200:
        p.feed(r.text)
    return r.status_code, p.alts, p.canonical

def check(start):
    status, alts, canonical = read(start)
    if status != 200:
        print(f"start page returned {status}"); return
    if start not in alts.values():
        print("missing self-reference")
    for code, href in alts.items():
        if not href.startswith("http"):
            print(f"{code}: relative URL {href}")
            continue
        s, their_alts, their_canon = read(href)
        if s != 200:
            print(f"{code}: {href} returned {s}")
            continue
        if their_canon and their_canon != href:
            print(f"{code}: canonical of {href} is {their_canon}")
        if start not in their_alts.values():
            print(f"{code}: {href} has no return link to {start}")
    print(f"checked {len(alts)} alternates")

check(sys.argv[1])

python3 hreflang.py https://example.com/en/pricing

allow_redirects=False is deliberate: a redirect is reported as a problem rather than silently followed.

Validating the codes

The script doesn't validate codes. A quick extra check with a small list catches the usual typos:

COMMON_MISTAKES = {"en-uk": "en-GB", "jp": "ja", "cn": "zh", "gr": "el", "se": "sv", "dk": "da"}
for code in alts:
    if code.lower() in COMMON_MISTAKES:
        print(f"{code}: did you mean {COMMON_MISTAKES[code.lower()]}?")

For full validation, check the language part against ISO 639-1 and the region against ISO 3166-1.

Sitemaps instead of HTML

You can also declare hreflang in the XML sitemap with xhtml:link elements. The same rules apply: every URL lists all alternates, including itself. Choose one method. Mixing HTML tags and sitemap entries makes conflicts more likely.

A quick checklist

[ ] Every page lists all its language versions, including itself
[ ] Every listed URL returns 200 without redirecting
[ ] Every target page links back
[ ] Codes are valid ISO language (and optional region) codes
[ ] Each target's canonical points to itself
[ ] An x-default is set if you have a language picker or global fallback

Run through it for one page in each template type (home, category, article, product). Hreflang problems are usually template problems, so one broken example often means all pages of that type are affected.

Keep it from breaking again

Hreflang breaks when a page is renamed in one language only, when a translation is unpublished, or when a template change drops the tags. Run the check after deploys that touch localized templates. Ongoing crawls that compare canonicals and status codes, like SEOtto does, will also surface the redirects and canonical changes that quietly break hreflang pairs.

Find duplicate and missing title tags with a Python crawl script

seotto.com — Sat, 31 Oct 2026 22:41:53 +0000

Duplicate title tags are one of the most common technical SEO problems, and one of the easiest to create by accident: a template change gives every product page the same title, a pagination setting repeats the category name, a CMS falls back to the site name when a field is empty. Search engines then have a harder time telling those pages apart. Here's a small script that reads your sitemap, fetches each page and reports duplicate, missing and overlong titles.

What it checks

Missing <title> or empty title
Titles shared by more than one URL
Titles that are very long or very short
Duplicate meta descriptions, as a bonus

The script

#!/usr/bin/env python3
import sys, re, collections, concurrent.futures as cf
import requests
from xml.etree import ElementTree as ET
from html.parser import HTMLParser

UA = {"User-Agent": "Mozilla/5.0 (title audit)"}
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def sitemap_urls(url):
    root = ET.fromstring(requests.get(url, headers=UA, timeout=20).content)
    if root.tag.endswith("sitemapindex"):
        for loc in root.findall("sm:sitemap/sm:loc", NS):
            yield from sitemap_urls(loc.text.strip())
    else:
        for loc in root.findall("sm:url/sm:loc", NS):
            yield loc.text.strip()

class Head(HTMLParser):
    def __init__(self):
        super().__init__()
        self.title, self.desc, self._in_title = None, None, False
    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "title" and self.title is None:
            self._in_title, self.title = True, ""
        if tag == "meta" and (a.get("name") or "").lower() == "description":
            self.desc = (a.get("content") or "").strip()
    def handle_endtag(self, tag):
        if tag == "title":
            self._in_title = False
    def handle_data(self, data):
        if self._in_title:
            self.title += data

def fetch(url):
    try:
        r = requests.get(url, headers=UA, timeout=20)
        p = Head(); p.feed(r.text)
        title = re.sub(r"\s+", " ", p.title or "").strip()
        return url, r.status_code, title, p.desc or ""
    except requests.RequestException as e:
        return url, type(e).__name__, "", ""

def main(sitemap):
    urls = list(dict.fromkeys(sitemap_urls(sitemap)))
    titles, descs = collections.defaultdict(list), collections.defaultdict(list)
    with cf.ThreadPoolExecutor(max_workers=4) as pool:
        for url, status, title, desc in pool.map(fetch, urls):
            if status != 200:
                print(f"[status {status}] {url}")
                continue
            if not title:
                print(f"[missing title] {url}")
            elif len(title) > 65 or len(title) < 15:
                print(f"[length {len(title)}] {url}: {title}")
            titles[title].append(url)
            if desc:
                descs[desc].append(url)
    for t, us in titles.items():
        if t and len(us) > 1:
            print(f"\n[duplicate title x{len(us)}] {t}")
            for u in us: print("   ", u)
    for d, us in descs.items():
        if len(us) > 1:
            print(f"\n[duplicate description x{len(us)}] {d[:80]}")
            for u in us: print("   ", u)

if __name__ == "__main__":
    main(sys.argv[1])

Run it:

pip install requests
python3 titles.py https://example.com/sitemap.xml

It uses only four parallel workers on purpose. Be gentle with your own server, especially in production.

Reading the results

Duplicates on paginated pages (/blog/page/2, /blog/page/3): add the page number to the title, e.g. "Blog, page 2". That's usually a template setting.

Duplicates on near-identical pages (product variants, filters, tracking parameters): decide whether they should exist as separate URLs at all. Often the fix is a canonical tag or removing them from the sitemap, not a new title.

Missing titles: almost always a template or CMS field fallback. Find which template the pages share.

Length warnings: there's no fixed limit, since search engines truncate by pixel width, not characters. Treat 65 characters as a rough guide for when the end of the title may be cut off, not a rule.

Why titles change without anyone noticing

Titles are generated from templates, plugins and CMS fields, so a single change can rewrite hundreds of them at once: a plugin update changes the separator, a theme update drops the brand suffix, a translation file overwrites a field. A one-off audit catches today's problems; it doesn't catch next week's.

Turning it into a regular check

You could run the script on a schedule and diff its output with the previous run. Save the URL-to-title map as JSON and compare. That's the core idea behind SEO monitoring tools. SEOtto does this for titles, meta descriptions, H1s, canonicals and noindex tags on every crawl, and alerts when a value changes.

How to check if robots.txt is blocking Google (and fix it without breaking anything)

seotto.com — Fri, 30 Oct 2026 19:58:33 +0000

robots.txt is a plain text file with a lot of power. One line in it can stop Google from crawling your entire site, and because every page still opens normally in a browser, nobody notices until traffic falls. Here is how to check whether yours is blocking Google, and how to read the rules correctly.

Step 1: Find the file

robots.txt always lives at the root of the host:

https://example.com/robots.txt

Note that each host has its own: www.example.com, example.com, blog.example.com and the http:// versions can all serve different files. Check the one your canonical URLs use.

From a terminal you can also see the status code:

curl -sI https://example.com/robots.txt | head -1
curl -s https://example.com/robots.txt

The status code matters more than people expect:

200: Google follows the rules in the file.
404 / 410: Google treats it as "no rules", so everything may be crawled.
5xx or timeout: Google may stop crawling the site for a while, because it can't tell what is allowed. A broken robots.txt is worse than a missing one.

Step 2: Understand the groups

Rules are grouped by User-agent. A crawler uses the most specific group that matches it and ignores the others:

User-agent: *
Disallow: /admin/

User-agent: Googlebot
Disallow: /

Here, Bing and others can crawl everything except /admin/, but Googlebot is blocked from the whole site, because the Googlebot group wins and the * group no longer applies to it. This catches people out when an old rule for one bot was left in the file.

Step 3: Read Allow and Disallow correctly

For Google, when several rules match a URL, the longest matching rule wins. If an Allow and a Disallow are the same length, Allow wins.

User-agent: *
Disallow: /shop/
Allow: /shop/products/

/shop/cart is blocked, /shop/products/red-shoes is allowed, because the Allow rule is longer.

Two wildcards are supported:

* matches any sequence of characters: Disallow: /*?sort= blocks sorted listing URLs.
$ anchors the end: Disallow: /*.pdf$ blocks URLs ending in .pdf.

And the most important line of all:

Disallow: /

means "nothing on this host may be crawled". An empty Disallow: (nothing after the colon) means the opposite: everything is allowed. One character separates "fully open" from "fully closed".

Step 4: Test a specific URL

Search Console has a robots.txt report (under Settings) that shows which robots.txt files Google found for your property, when it last fetched them and any parsing problems. For a single page, the URL Inspection tool shows "Crawl allowed?" and says "No: blocked by robots.txt" when a rule matches.

If you prefer to test locally, Python ships a basic parser:

from urllib import robotparser

rp = robotparser.RobotFileParser("https://example.com/robots.txt")
rp.read()
print(rp.can_fetch("Googlebot", "https://example.com/shop/products/red-shoes"))

Be aware that Python's parser uses first-match order rather than Google's longest-match rule, so for tricky Allow/Disallow combinations, trust Google's own report or the open-source Google robots.txt parser.

Step 5: Know what robots.txt does not do

Blocking a URL in robots.txt stops crawling, not indexing. If other pages link to a blocked URL, Google can still list it, usually without a description. If you want a page out of search results, use a noindex meta tag or X-Robots-Tag header and leave the page crawlable so Google can see it.

Also, robots.txt is public and advisory. It is not a security tool. Anything private needs authentication.

The usual ways a site gets blocked by accident

Staging file shipped to production. The staging server had Disallow: / to keep it out of Google, and the deploy copied it over.
CMS "discourage search engines" switches. Some platforms change the virtual robots.txt when you tick that box.
A blanket rule written for one section. Disallow: /p meant for /print/ also blocks /products/ and /pricing, because rules are prefix matches.
Blocking CSS and JavaScript. Google renders pages; if /assets/ is blocked, it may see a broken layout.
The file starts returning 5xx after a server or CDN change.

Fixing it safely

Edit the rule, and keep the change as small as possible.
Check the file returns 200 and the new content.
In Search Console, open the robots.txt report and request a recrawl of the file.
Use URL Inspection on a few key pages to confirm "Crawl allowed? Yes".

Keep an eye on it

robots.txt rarely breaks on its own; it breaks when someone deploys. That's why it is worth checking after every release, or letting a tool do it for you. A monitor such as SEOtto reads robots.txt at the start of every crawl, keeps each version line by line, and alerts you when it starts blocking the site or Googlebot. Either way, five minutes reading the file today can save you weeks of lost traffic later.

DEV Community: seotto.com

How to check hreflang tags for errors (with a script)

What correct hreflang looks like

The common mistakes

A checker script

Validating the codes

Sitemaps instead of HTML

A quick checklist

Keep it from breaking again

Related reading

Find duplicate and missing title tags with a Python crawl script

What it checks

The script

Reading the results

Why titles change without anyone noticing

Turning it into a regular check

Related reading

How to check if robots.txt is blocking Google (and fix it without breaking anything)

Step 1: Find the file

Step 2: Understand the groups

Step 3: Read Allow and Disallow correctly

Step 4: Test a specific URL

Step 5: Know what robots.txt does not do

The usual ways a site gets blocked by accident

Fixing it safely

Keep an eye on it

Related reading