Your certificate renewed. Does your service know?

lastmileops — Tue, 27 Oct 2026 13:04:27 +0000

Tested with OpenSSL 3.6 and GNU date.

Automatic certificate renewal was invented to make expiry a non-event. Most of the time it is. But when something does break,
it breaks quietly and with imagination:

the DNS provider's API token expired, and domain validation no longer passes;
a new firewall politely drops the HTTP challenge, and nobody knows;
the deploy hook restarts a service that was renamed six months ago;
renewal went perfectly, and the service keeps serving the old certificate from memory.

And if you were counting on reminder emails from your CA: Let's Encrypt stopped sending them back in 2025. Nobody is going to
remind you now except you.

One simple idea covers that whole list: a check that doesn't trust the renewal job at all. It connects to the service the way
an ordinary client does and looks at the certificate that actually comes back. Not the file on disk, not the certbot log, but what
your users' browsers see.

#!/usr/bin/env bash
# check-cert.sh HOST[:PORT] [MIN_DAYS] — check the certificate a service actually presents
set -uo pipefail
target="$1"; min="${2:-14}"
host="${target%%:*}"; [[ "$target" == *:* ]] || target="$target:443"
pem=$(echo | timeout 10 openssl s_client -servername "$host" -connect "$target" 2>/dev/null \
      | openssl x509 2>/dev/null) || true
if [[ -z "$pem" ]]; then echo "UNREACHABLE: $target presented no certificate"; exit 2; fi
end=$(openssl x509 -noout -enddate <<<"$pem" | cut -d= -f2)
days=$(( ( $(date -u -d "$end" +%s) - $(date -u +%s) ) / 86400 ))
if ! openssl x509 -noout -checkhost "$host" <<<"$pem" | grep -q "does match"; then
  echo "MISMATCH: $target presents a certificate not valid for $host (expires in $days days)"; exit 2
fi
if (( days < 0 )); then echo "EXPIRED: $target expired on $end"; exit 1; fi
if (( days < min )); then echo "EXPIRING: $target expires in $days days ($end)"; exit 1; fi
echo "OK: $target valid for $days more days"

What it says:

OK: intranet.example.com:443 valid for 54 more days
EXPIRING: wiki.example.com:443 expires in 9 days (Nov  2 18:13:52 2026 GMT)
EXPIRED: old-vpn.example.com:443 expired on Feb  1 00:00:00 2025 GMT
MISMATCH: files.example.com:443 presents a certificate not valid for files.example.com (expires in 399 days)
UNREACHABLE: mail.example.com:465 presented no certificate

Pick a threshold below the point where renewal should already have happened. From February 10, 2027, Let's Encrypt will issue
64-day certificates by default, and it already recommends renewing at about two-thirds of the lifetime, which leaves roughly
21 days. An alert at 14 days stays quiet on normal renewals and fires when one is a week late. A 21-day threshold would page
someone on every routine renewal, and within a month everyone learns to ignore it. Which is exactly what we were trying to avoid.

MISMATCH is the sneakiest one. A virtual host lost its own certificate and quietly serves the server's default one. Technically
there is a certificate, and it's valid. Just not for this name. Your users' browsers will disagree with you.

Run it for every name and port that matters, and from a different machine than the one doing the renewals. Otherwise the
check and the renewal break together and stay silent together.

#!/usr/bin/env bash
# check-all.sh — run check-cert.sh for every host:port in certs.txt, print only problems
while read -r t _; do
  [[ -z "$t" || "$t" == \#* ]] && continue
  /opt/tools/check-cert.sh "$t" 14 || true
done < /opt/tools/certs.txt | grep -v '^OK'

0 7 * * * out=$(/opt/tools/check-all.sh); [ -z "$out" ] || echo "$out" | notify-oncall

Send the result to a role or a shared queue, not to whoever set all this up once. The address that gets the alert should outlive
the person who typed it in. Otherwise you end up like one thread on Spiceworks: renewal failed, and the notifications were going
to the mailbox of a former contractor who was long gone.

What it doesn't do. It doesn't validate the chain or revocation: if you need that, add -verify_return_error and your own CA
bundle to s_client. Services that upgrade with STARTTLS (SMTP on 25 and 587, IMAP on 143) need -starttls smtp or
-starttls imap. And it tells you that something broke, not why. For the why, keep a record of every renewal run: which
version, when, and how it ended.

Whenever I get the chance to make my geeky thoughts more pleasant for a wider audience, I don't shy away from what modern AI can do.

Run records and missed-run checks for scheduled scripts

lastmileops — Fri, 23 Oct 2026 17:29:50 +0000

Tested on Linux with bash, jq and GNU date.

A quiet Slack channel can mean two things: nothing went wrong, or the job never ran. Most small scheduled scripts can't
tell those apart, because the only trace they leave is the alert they send when something looks bad.

Two small additions fix that: write a record for every run, including the boring successful ones, and check separately
that each job actually finished recently.

1. One line per run

#!/usr/bin/env bash
# run-logged.sh JOB_NAME COMMAND... — run a job and append one JSON line per run to $RUN_LOG
set -uo pipefail
job="$1"; shift
log="${RUN_LOG:-/var/log/run-records.jsonl}"
rev=$(sha256sum "$(command -v "$1")" 2>/dev/null | cut -c1-12)
start=$(date -u +%FT%TZ)
out=$("$@" 2>&1); status=$?
end=$(date -u +%FT%TZ)
jq -nc --arg job "$job" --arg rev "${rev:-unknown}" --arg start "$start" --arg end "$end" \
      --argjson status "$status" --arg out "$(printf "%s" "$out" | tail -c 2000)" \
      '{job:$job, rev:$rev, start:$start, end:$end, status:$status, output:$out}' >> "$log"
exit "$status"

Use it in cron instead of calling the script directly:

0 6 * * * RUN_LOG=/var/log/run-records.jsonl /opt/tools/run-logged.sh nightly-report /opt/tools/nightly-report.sh

Every run now leaves a line with the job name, a short hash of the program that ran, start and end time, exit status and
the tail of its output — whether it found something or not. When someone asks "did it run on Tuesday, and which version?",
the answer is one jq away.

2. A separate check for missed runs

#!/usr/bin/env bash
# check-missed.sh JOB_NAME MAX_AGE_MINUTES — fail if the job has no finished, successful run recently
set -uo pipefail
job="$1"; max="$2"; log="${RUN_LOG:-/var/log/run-records.jsonl}"
last=$(jq -r --arg job "$job" 'select(.job==$job and .status==0) | .end' "$log" 2>/dev/null | tail -1)
if [[ -z "$last" ]]; then echo "MISSED: $job has no successful run on record"; exit 2; fi
age=$(( ( $(date -u +%s) - $(date -u -d "$last" +%s) ) / 60 ))
if (( age > max )); then echo "MISSED: $job last succeeded ${age} min ago (limit ${max})"; exit 2; fi
echo "OK: $job succeeded ${age} min ago"

Schedule it after the job's deadline and send its output wherever alerts go:

30 7 * * * RUN_LOG=/var/log/run-records.jsonl /opt/tools/check-missed.sh nightly-report 120 || notify-oncall

It complains in three cases: the job never succeeded, it last succeeded too long ago, or it keeps failing. A failed data
fetch shows up as a failed check instead of a clean report.

What this doesn't do

It doesn't make the script correct, and it doesn't tell you why a run was skipped — only that it was. It's also still a
file on one machine: if several people depend on the job, they need somewhere to see these records that isn't your shell.

This article was created with the help of AI. The code was run on Linux before publishing.

DEV Community: lastmileops

Your certificate renewed. Does your service know?

Run records and missed-run checks for scheduled scripts

1. One line per run

2. A separate check for missed runs

What this doesn't do