DEV Community: Scott Williams

Getting only the new DOIs from the Crossref API every day: dates, cursors and rate limits

Scott Williams — Sun, 01 Nov 2026 17:01:47 +0000

AI disclosure: this post was drafted with AI assistance. I ran every request and every number below myself against the live Crossref REST API on November 1, 2026, and checked the code and tests before publishing.

I wanted a boring thing: every morning, a CSV of the journal articles that got a DOI since yesterday, for a handful of journals, and never the same DOI twice. Crossref's REST API is free, needs no key, and held 188,319,891 records when I asked, so it should be one request. It is close, but it took me a few wrong turns. Here are the ones worth knowing about.

The finished script is about 400 lines of standard-library Python with a daily GitHub Actions workflow. It is my repo, free and MIT licensed: https://github.com/fecklesspleb/crossref-new-works-feed

1. Pick the right date: created, not published, not indexed

/works has four families of date filters, and for "what is new" they give very different answers. Here are the counts for a single day, 30 October 2026, all work types, each filter with from-… and until-… set to that day. The created count is stable; the other three drift a little as records change, so they are rounded:

Filter	Works on 30 Oct 2026
`from-created-date` / `until-created-date`	86,622
`from-index-date` / `until-index-date`	about 789,000
`from-update-date` / `until-update-date`	about 155,000
`from-pub-date` / `until-pub-date`	about 18,900

created is when the DOI was first registered. That is the one you want for "new".
index changes whenever Crossref re-indexes a record, including citation counts. Nine times more rows, nearly all old works.
update is when the publisher last deposited a change to the metadata.
pub-date is the publication date the publisher wrote down. It can be a year, a month, or a print issue date months away. "Published yesterday" misses most of what appeared yesterday and includes old material deposited late.

For scale, type:journal-article with created gave 26,945 on 28 October, 29,785 on 29 October and 26,966 on 30 October. On Saturday 31 October it was 16,641. Filter by journal, publisher or funder unless you really want ~9 MB of CSV a day.

2. Always set both ends of the range

from-created-date:2026-10-30 on its own means "from then until now". That is fine today and silently wrong when you re-run a past day. Set until-created-date too; both ends are inclusive.

The filters also take a time: from-created-date:2026-10-30T12:00:00,until-created-date:2026-10-30T12:59:59 returned 2,829 works for that hour.

3. Page with a cursor, and you can still sort

rows maxes out at 1,000. rows=1001 gets a 400:

Integer specified as 1001 but must be a positive integer less than or equal to 1000.

offset stops at 10,000 (offset must be ... less than or equal to 9999. Use the cursor parameter ...). So use deep paging:

Send cursor=* with the first request.
Read message.next-cursor from the answer and send it back with the next request.
Stop when a page comes back empty.

I was not sure cursors keep a sort order, so I tested it. With sort=created&order=asc and rows=2, three cursor pages came back in ascending created order, with the same DOIs as a single rows=6 request.

Reading oldest first matters for a feed with a cap. If you stop after N new works, the next run can carry on where you stopped instead of rereading the newest ones.

params = {
    "filter": "from-created-date:2026-10-31,until-created-date:2026-11-01,"
              "type:journal-article,issn:0028-0836",
    "rows": 1000,
    "cursor": cursor,                # "*" first, then message["next-cursor"]
    "sort": "created",
    "order": "asc",
    "select": "DOI,type,title,container-title,ISSN,publisher,member,created,published",
}

select= keeps responses small. A field name it does not know returns a 400 that lists the valid ones, which is handy.

4. The 2026 rate limits are lower than old blog posts say

Crossref changed its limits in July 2026. For list and filter requests like these:

1 request per second in the public pool;
3 per second in the polite pool;
polite-pool limits counted per mailto email address.

The response headers tell you where you are. On a public-pool request I saw:

x-rate-limit-limit: 1
x-rate-limit-interval: 1s
x-concurrency-limit: 1
x-api-pool: public-array

So my client:

sends one request at a time, at most once a second;
slows down further if those headers ask for less.

I still hit a real 429. I ran the script twice back to back from the shell, each process started fresh, and the first request of the second came too soon after the last request of the first. The fix is the usual one:

honour Retry-After (seconds or an HTTP date);
otherwise back off 2, 4, 8 and 16 seconds;
give up after 5 tries;
retry 5xx and network errors the same way.

A 403, or an answer that is not JSON, is different. Stop and write nothing.

5. Say who you are

To get into the polite pool, add mailto=you@example.org to the query string or the User-Agent. I do both, with a User-Agent that names the tool and links to its source:

crossref-new-works-feed/1.0 (+https://github.com/fecklesspleb/crossref-new-works-feed; mailto:you@example.org)

Use your own address. The limit is per email, so a shared one in a popular template would put everyone in the same bucket. I also validate it loosely and drop it if it does not look like an email. (api.crossref.org/robots.txt was a 404 when I checked, which means no rules, but the script checks it anyway.)

6. "New" arrives late, so overlap your window

A DOI's created timestamp is when it was registered, not when the API can serve it. The newest work I could see was 20 to 30 minutes behind its created time. Indexing can take longer on a busy day.

So a daily run should not ask for "yesterday" only. Mine asks for yesterday plus today and lets a seen-DOI list drop repeats. A work is only missed if it reaches the API more than a day late.

7. `query` is a relevance search, not a filter

query.bibliographic=perovskite solar over two days matched 246 works. Only 29 had both words in the title. If you sort by date (as a feed must), you no longer get the most relevant first, so you get all the loose matches in date order.

My fix was a client-side --title-word check: all the words must be in the title. Rejected works are not marked as seen, in case you loosen the check later.

8. Small normalisation traps

ISSNs. issn:0028-0836, issn:00280836 and Nature's electronic ISSN issn:1476-4687 all returned the same 88 works for 25 October to 1 November. Normalise user input to NNNN-NNNC anyway, with an uppercase X check digit.
DOIs are case-insensitive. /works/10.1038/S41586-026-11328-3 returns the lower-case DOI. Store them lower-cased, or your dedupe will see two works.
Dates come as date-parts. That means [[2026]], [[2026, 10]] or [[2026, 10, 25]]. Keep the precision you were given rather than inventing a day.
Titles and ISSNs are lists. A work can have no title or several ISSNs. I join ISSNs with |.
Overlapping windows return duplicates by design. Dedupe by DOI before you write, both within a run and across runs.

9. Finding the IDs to filter on

You rarely know a member ID or funder ID by heart. The API has lookups. They are fuzzy searches, so the best match is not always first; check the names in the results:

/journals?query=nature gives the ISSNs, used as issn:.
/members?query=springer gives member IDs, used as member:.
/funders?query=national science foundation gives funder IDs; prefix the id with 10.13039/ for funder:.

What the feed looks like

The first run for Nature over seven days found 88 works with 2 requests (robots.txt plus one page). The second run found 0 new. On the broad side, all journal articles since midnight with --max-works 1000 stopped at 1,000. The next run carried on and wrote the next 1,000, rather than the same ones again.

doi,type,title,journal,issn,publisher,member,published,created,url,first_seen
10.1038/s41586-026-11328-3,journal-article,ctDNA clearance and outcomes for resectable lung cancer from NeoCOAST-2,Nature,0028-0836|1476-4687,Springer Science and Business Media LLC,297,2026-10-25,2026-10-25T07:01:31Z,https://doi.org/10.1038/s41586-026-11328-3,2026-11-01T16:37:25Z

The GitHub Actions workflow runs it once a day and commits the CSVs. One catch: GitHub turns off scheduled workflows in public repositories after 60 days without activity.

Code, tests (offline with saved Crossref responses, plus a few live checks) and the workflow are MIT licensed: https://github.com/fecklesspleb/crossref-new-works-feed

Numbers in this post come from requests I made on November 1, 2026; Crossref's counts and limits change, so check the response headers on your own requests.

Reading Workable, Recruitee and Teamtailor job boards from their public JSON

Scott Williams — Sun, 01 Nov 2026 14:57:53 +0000

AI disclosure: this post was drafted with AI assistance. I ran every request and code sample below myself on November 1, 2026 and checked the output.

Greenhouse, Lever and Ashby get most of the attention when people talk about reading job boards. Plenty of companies, especially in Europe, use Workable, Recruitee or Teamtailor instead. All three publish the jobs on a company's career page as JSON, with no key, because the career page itself needs them. This post covers what each endpoint looks like, how they differ, and what to watch for before you poll them every day.

Workable

GET https://apply.workable.com/api/v1/widget/accounts/{account}?details=false

The account name is the path segment in the career page URL: apply.workable.com/huggingface/ is account huggingface. The response is one object with name, description and jobs, and the whole board comes back in one response, with no paging.

A job, trimmed:

{
  "title": "Open-Source Machine Learning Engineer - EMEA Remote",
  "shortcode": "81B46579FE",
  "employment_type": "Full-time",
  "telecommuting": true,
  "department": "Open Source",
  "url": "https://apply.workable.com/j/81B46579FE",
  "published_on": "2026-05-29",
  "city": "Paris",
  "country": "France",
  "locations": [{"country": "France", "countryCode": "FR", "city": "Paris", "region": "Île-de-France", "hidden": false}]
}

Notes:

shortcode is the stable id, and it is also the last part of url.
published_on and created_at are dates with no time.
Remote is the boolean telecommuting. There is no salary field.
An unknown account answers 404 with a plain-text Not Found body, not JSON.

Rate limits are the real issue. apply.workable.com sits behind Cloudflare (the response headers include server: cloudflare). From a cloud server I have been answered with HTTP 429 and Cloudflare's "error code: 1015" page. On the day I wrote this the same requests went through. Either way, space your requests out, and when you get a 429, stop hitting the host for that run instead of retrying in a loop.

Recruitee

GET https://{company}.recruitee.com/api/offers/

The response is {"offers": [...]}, again the whole board with no paging. bunq's board had 25 offers. With descriptions included it was 731,785 bytes, so expect large responses.

An offer, trimmed:

{
  "id": 2800490,
  "slug": "data-scientist-marketing-3",
  "title": "Data Scientist Marketing",
  "company_name": "bunq",
  "location": "Amsterdam, Noord-Holland, Netherlands",
  "remote": false,
  "hybrid": true,
  "on_site": false,
  "status": "published",
  "published_at": "2026-10-28 13:21:58 UTC",
  "careers_url": "https://careers.bunq.com/o/data-scientist-marketing-3",
  "employment_type_code": "fulltime_permanent",
  "department": "Growth",
  "salary": {"min": null, "max": null, "period": null, "currency": null}
}

Notes:

Work mode comes as three booleans: remote, hybrid and on_site.
published_at is not ISO 8601. It uses a space and a literal UTC, so parse it with a format string.
careers_url points at the company's own domain when it has one (careers.bunq.com, not bunq.recruitee.com).
The salary object exists even when every value in it is null.
An unknown company answers 404 with {"error":"Not Found"}.

A token is coming. Recruitee's Careers Site API documentation says the deadline for adding an X-Careers-Sites-Token header to these calls is 10 February 2027, and that calls without it will return 401 Unauthorized. The token is set per company, so a reader of other companies' boards will not have one. The same page says the XML offer feed is not covered by the token:

GET https://{company}.recruitee.com/api/feeds/offers.xml

On November 1 that feed returned the same 25 offers for bunq, with the same fields as <offer> elements. Booleans are the text true/false, and locations are nested as <locations><location>. If you parse it, refuse documents with a DOCTYPE or entities, since you are parsing XML from servers you do not control.

Teamtailor

GET https://{company}.teamtailor.com/jobs.json

This is a JSON Feed 1.1 document, served as application/feed+json: version, title, home_page_url, feed_url and items. Each item has id, title, url, date_published, content_html and a _jobposting object, which is a schema.org JobPosting:

{
  "id": "0817d5ec-0e95-46f8-8baf-1a566f2ecc35",
  "title": "Principal Consultant - Generalist",
  "url": "https://proxima.teamtailor.com/jobs/8630029-principal-consultant-generalist",
  "date_published": "2026-10-28T11:14:46+00:00",
  "_jobposting": {
    "@type": "JobPosting",
    "identifier": {"@type": "PropertyValue", "name": "Proxima", "value": "8630029"},
    "datePosted": "2026-10-28T11:14:46+00:00",
    "employmentType": "FULL_TIME",
    "hiringOrganization": {"@type": "Organization", "name": "Proxima"},
    "jobLocation": [{"@type": "Place", "address": {"addressLocality": "Düsseldorf", "addressCountry": "DE"}}]
  }
}

Notes:

There are two ids. id is a UUID, and _jobposting.identifier.value is the number in the job URL (it matched the URL on every item I checked). Pick one and keep it.
Some postings carry _jobposting.baseSalary. On Teamtailor's own board, 3 of 17 did, in SEK per month.
There is no remote/hybrid field in the JSON. The RSS version, /jobs.rss, has a <remoteStatus> element (none, hybrid, onsite on the boards I read), so use that if you need work mode.
An empty board answers 200 with zero items. An unknown subdomain answers 404.

Pagination exists but is easy to miss. Without parameters, Proxima's 40 jobs and SATS's 57 came back in one response. With ?per_page=15, the response includes a next_url while more pages remain:

https://proxima.teamtailor.com/jobs.json?page=2&per_page=15

Pages 1 to 3 held 15, 15 and 10 items. The last page had no next_url, and page 4 returned zero items. I could not find the default page size; it is at least 57. So follow next_url until it is missing, and do not assume one response is the whole board.

Teamtailor career sites can also live on a company's own domain. tibber.teamtailor.com/jobs redirects to jobs.tibber.com/jobs, and /jobs.json returned the same feed on both hosts, with feed_url set to whichever host you asked.

robots.txt and signals

All three publish a robots.txt worth reading before you poll:

apply.workable.com allows everything for User-agent: * and adds a Content-Signal: search=yes, ai-input=yes, ai-train=no line.
{company}.recruitee.com/robots.txt redirects to the company's career domain. bunq's allows /, and Tellent's (jobs.recruitee.com) disallows only /v/. For a company name that does not exist, the redirect lands on an HTML page, not a robots file.
*.teamtailor.com blocks one named crawler entirely and, for everyone else, disallows paths like /app/ and /jobs/internal/, not /jobs.json. The Content-Signal line varies by company. Most boards I read had search=yes, but one had search=no, ai-input=no, so check each board rather than the platform.

For comparison, SmartRecruiters' API host (api.smartrecruiters.com) disallows everything for User-agent: * and allows only LinkedInBot. That is why it is not in this post.

A minimal reader

Standard library only. It turns each board into (uid, title, url) with a key that will not change between runs:

import json
import time
import urllib.request

UA = "my-jobs-reader/1.0"  # name your script; add a contact URL or email if you have one


def get_json(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept": "application/json"})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)


def workable(account):
    d = get_json(f"https://apply.workable.com/api/v1/widget/accounts/{account}?details=false")
    for j in d["jobs"]:
        yield f"workable:{account}:{j['shortcode']}", j["title"], j["url"]


def recruitee(company):
    d = get_json(f"https://{company}.recruitee.com/api/offers/")
    for o in d["offers"]:
        yield f"recruitee:{company}:{o['id']}", o["title"], o["careers_url"]


def teamtailor(company):
    url = f"https://{company}.teamtailor.com/jobs.json?per_page=15"
    while url:
        d = get_json(url)
        for item in d["items"]:
            job_id = item["_jobposting"]["identifier"]["value"]
            yield f"teamtailor:{company}:{job_id}", item["title"], item["url"]
        url = d.get("next_url")
        time.sleep(1)


for reader, name in [(workable, "huggingface"), (recruitee, "bunq"), (teamtailor, "proxima")]:
    jobs = list(reader(name))
    print(f"{reader.__name__}:{name}: {len(jobs)} jobs, first: {jobs[0]}")
    time.sleep(5)

Output on November 1:

workable:huggingface: 6 jobs, first: ('workable:huggingface:81B46579FE', 'Open-Source Machine Learning Engineer - EMEA Remote', 'https://apply.workable.com/j/81B46579FE')
recruitee:bunq: 25 jobs, first: ('recruitee:bunq:2800490', 'Data Scientist Marketing', 'https://careers.bunq.com/o/data-scientist-marketing-3')
teamtailor:proxima: 40 jobs, first: ('teamtailor:proxima:8630029', 'Principal Consultant - Generalist', 'https://proxima.teamtailor.com/jobs/8630029-principal-consultant-generalist')

This is the happy path. A version you can leave running needs a few more things:

checks robots.txt for each host
turns a 429 into "skip this host today"
falls back to Recruitee's XML feed on a 401
normalises the three shapes into one schema
keeps only jobs it has not seen before

Field mapping at a glance

	Workable	Recruitee	Teamtailor
Stable id	`shortcode`	`id`	`_jobposting.identifier.value`
Posted	`published_on` (date)	`published_at` (`... UTC`)	`date_published` (ISO 8601)
Work mode	`telecommuting`	`remote` / `hybrid` / `on_site`	only in `jobs.rss`
Employment type	`Full-time`	`fulltime_permanent`	`FULL_TIME`
Paging	none	none	`next_url`
Unknown board	404 text	404 JSON	404, empty body

The version I leave running is in my repo, fecklesspleb/careers-page-jobs-feed, under the MIT license. It does everything in that list, and adds offline tests against saved responses and a GitHub Actions workflow that runs it daily and commits the new jobs to CSV.

This article was created with the help of AI. The endpoints, numbers and code were tested by me before publishing.

Only the new jobs: a daily Greenhouse, Lever and Ashby feed with Python and GitHub Actions

Scott Williams — Sat, 31 Oct 2026 14:07:36 +0000

AI note: this post was drafted with AI assistance. I ran every request and code sample below myself on October 31, 2026 and checked the output.

In an earlier post I went through the public JSON endpoints behind Greenhouse, Lever and Ashby career pages. Reading a board once is easy. The useful version is a feed: check the same boards every morning and see only the postings that appeared since yesterday.

That turns out to be mostly a state problem, not a scraping problem. This post builds the feed in about 80 lines of standard-library Python, runs it on a GitHub Actions schedule, and covers the details that make a daily diff go wrong.

The plan

A text file lists the boards to watch, one ats:board per line.
Each run reads every board in one request.
Every job gets a key that will not change: ats:board:id.
data/jobs.csv holds every key ever seen. A job is new when its key is not in that file.
New jobs go to data/new_jobs.csv and are appended to data/jobs.csv.
GitHub Actions runs it daily and commits the two CSVs.

The repo keeps its own history, so there is no database and nothing to host.

Pick a key that survives

The tempting key is the job URL. Do not use it for Greenhouse. On the day I checked, all 202 jobs on Airbnb's Greenhouse board had an absolute_url on careers.airbnb.com, not on Greenhouse:

https://careers.airbnb.com/positions/7532824?gh_jid=7532824

A company that moves its careers site, or changes how it links to jobs, changes every URL at once, and your feed reports the whole board as new. The numeric id stays the same.

Titles are not keys either. On the same day, 39 of Airbnb's 202 jobs shared a title with another Airbnb job, and Spotify's Lever board had 89 postings but only 80 distinct titles ("Account Executive - Commercial, Spotify Technology" appeared three times, once per location). Each of those is a separate posting with its own ID.

IDs are only unique within one ATS and one board, so the key carries both: greenhouse:airbnb:7532824, lever:spotify:<uuid>, ashby:ramp:<uuid>.

The script

boards.txt:

greenhouse:airbnb
lever:spotify
ashby:ramp

jobs_feed.py:

import csv, json, sys, time, urllib.error, urllib.request
from datetime import date
from pathlib import Path

UA = "jobs-feed/1.0 (+https://github.com/fecklesspleb/ats-new-jobs-feed)"
SEEN = Path("data/jobs.csv")
NEW = Path("data/new_jobs.csv")
COLS = ["uid", "company", "title", "location", "url", "first_seen"]


def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept": "application/json"})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)


def greenhouse(board):
    for j in get(f"https://boards-api.greenhouse.io/v1/boards/{board}/jobs")["jobs"]:
        yield f"greenhouse:{board}:{j['id']}", j["title"], j["location"]["name"], j["absolute_url"]


def lever(site):
    data = get(f"https://api.lever.co/v0/postings/{site}?mode=json")
    if not isinstance(data, list):  # unknown sites answer {"ok": false, ...}
        raise ValueError(f"lever:{site}: {data}")
    for j in data:
        yield f"lever:{site}:{j['id']}", j["text"], j["categories"].get("location", ""), j["hostedUrl"]


def ashby(board):
    for j in get(f"https://api.ashbyhq.com/posting-api/job-board/{board}")["jobs"]:
        if j.get("isListed", True):
            yield f"ashby:{board}:{j['id']}", j["title"].strip(), j["location"], j["jobUrl"]


READERS = {"greenhouse": greenhouse, "lever": lever, "ashby": ashby}


def main(boards, keywords):
    seen = set()
    if SEEN.exists():
        with SEEN.open(newline="", encoding="utf-8") as f:
            seen = {row["uid"] for row in csv.DictReader(f)}
    new, failed = [], []
    for entry in boards:
        ats, board = entry.split(":", 1)
        try:
            for uid, title, location, url in READERS[ats](board):
                if uid in seen or (keywords and not any(k in title.lower() for k in keywords)):
                    continue
                seen.add(uid)
                new.append({"uid": uid, "company": board, "title": title, "location": location,
                            "url": url, "first_seen": date.today().isoformat()})
        except (urllib.error.URLError, ValueError, KeyError) as e:
            failed.append(entry)
            print(f"skipped {entry}: {e}", file=sys.stderr)
        time.sleep(1)
    SEEN.parent.mkdir(exist_ok=True)
    first = not SEEN.exists()
    with SEEN.open("a", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, COLS)
        if first:
            w.writeheader()
        w.writerows(new)
    with NEW.open("w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, COLS)
        w.writeheader()
        w.writerows(new)
    print(f"{len(new)} new jobs, {len(failed)} boards failed")
    return 1 if failed and len(failed) == len(boards) else 0


if __name__ == "__main__":
    boards = [b.strip() for b in open("boards.txt") if b.strip() and not b.startswith("#")]
    keywords = [k.lower() for k in sys.argv[1:]]
    sys.exit(main(boards, keywords))

Run it with optional title keywords (matched case-insensitively against the title):

python3 jobs_feed.py engineer
python3 jobs_feed.py engineer

Output on October 31, 2026:

128 new jobs, 0 boards failed
0 new jobs, 0 boards failed

The first run found 128 jobs with "engineer" in the title across the three boards (out of 462 listed in total). The second run, a few seconds later, found nothing new, which is the point. data/new_jobs.csv starts like this:

uid,company,title,location,url,first_seen
greenhouse:airbnb:7532824,airbnb,"Engineering Manager, Guest & Host","London, United Kingdom",https://careers.airbnb.com/positions/7532824?gh_jid=7532824,2026-10-31

Substring matching is crude on purpose: "engineer" also matches "Engineering Manager". Use more specific phrases ("data engineer", "backend") if that is too broad.

Details that break a daily diff

A failed board must not look like an empty board. If you ever extend this to report closed jobs ("in yesterday's file, not in today's"), only compare boards that were read successfully. A timeout on one board would otherwise mark every job on it as closed, and the next good run would report them all as new again. The script above tracks failed boards separately for that reason.

404 and an empty list mean different things. An unknown Lever site returns HTTP 404 with {"ok": false, "error": "Document not found"}. A real Lever site with no openings returns 200 and []. Greenhouse and Ashby return 404 for unknown boards too. Companies change ATS more often than you would think: one of the Lever examples I planned to use for this post had moved off Lever and now returns 404. Log 404s loudly, because they mean the board name in your list is stale, not that the company stopped hiring.

skipped lever:plaid: HTTP Error 404: Not Found
skipped ashby:not-a-board-xyz: HTTP Error 404: Not Found
0 new jobs, 2 boards failed

Each board is one request, and some are large. Greenhouse and Ashby return the whole board in one response with no paging. Without descriptions, Airbnb's Greenhouse board was about 155 KB; with ?content=true it was about 2.9 MB. Ramp's Ashby board (171 jobs) was about 3.1 MB because Ashby includes descriptions by default. For a once-a-day diff you only need titles and IDs, so leave content=true off Greenhouse.

Clean the fields you compare. 14 of Ramp's 171 Ashby titles had leading or trailing spaces. That does not affect the key, but it does affect keyword matching and spreadsheet sorting, hence the .strip(). Ashby also has an isListed flag; the script skips jobs where it is false.

Be a polite client. Lever's robots.txt asks for Crawl-delay: 1, so the loop sleeps one second between boards. Send a User-Agent that says what the client is and where to find it. Postings change over hours and days, not minutes, so once or twice a day is plenty for a job feed.

Run it every day on GitHub Actions

.github/workflows/jobs.yml:

name: New jobs
on:
  schedule:
    - cron: "17 11 * * *"
  workflow_dispatch:
permissions:
  contents: write
jobs:
  feed:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python jobs_feed.py engineer "data scientist"
      - run: |
          git config user.name "github-actions[bot]"
          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git add data/
          git diff --cached --quiet || { git commit -m "New jobs $(date -u +%F)" && git push; }

permissions: contents: write lets the job push the updated CSVs. The git diff --cached --quiet || guard skips the commit on days with nothing new. I linted this file with actionlint and ran the commit step in a local repo: the first pass committed, the second was a no-op.

Two things about scheduled workflows worth knowing before you rely on one:

Schedules run on the default branch and can start late when GitHub is busy; the docs suggest avoiding the top of the hour, hence minute 17.
In a public repository, GitHub disables scheduled workflows after 60 days with no repository activity. This workflow only commits on days with new jobs, so a quiet watch list can switch itself off. Check the Actions tab now and then.

Results land in the repo, so data/new_jobs.csv is a one-click view of today's postings, and the commit history doubles as a log of when each job first appeared.

Where this stops working

This covers boards on Greenhouse, Lever and Ashby whose names you already know. It does not cover Workday, which needs a paged POST per tenant (I wrote that up separately), or ATSs whose terms do not allow automated reading. It also only reads what companies publish on their public boards, which is what those endpoints are for.

A tested version of the script, with offline tests against saved responses and an n8n workflow, is available under the MIT license. Its README lists more options.

Code is in my repo: https://github.com/fecklesspleb/ats-new-jobs-feed

This article was created with the help of AI. The endpoints, numbers and code were tested by me before publishing.

Reading Workday job postings as JSON, and getting past the 2,000-result ceiling

Scott Williams — Fri, 30 Oct 2026 14:16:00 +0000

AI disclosure: this post was drafted with AI assistance, and every request and code sample was run and its output checked on October 29, 2026.

Many large employers run their career sites on Workday. You can recognize them by the host: {company}.wd{N}.myworkdayjobs.com. The page is a JavaScript app, but the job list behind it is a plain JSON endpoint that the page calls itself. No API key, no login, no headless browser.

This post covers that endpoint, three limits that are not obvious until your numbers come out wrong, and a small standard-library Python reader that handles them.

1. From a career site URL to the API

A Workday career site URL has three parts you need:

https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite
        ^^^^^^ ^^^                   ^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^
        tenant pod                   locale site

Host: nvidia.wd5.myworkdayjobs.com. The wd5 part varies by company (wd1, wd3, wd5 ...), so keep the host exactly as it appears.
Tenant: the first label of the host, nvidia.
Site: the first path segment after an optional locale like en-US. One tenant often has several sites (external, internal, students, contractors).

The job list is a POST to:

POST https://{host}/wday/cxs/{tenant}/{site}/jobs
Content-Type: application/json

{"appliedFacets": {}, "limit": 20, "offset": 0, "searchText": ""}

The first page answers with four keys: total, jobPostings, facets and userAuthenticated. Each posting is short:

{
  "title": "Power Methodology and Analysis Engineer",
  "externalPath": "/job/China-Shanghai/Power-Methodology-and-Analysis-Engineer_JR2028881",
  "locationsText": "China, Shanghai",
  "postedOn": "Posted Today",
  "bulletFields": ["JR2028881"]
}

externalPath is the key to everything else. The public job page is https://{host}/{site}{externalPath}, and the full posting is a GET on the API base plus the same path:

GET https://{host}/wday/cxs/{tenant}/{site}{externalPath}

That returns jobPostingInfo with jobDescription (HTML), location, additionalLocations, timeType, jobReqId, startDate, postedOn and a few more, plus a top-level hiringOrganization with the hiring entity's name.

2. Three limits worth knowing

I tested these against NVIDIA, Salesforce, Intel, Accenture and Workday's own career site.

Page size is capped at 20. "limit": 20 works. "limit": 21 and "limit": 100 both returned HTTP 400. An empty body {} also works and gives you 20.

total is only on the first page. With offset: 0, NVIDIA returned total: 2000. With offset: 20, the same query returned total: 0. Read the total once and keep it; do not re-read it per page, or your loop ends after one page.

One query stops at 2,000. This is the one that silently loses data. NVIDIA's first page said total: 2000, but its facet counts (more on those below) added up to 2,719. Accenture's said 2,000 while its job family facet added up to 41,402. Paging past the cap does not help: offsets 2000, 2020, 2500 and 5000 all returned the first page again. If your loop is "fetch until offset >= total", a large employer gives you 2,000 jobs and no error.

Smaller sites are under the cap and page normally: Salesforce reported 1,566, Intel 802 and Workday's own site 395.

3. Getting past the cap with facets

The first page also returns facets, the same filters you see on the career site. Each facet has a facetParameter and a list of values with an id, a descriptor and a count. On NVIDIA, the facets were jobFamilyGroup, workerSubType, timeType and locationMainGroup (that last one is nested and has no counts at the top level).

Filtering by a facet value is just a different appliedFacets:

{"appliedFacets": {"jobFamilyGroup": ["<value id>"]}, "limit": 20, "offset": 0, "searchText": ""}

Each filtered query gets its own 2,000 ceiling. NVIDIA's jobFamilyGroup had 15 values, the biggest being Engineering with 1,730, then Sales with 320 and University Employment with 208. Every value is under 2,000, so paging through each value and merging the results by externalPath covers the whole board.

workerSubType would not work for this board: its "Regular Employee" bucket had 2,240. That is why the reader below looks for a facet whose every bucket is under the cap.

Check one more thing: that the buckets add up to the total. Accenture's site also returns a distance facet with five values ("10 miles" up to "Any Distance"), each with a count of 1. Every bucket is "under the cap", but the counts are not job counts, and an earlier version of my reader picked that facet and would have returned a partial list without complaint. Requiring the counts to sum to at least the first page's total rules it out.

4. The reader

Standard library only (I ran it on Python 3.13.5):

import json
import re
import time
import urllib.error
import urllib.parse
import urllib.request
import urllib.robotparser

UA = "workday-jobs-reader/1.0 (+you@example.com)"
PAGE = 20      # the largest page size the endpoint accepted in my tests
CAP = 2000     # the most results one query will page through


def parse_site(url):
    """https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite -> (host, tenant, site)"""
    u = urllib.parse.urlsplit(url)
    m = re.match(r"^([\w-]+)\.wd\d+\.myworkdayjobs\.com$", u.netloc, re.I)
    if not m:
        raise ValueError("not a myworkdayjobs.com URL")
    parts = [p for p in u.path.split("/") if p and not re.fullmatch(r"[a-z]{2}-[A-Z]{2}", p)]
    if not parts:
        raise ValueError("the URL needs the site segment after the host")
    return u.netloc.lower(), m.group(1).lower(), parts[0]


def site_allowed(host, site):
    """Is /<site>/ allowed for us in the tenant's robots.txt?"""
    req = urllib.request.Request(f"https://{host}/robots.txt",
                                 headers={"User-Agent": UA, "Accept": "text/plain"})
    try:
        with urllib.request.urlopen(req, timeout=15) as r:
            lines = r.read().decode("utf-8", "replace").splitlines()
    except urllib.error.HTTPError as e:
        if 400 <= e.code < 500:
            return True          # RFC 9309: no robots.txt means no restrictions
        raise                    # 5xx: assume disallowed, so stop here
    rp = urllib.robotparser.RobotFileParser()
    rp.parse(lines)
    return rp.can_fetch(UA, f"https://{host}/{site}/")


class Workday:
    def __init__(self, url, delay=0.25):
        self.host, self.tenant, self.site = parse_site(url)
        self.api = f"https://{self.host}/wday/cxs/{self.tenant}/{self.site}"
        self.delay = delay

    def _call(self, path, body=None):
        time.sleep(self.delay)
        req = urllib.request.Request(
            self.api + path,
            data=json.dumps(body).encode() if body is not None else None,
            headers={"User-Agent": UA, "Accept": "application/json",
                     "Content-Type": "application/json"},
        )
        with urllib.request.urlopen(req, timeout=30) as r:
            return json.load(r)

    def search(self, facets=None, text="", offset=0):
        return self._call("/jobs", {"appliedFacets": facets or {}, "limit": PAGE,
                                    "offset": offset, "searchText": text})

    def page_through(self, facets=None, text=""):
        first = self.search(facets, text)
        total = first["total"]           # only the first page carries the real total
        yield from first["jobPostings"]
        for offset in range(PAGE, min(total, CAP), PAGE):
            yield from self.search(facets, text, offset)["jobPostings"]

    def all_jobs(self):
        self.skipped = []                # list entries with no externalPath (see below)
        first = self.search()
        if first["total"] < CAP:
            queries = [self.page_through()]
        else:
            # Too many to page through: split on a facet whose every bucket is under
            # the cap and whose buckets add up to at least the total (see below).
            for facet in first["facets"]:
                values = facet.get("values") or []
                counts = [v.get("count", CAP) for v in values]
                if values and max(counts) < CAP and sum(counts) >= first["total"]:
                    break
            else:
                raise RuntimeError("no facet splits this board under the cap")
            queries = (self.page_through({facet["facetParameter"]: [v["id"]]}) for v in values)
        seen = set()
        for query in queries:
            for job in query:
                path = job.get("externalPath")
                if not path:             # some entries carry only bulletFields
                    self.skipped.append(job)
                elif path not in seen:
                    seen.add(path)
                    yield job

    def detail(self, job):
        return self._call(job["externalPath"])["jobPostingInfo"]

And the run:

wd = Workday("https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite")
print("robots.txt allows the site:", site_allowed(wd.host, wd.site))
print("first page total:", wd.search()["total"])
jobs = list(wd.all_jobs())
print("jobs after facet split:", len(jobs))
print("entries without externalPath:", wd.skipped)
info = wd.detail(jobs[0])
print(info["title"], "|", info["jobReqId"], "|", info["postedOn"], "|", info["startDate"])

robots.txt allows the site: True
first page total: 2000
jobs after facet split: 2718
entries without externalPath: [{'bulletFields': ['JR2024815']}]
Power Methodology and Analysis Engineer | JR2028881 | Posted Today | 2026-10-29

That took 147 list requests (146 of them inside all_jobs) and about 2 minutes 40 seconds with a quarter-second pause between requests. The full descriptions are one more request per job, so fetch details only for the jobs that pass your title or location filter.

If you want this on a schedule, workday-jobs-feed is my free, MIT-licensed version of the reader with a daily GitHub Actions run: it splits sites over the cap by up to three facets (the two-facet case from the next section included), dedupes by job ID, and writes only new jobs to CSV.

5. Things that bite

Dates in the list are relative. postedOn is "Posted Today", "Posted Yesterday", "Posted 6 Days Ago" or "Posted 30+ Days Ago". The detail response has startDate as an ISO date, which is the one to store. If you only have the list, an approximation is fine for "posted in the last week" filters:

  import re
  from datetime import date, timedelta


  def approx_posted(posted_on, today=None):
      """'Posted 3 Days Ago' -> a date. '30+ Days Ago' only says 'at least 30 days'."""
      today = today or date.today()
      t = (posted_on or "").lower()
      if "today" in t:
          return today
      if "yesterday" in t:
          return today - timedelta(days=1)
      m = re.search(r"(\d+)\+?\s+days?\s+ago", t)
      return today - timedelta(days=int(m.group(1))) if m else None

Two caveats from my runs. "30+ Days Ago" is a floor, not a date: one such job had a startDate 31 days back. And the label does not say which time zone "today" is in, so near midnight an estimate from your own clock can be a day off. When the exact day matters, store startDate from the detail response.

locationsText can be a count. Multi-location jobs say "3 Locations" or similar. The real list is location plus additionalLocations in the detail response.
The request ID is in bulletFields. On the sites I tried, bulletFields[0] was the requisition ID (JR2028881), and the detail response has it as jobReqId. Use it, or externalPath, as the stable ID for "new since last run" checks.
A wrong site name is a 404. A misspelled site returned HTTP 404 from the jobs endpoint, so a typo fails loudly. Copy the site segment from the browser rather than guessing it.
Some list entries have no externalPath. In the NVIDIA crawl, one entry in the Engineering bucket was just {"bulletFields": ["JR2024815"]}. My first version indexed job["externalPath"] directly and crashed with a KeyError a minute in. The reader now sets such entries aside in wd.skipped so you can see how many there were.
The sitemap is not a shortcut. Tenants list a siteMap.xml per site in robots.txt. NVIDIA's had 100 URLs, against 2,718 open jobs.
Very large tenants need two facets. On Accenture's site, the biggest job family bucket alone had 19,364 jobs and no facet split the board under 2,000 (the reader above raises its error there). There you have to combine two facets, such as job family and location, in one appliedFacets.

6. Be a polite client

Workday tenants publish a robots.txt, and it is specific about which sites are public. Pfizer's, for example:

User-agent: *
Allow: /PfizerCareers/
Allow: /ContractorConversions/
Allow: /Canada_Only_External/
Allow: /Costa_Rica_External/
Disallow: /InternalJobs/
Disallow: /refreshFacet/

The API path (/wday/cxs/...) is not listed, so a plain robots check on the API URL says "allowed" even for the internal site. The useful check is on the site path, /{site}/, which is what site_allowed does. For Pfizer it returns False for InternalJobs and True for PfizerCareers.

One quirk: ask for robots.txt with Accept: text/plain. When I asked for it with an Accept header that preferred JSON or HTML, the tenant answered with the HTML app shell instead of the robots file, and a parser reading that would find no rules and allow everything.

Beyond robots.txt:

Send a descriptive User-Agent with a contact address.
Keep it slow. One request at a time with a short pause is plenty; a full NVIDIA crawl was 146 list requests.
Once a day is enough for almost any job-tracking use. Fetch descriptions only for the jobs you will keep.
Some employers' own site terms say more than robots.txt does. Read them for the companies you track, and only collect what is already public on the external career site.

Diagnose a stuck Windows Update with PowerShell: services, pending restarts, update cache and event-log errors, read-only first

Scott Williams — Fri, 30 Oct 2026 14:02:35 +0000

AI disclosure: this post was written with AI assistance. The code was syntax-checked and linted, and its logic was tested with simulated services, registry values and event-log entries, but it has not yet been run on Windows: not on a real Windows 11 PC, and not on Windows PowerShell 5.1. Read it before you run it, and try the reset with -WhatIf first.

When an update sits at 0% for hours or fails with the same code every night, Microsoft's troubleshooting page has a long list of things to try. Most are quick, but it helps to know which one applies before you start renaming system folders. This post builds a small read-only health check in PowerShell that answers five questions:

Are the update services able to run?
Is a restart already pending?
How big is the update cache?
Is there enough free space?
What errors has Windows Update logged recently, and with which codes?

Then, only if you need it, there's one repair: the cache reset from Microsoft's manual steps, wrapped so it asks first and supports -WhatIf.

Before any of it, try the built-in option: Settings > System > Troubleshoot > Other troubleshooters, then Run next to Windows Update.

Step 1: the services

Microsoft's manual reset steps stop and start two services, bits and wuauserv (Windows Update). Its broader reset guidance also involves cryptsvc. Get-Service gives you each one's Status and StartType:

function Get-UpdateServiceState {
    [CmdletBinding()]
    param([string[]] $Name = @('bits', 'wuauserv', 'cryptsvc'))

    foreach ($n in $Name) {
        $svc = Get-Service -Name $n -ErrorAction SilentlyContinue
        if (-not $svc) {
            [pscustomobject]@{ Name = $n; Status = 'NotFound'; StartType = $null; Note = 'Service not found.' }
            continue
        }
        $note = if ("$($svc.StartType)" -eq 'Disabled') {
            'Disabled: it cannot start. Microsoft lists error 0x80070422 for a disabled or stopped Windows Update service (on its Windows 10 troubleshooting tab).'
        }
        elseif ("$($svc.Status)" -eq 'Stopped') {
            'Stopped. Not a problem by itself if the start type lets it start when needed.'
        }
        else { 'OK' }
        [pscustomobject]@{ Name = $svc.Name; Status = "$($svc.Status)"; StartType = "$($svc.StartType)"; Note = $note }
    }
}

Get-UpdateServiceState | Format-Table -AutoSize

The important distinction is Stopped versus Disabled. Set-Service's documentation describes a Manual service as one that's started by a user, the Service Control Manager or an application when needed, so seeing it stopped is normal. A Disabled service can't be started at all. For 0x80070422, Microsoft's fix is to open services.msc, set the Windows Update service's Startup type to Automatic, start it if it's stopped, then restart and check for updates again.

Step 2: is a restart pending?

There's no single "restart pending" flag, so check several signals and report each one separately:

function Get-WuaRebootRequired {
    [CmdletBinding()]
    [OutputType([bool])]
    param()
    try { [bool](New-Object -ComObject Microsoft.Update.SystemInfo).RebootRequired }
    catch { $null }
}

function Test-PendingReboot {
    [CmdletBinding()]
    param()
    $sm = Get-ItemProperty -LiteralPath 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager' `
        -Name 'PendingFileRenameOperations' -ErrorAction SilentlyContinue

    $signals = [ordered]@{
        WindowsUpdateAgent          = Get-WuaRebootRequired
        ComponentBasedServicing     = Test-Path -LiteralPath 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending'
        WindowsUpdateAutoUpdate     = Test-Path -LiteralPath 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired'
        PendingFileRenameOperations = [bool]($sm -and $sm.PendingFileRenameOperations)
    }
    $out = [ordered]@{ RebootPending = @($signals.Values | Where-Object { $_ -eq $true }).Count -gt 0 }
    foreach ($k in $signals.Keys) { $out[$k] = $signals[$k] }
    [pscustomobject]$out
}

Test-PendingReboot

Where each signal comes from, and how much to trust it:

WindowsUpdateAgent is the Windows Update Agent's own ISystemInformation.RebootRequired. Microsoft documents it as "a restart is required to complete the installation or uninstallation of one or more updates". This is the most direct answer.
PendingFileRenameOperations is where MoveFileEx records files to be replaced at the next restart. Any installer can use it, not just Windows Update, so True here means "something wants a restart".
The two registry keys come from a Microsoft Scripting Blog post on detecting pending restarts, not from Microsoft's reference docs. They're useful, but treat them as hints.

If any of them is True, restart first and check for updates again before you try anything else. Microsoft's troubleshooting page has restarting as its own step, because many updates need one to finish.

Step 3: the update cache and free space

%SystemRoot%\SoftwareDistribution holds Windows Update's working files, and Download inside it holds what has been downloaded. The size alone doesn't tell you it's broken, but it's useful context before a reset, and it shows any .bak folders left over from earlier resets. The full script measures it with the same link-safe folder walk as the disk-space-audit repo.

Free space matters more. Microsoft says a device needs at least 16 GB free to upgrade a 32-bit OS, or 20 GB for a 64-bit OS, and lists 0x800f0922 as "unable to install due to insufficient disk space" (the Windows 11 table adds .NET Framework or other component issues):

function Get-SystemDriveSpace {
    [CmdletBinding()]
    param([string] $SystemDrive = $env:SystemDrive, [int] $RecommendedFreeGB = 20)
    $d = Get-CimInstance -ClassName Win32_LogicalDisk -Filter "DeviceID = '$SystemDrive'" -ErrorAction Stop
    $free = [math]::Round($d.FreeSpace / 1GB, 1)
    [pscustomobject]@{ Drive = $SystemDrive; FreeGB = $free; Enough = $free -ge $RecommendedFreeGB }
}

Step 4: recent Windows Update errors

Microsoft's guidance is to look for Windows Update Agent events in the System log (Event Viewer also has a separate WindowsUpdateClient > Operational log). They come from the Microsoft-Windows-WindowsUpdateClient source, and its example of a failed install is an Error-level event with ID 20 and the error code in the message. Get-WinEvent -FilterHashtable filters on the log, the provider and the level (2 = Error) in one query:

function Get-RecentUpdateError {
    [CmdletBinding()]
    param([int] $Days = 30, [int] $MaxEvents = 20)
    $filter = @{
        LogName      = 'System'
        ProviderName = 'Microsoft-Windows-WindowsUpdateClient'
        Level        = 2
        StartTime    = (Get-Date).AddDays(-$Days)
    }
    try {
        $events = Get-WinEvent -FilterHashtable $filter -MaxEvents $MaxEvents -ErrorAction Stop
    }
    catch {
        if ($_.FullyQualifiedErrorId -like 'NoMatchingEventsFound*' -or $_.Exception.Message -like '*No events were found*') { return }
        Write-Warning "Could not read the System event log: $($_.Exception.Message)"
        return
    }
    foreach ($e in $events) {
        $code = if ($e.Message -match '0x[0-9A-Fa-f]{8}') { $Matches[0] } else { $null }
        [pscustomobject]@{ TimeCreated = $e.TimeCreated; Id = $e.Id; ErrorCode = $code; Message = $e.Message }
    }
}

Get-RecentUpdateError -Days 30 | Format-List TimeCreated, Id, ErrorCode, Message

Two practical notes:

Get-WinEvent reports "no events found" as an error. That's good news, not a failure, so the function returns nothing instead of a wall of red.
Pulling out the 0x... code lets you look it up straight away in Microsoft's error tables. Two from Microsoft's table: 0x80070422 (Windows Update service is disabled or stopped) and 0x800f0922 (unable to install due to insufficient disk space).

For more detail, Get-WindowsUpdateLog merges Windows Update's trace files into a readable WindowsUpdate.log. It's a static copy, so run it again for newer entries. Servicing details are in %systemroot%\Logs\CBS\CBS.log.

Step 5 (opt-in): reset the update cache

If the services are fine, you've restarted, there's enough space and updates still fail, Microsoft's manual steps are to stop bits and wuauserv, rename %systemroot%\SoftwareDistribution to SoftwareDistribution.bak, start the services again, and restart. Here's that sequence as a function that asks first:

function Reset-UpdateCache {
    [CmdletBinding(SupportsShouldProcess, ConfirmImpact = 'High')]
    param([string] $WindowsPath = $env:SystemRoot)

    $sd = Join-Path -Path $WindowsPath -ChildPath 'SoftwareDistribution'
    if (-not (Test-Path -LiteralPath $sd -PathType Container)) { throw "Folder not found: $sd" }
    $newName = 'SoftwareDistribution.bak-{0}' -f (Get-Date -Format 'yyyyMMdd-HHmmss')

    if (-not $PSCmdlet.ShouldProcess($sd, "Stop bits and wuauserv, rename to $newName, start them again")) { return }
    $id = [Security.Principal.WindowsIdentity]::GetCurrent()
    if (-not ([Security.Principal.WindowsPrincipal]$id).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) {
        throw 'Run PowerShell as administrator to use Reset-UpdateCache.'
    }

    try {
        foreach ($s in 'wuauserv', 'bits') { Stop-Service -Name $s -ErrorAction Stop -Confirm:$false }
        Rename-Item -LiteralPath $sd -NewName $newName -ErrorAction Stop -Confirm:$false
    }
    finally {
        foreach ($s in 'bits', 'wuauserv') {
            try { Start-Service -Name $s -ErrorAction Stop -Confirm:$false }
            catch { Write-Warning "Could not start ${s}: $($_.Exception.Message)" }
        }
    }
    [pscustomobject]@{ RenamedTo = Join-Path -Path $WindowsPath -ChildPath $newName }
}

What makes it safe to run:

Rename, not delete. The old cache stays on disk until you remove it, so you can put it back.
ConfirmImpact = 'High' means it asks before doing anything, even without -Confirm. -WhatIf prints what it would do and touches nothing.
-ErrorAction Stop plus finally. If a service won't stop, the rename never happens. Whether the rename works or fails, the services are started again.
A timestamp in the backup name, so a second reset doesn't collide with an older .bak.
No -Force on Stop-Service, so it won't quietly stop other services that depend on these two.

In an elevated PowerShell window:

Reset-UpdateCache -WhatIf
Reset-UpdateCache

Then restart and go to Settings > Windows Update > Check for updates.

To undo, stop the two services, move the new folder aside, rename your backup back to SoftwareDistribution, and start them again:

Stop-Service -Name wuauserv, bits
Rename-Item -LiteralPath "$env:SystemRoot\SoftwareDistribution" -NewName 'SoftwareDistribution.new'
Rename-Item -LiteralPath "$env:SystemRoot\SoftwareDistribution.bak-20261030-101500" -NewName 'SoftwareDistribution'
Start-Service -Name bits, wuauserv

Use the folder name the function printed. Once updates work again, you can delete the .bak-... folder to get the space back.

If that doesn't fix it

Microsoft's next step for corrupted system files is to repair the image and then check the system files, in this order, in an elevated window:

DISM /Online /Cleanup-Image /RestoreHealth
sfc /scannow

Microsoft's manual steps also rename catroot2. This script leaves it alone on purpose to keep the change small. If you need it, follow the steps on Microsoft's page directly.

Limits, honestly

The code was tested with PowerShell 7 on Linux, with every Windows command (services, registry, event log, the Windows Update COM object) replaced by fakes. PSScriptAnalyzer reported nothing, and Pester tests of each check, the -WhatIf path, the failure paths and the admin check passed. It has not been run on a real Windows PC or on Windows PowerShell 5.1. Try the reset with -WhatIf first.
The two registry restart keys come from a blog post, not reference docs, and PendingFileRenameOperations isn't specific to Windows Update.
Event messages are localized. The 0x... code extraction doesn't depend on language, but the "no events found" text check is a fallback for the error ID.
The full script, with its Pester tests, is at https://github.com/fecklesspleb/windows-update-health (MIT).

Sources (Microsoft)

Troubleshoot problems updating Windows (troubleshooter, restart, free space, error table, manual reset commands, DISM then SFC): https://support.microsoft.com/en-us/windows/deployment/updates-lifecycle/troubleshoot-problems-updating-windows
Additional resources for Windows Update (manual reset steps): https://learn.microsoft.com/en-us/troubleshoot/windows-client/installing-updates-features-roles/additional-resources-for-windows-update
Common Windows Update errors: https://learn.microsoft.com/en-us/troubleshoot/windows-client/installing-updates-features-roles/common-windows-update-errors
Windows Update troubleshooting guidance (Windows Update Agent events in the System log): https://learn.microsoft.com/en-us/troubleshoot/windows-server/installing-updates-features-roles/troubleshoot-windows-server-update-guidance
Windows Update error 0x80070002 (example System log event from Microsoft-Windows-WindowsUpdateClient): https://learn.microsoft.com/en-us/troubleshoot/windows-server/installing-updates-features-roles/troubleshoot-windows-update-error-0x80070002
ISystemInformation interface and RebootRequired: https://learn.microsoft.com/en-us/windows/win32/api/wuapi/nn-wuapi-isysteminformation and https://learn.microsoft.com/en-us/windows/win32/api/wuapi/nf-wuapi-isysteminformation-get_rebootrequired
MoveFileEx (PendingFileRenameOperations): https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-movefileexa
Determine pending reboot status, PowerShell style (Microsoft Scripting Blog): https://devblogs.microsoft.com/scripting/determine-pending-reboot-statuspowershell-style-part-1/
Windows Update log files: https://learn.microsoft.com/en-us/windows/deployment/update/windows-update-logs
Creating Get-WinEvent queries with FilterHashtable: https://learn.microsoft.com/en-us/powershell/scripting/samples/creating-get-winevent-queries-with-filterhashtable
Get-Service: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-service?view=powershell-7.5
Set-Service (startup types): https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/set-service
Win32_LogicalDisk class: https://learn.microsoft.com/en-us/windows/win32/cimwin32prov/win32-logicaldisk
Everything you wanted to know about ShouldProcess: https://learn.microsoft.com/en-us/powershell/scripting/learn/deep-dives/everything-about-shouldprocess

This article was created with the help of AI. Not affiliated with or endorsed by Microsoft.

Find what's filling your Windows drive with PowerShell: a read-only audit first, then a cleanup you can preview with -WhatIf

Scott Williams — Fri, 30 Oct 2026 14:01:41 +0000

AI disclosure: this post was written with AI assistance. The code was syntax-checked and linted, and its logic was tested with simulated drives and a throwaway test folder, but it has not yet been run on Windows: not on a real Windows 11 PC, and not on Windows PowerShell 5.1. Read it before you run it, and try the cleanup with -WhatIf first.

When the C: drive turns red, the first stop is Windows itself. Settings > System > Storage > Cleanup recommendations lists temporary files, large or unused files, files synced to the cloud and unused apps, and shows how much each would free. Storage Sense on the same page can clear temporary files and the Recycle Bin automatically. Use those first.

What Settings doesn't give you is a quick, scriptable answer to where did the space go? This post builds that answer in PowerShell, in two parts:

A read-only audit. Free space per drive, the biggest folders under your profile, and the size of the places that usually hold junk. It changes nothing.
One narrow cleanup. Delete files from your own TEMP folder that haven't changed in a week. It asks first, supports -WhatIf, and refuses to run if TEMP looks wrong.

Everything else (Downloads, the Recycle Bin, the Windows Update cache, Windows.old) is only measured. You decide what to do with those.

Step 1: free space per drive

Win32_LogicalDisk reports every logical disk. DriveType 3 means "Local Disk", and Size and FreeSpace are in bytes:

function Get-FixedDriveSpace {
    [CmdletBinding()]
    param()
    Get-CimInstance -ClassName Win32_LogicalDisk -Filter 'DriveType = 3' -ErrorAction Stop |
        ForEach-Object {
            $pct = if ($_.Size -gt 0) { [math]::Round(100 * $_.FreeSpace / $_.Size, 1) } else { $null }
            [pscustomobject]@{
                Drive       = $_.DeviceID
                SizeGB      = [math]::Round($_.Size / 1GB, 1)
                FreeGB      = [math]::Round($_.FreeSpace / 1GB, 1)
                PercentFree = $pct
            }
        }
}

Get-FixedDriveSpace | Format-Table -AutoSize

Only DriveType = 3 disks come back. Removable and network drives have other DriveType values, so they're left out.

Step 2: measure a folder without following links

The obvious one-liner is Get-ChildItem -Recurse | Measure-Object Length -Sum. It has a catch: junctions and symbolic links. A link can point at a folder somewhere else, so it gets counted twice or sends you into a loop. The rules also differ between versions: in PowerShell 6 and later, -Recurse doesn't follow symbolic links to directories unless you add -FollowSymlink, and Windows PowerShell 5.1 doesn't have that switch. So walk the tree yourself, and skip anything with the ReparsePoint attribute:

function Get-FolderSize {
    [CmdletBinding()]
    param([Parameter(Mandatory)] [string] $Path)

    $result = [pscustomobject]@{ Path = $Path; Bytes = [long]0; Files = 0; SkippedLinks = 0; Unreadable = 0 }
    if (-not (Test-Path -LiteralPath $Path -PathType Container)) { return $result }

    $stack = New-Object System.Collections.Generic.Stack[string]
    $stack.Push($Path)
    while ($stack.Count -gt 0) {
        $dir = $stack.Pop()
        $errs = $null
        $items = Get-ChildItem -LiteralPath $dir -Force -ErrorAction SilentlyContinue -ErrorVariable errs
        $result.Unreadable += @($errs).Count
        foreach ($item in $items) {
            $isLink = ($item.Attributes -band [System.IO.FileAttributes]::ReparsePoint) -ne 0
            if ($item.PSIsContainer) {
                if ($isLink) { $result.SkippedLinks++ } else { $stack.Push($item.FullName) }
            }
            elseif ($isLink) { $result.SkippedLinks++ }
            else { $result.Bytes += $item.Length; $result.Files++ }
        }
    }
    $result
}

Two details:

-ErrorAction SilentlyContinue -ErrorVariable errs keeps going past folders you can't read (some system folders need admin rights) and counts them, instead of filling the screen with red.
The total is the sum of file lengths. That can differ from "size on disk" in File Explorer, for example for compressed files or OneDrive files that are only stored online.

Then list the biggest folders directly under your profile:

function Get-LargestFolder {
    [CmdletBinding()]
    param([string] $Path = $env:USERPROFILE, [int] $Top = 10)
    Get-ChildItem -LiteralPath $Path -Directory -Force -ErrorAction Stop |
        Where-Object { ($_.Attributes -band [System.IO.FileAttributes]::ReparsePoint) -eq 0 } |
        ForEach-Object {
            $s = Get-FolderSize -Path $_.FullName
            [pscustomobject]@{ Folder = $_.FullName; SizeGB = [math]::Round($s.Bytes / 1GB, 2); Files = $s.Files; Bytes = $s.Bytes }
        } |
        Sort-Object -Property Bytes -Descending |
        Select-Object -First $Top
}

Get-LargestFolder -Top 10 | Format-Table Folder, SizeGB, Files -AutoSize

Run it again with -Path "$env:USERPROFILE\AppData\Local" (or any folder that stands out) to drill down. Measuring a big profile can take a few minutes.

Step 3: the usual suspects

These are the places Microsoft's own cleanup advice keeps coming back to:

Location	How the script finds it	What to do about it
Your TEMP folder	`$env:TEMP`	Cleanup recommendations > Temporary files, or Step 4 below
Windows Update download cache	`%SystemRoot%\SoftwareDistribution\Download`	Leave it to Cleanup recommendations or Disk Cleanup
Recycle Bin	Shell's Recycle Bin folder (see below)	Look inside, then `Clear-RecycleBin`
Downloads	`%USERPROFILE%\Downloads` (the default location)	Your files: sort by size and decide
Previous Windows install	`%SystemDrive%\Windows.old`	Removing it means you can't go back to the previous version

The Recycle Bin is a virtual folder, so there's no plain path to measure. The Shell object exposes it as special folder 10 (ssfBITBUCKET), and each item has a Size:

function Get-RecycleBinByteCount {
    [CmdletBinding()]
    [OutputType([long])]
    param()
    try {
        $bin = (New-Object -ComObject Shell.Application).NameSpace(10)
        $total = [long]0
        foreach ($i in $bin.Items()) { $total += [long]$i.Size }
        $total
    }
    catch {
        Write-Warning "Could not read the Recycle Bin size: $($_.Exception.Message)"
        $null
    }
}

The snippets here are trimmed; the repo version adds parameter validation and more output fields. The full script wraps all of this in one Get-CleanupCandidate function that prints a row per location with its size and advice:

Get-CleanupCandidate | Format-Table Location, SizeGB, Advice -Wrap

A note on Downloads: %USERPROFILE%\Downloads is only the default. Microsoft notes that known folders can be redirected, so if yours has moved, pass the real path with -DownloadsPath.

Step 4: clear old TEMP files, with a preview

This is the only part that deletes anything, so it has the most guard rails:

function Assert-SafeTempPath {
    [CmdletBinding()]
    param([string] $TempPath)
    if ([string]::IsNullOrWhiteSpace($TempPath)) { throw 'TEMP path is empty. Refusing to clean.' }
    if (-not (Test-Path -LiteralPath $TempPath -PathType Container)) { throw "TEMP folder not found: $TempPath" }
    $leaf = Split-Path -Path $TempPath.TrimEnd('\', '/') -Leaf
    if ($leaf -notin @('Temp', 'Tmp')) {
        throw "Refusing to clean '$TempPath': the folder is not named Temp or Tmp."
    }
}

function Get-OldTempFile {
    [CmdletBinding()]
    param([string] $TempPath = $env:TEMP, [ValidateRange(0, 3650)] [int] $OlderThanDays = 7)
    if ([string]::IsNullOrWhiteSpace($TempPath) -or -not (Test-Path -LiteralPath $TempPath -PathType Container)) { return }
    $cutoff = (Get-Date).AddDays(-$OlderThanDays)
    $stack = New-Object System.Collections.Generic.Stack[string]
    $stack.Push($TempPath)
    while ($stack.Count -gt 0) {
        foreach ($item in (Get-ChildItem -LiteralPath $stack.Pop() -ErrorAction SilentlyContinue)) {
            if (($item.Attributes -band [System.IO.FileAttributes]::ReparsePoint) -ne 0) { continue }
            if ($item.PSIsContainer) { $stack.Push($item.FullName) }
            elseif ($item.LastWriteTime -lt $cutoff) { $item }
        }
    }
}

function Remove-OldTempFile {
    [CmdletBinding(SupportsShouldProcess, ConfirmImpact = 'High')]
    param([ValidateRange(1, 3650)] [int] $OlderThanDays = 7, [string] $TempPath = $env:TEMP)

    Assert-SafeTempPath -TempPath $TempPath
    $files = @(Get-OldTempFile -TempPath $TempPath -OlderThanDays $OlderThanDays)
    $mb = [math]::Round((($files | Measure-Object -Property Length -Sum).Sum) / 1MB, 1)

    $deleted = 0; $skipped = 0
    $target = "$($files.Count) file(s) ($mb MB) in $TempPath not changed in $OlderThanDays days"
    if ($files.Count -gt 0 -and $PSCmdlet.ShouldProcess($target, 'Delete (no undo)')) {
        foreach ($f in $files) {
            try   { Remove-Item -LiteralPath $f.FullName -ErrorAction Stop -Confirm:$false; $deleted++ }
            catch { $skipped++ }   # usually a file a running program still has open
        }
    }
    [pscustomobject]@{ Found = $files.Count; Deleted = $deleted; Skipped = $skipped }
}

Why each piece is there:

SupportsShouldProcess gives the function -WhatIf and -Confirm. Nothing inside the if ($PSCmdlet.ShouldProcess(...)) block runs under -WhatIf.
ConfirmImpact = 'High' means PowerShell asks before deleting even when you don't pass -Confirm, because the default $ConfirmPreference is High. You confirm once for the whole batch. -Confirm:$false on Remove-Item stops it asking again for every file.
The name check is there because .NET's GetTempPath falls back to your profile folder, or even the Windows folder, when the TMP and TEMP variables are missing. A cleanup that trusts the variable blindly could be pointed at the wrong place. This one only runs on a folder actually named Temp or Tmp.
No link-following, for the same reason as Step 2. A junction inside TEMP can't lead the script out of TEMP.
No -Force on Remove-Item. Without -Force, PowerShell won't delete hidden or read-only files, and those are best left alone here.

Preview, then run:

Remove-OldTempFile -OlderThanDays 7 -WhatIf
Remove-OldTempFile -OlderThanDays 7

What about the Recycle Bin?

PowerShell already has Clear-RecycleBin, and it supports -WhatIf and -Confirm:

Clear-RecycleBin -DriveLetter C -WhatIf
Clear-RecycleBin -DriveLetter C

Open the Recycle Bin and look first: once it's emptied, the files are gone. Don't add -Force. Microsoft's documentation for Clear-RecycleBin says -Force also overrides -WhatIf and -Confirm, which defeats the whole point of previewing.

What not to touch by hand

C:\Windows\WinSxS. Microsoft says not to delete the WinSxS folder and to shrink it with tools built into Windows instead, for example Dism.exe /online /Cleanup-Image /StartComponentCleanup from an elevated prompt.
Windows.old. Remove it from Cleanup recommendations, and only once you're sure you won't roll back.
The Windows Update cache. Disk Cleanup and Cleanup recommendations handle system files. If Windows Update itself is stuck, follow Microsoft's reset steps rather than deleting the folder while updates are running.

Limits, honestly

The code was tested with PowerShell 7 on Linux, using fake drives, a fake Recycle Bin and real files in a throwaway test folder. PSScriptAnalyzer reported nothing, and Pester tests of the size maths, the link skipping, the -WhatIf path and the refusal checks passed. It has not been run on a real Windows PC or on Windows PowerShell 5.1. Try it with -WhatIf first.
Sizes are file lengths, not "size on disk".
There's no undo for the TEMP cleanup. That's why it's limited to old files in a folder meant for temporary files.
The full script, with its Pester tests, is at https://github.com/fecklesspleb/disk-space-audit (MIT).

Sources (Microsoft)

Free up drive space in Windows (Cleanup recommendations, Storage Sense, Disk Cleanup, Windows.old, emptying the Recycle Bin): https://support.microsoft.com/en-us/windows/experience/storage-filemanagement/free-up-drive-space-in-windows
Manage drive space with Storage Sense: https://support.microsoft.com/en-us/windows/experience/storage-filemanagement/manage-drive-space-with-storage-sense
Win32_LogicalDisk class: https://learn.microsoft.com/en-us/windows/win32/cimwin32prov/win32-logicaldisk
Get-ChildItem (-FollowSymlink, links and junctions): https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-childitem
Path.GetTempPath and the GetTempPath fallback order: https://learn.microsoft.com/en-us/dotnet/api/system.io.path.gettemppath and https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-gettemppathw
ShellSpecialFolderConstants (ssfBITBUCKET): https://learn.microsoft.com/en-us/windows/win32/api/shldisp/ne-shldisp-shellspecialfolderconstants
FolderItem.Size: https://learn.microsoft.com/en-us/windows/win32/shell/folderitem-size
KNOWNFOLDERID (Downloads): https://learn.microsoft.com/en-us/windows/win32/shell/knownfolderid
Remove-Item: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/remove-item?view=powershell-7.5
Clear-RecycleBin: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/clear-recyclebin?view=powershell-5.1
Clean up the WinSxS folder: https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/clean-up-the-winsxs-folder
Everything you wanted to know about ShouldProcess: https://learn.microsoft.com/en-us/powershell/scripting/learn/deep-dives/everything-about-shouldprocess
about_Preference_Variables ($ConfirmPreference defaults to High): https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_preference_variables?view=powershell-7.4

This article was created with the help of AI. Not affiliated with or endorsed by Microsoft.

Audit Windows startup entries with PowerShell without breaking anything: read-only first, -WhatIf, a JSON backup and an undo

Scott Williams — Thu, 29 Oct 2026 15:23:15 +0000

AI disclosure: this post was written with AI assistance. The code was syntax-checked and linted, and its logic was tested with simulated registry objects, but it has not yet been run on Windows: not against a real registry, and not on Windows PowerShell 5.1. Read it before you run it, and try it with -WhatIf first.

Settings > Apps > Startup and Task Manager's Startup apps tab are the right first stop when a Windows PC is slow after sign-in. Use them first. But sometimes something still starts at sign-in and you can't see where it comes from, or Settings shows an entry called "We couldn't find this app" left behind by something you uninstalled.

Microsoft documents where those entries live: the Run registry keys and the two Startup folders. This post builds three small PowerShell functions around them:

Get-StartupEntry: a read-only list. Changes nothing.
Disable-RunEntry: removes one Run value, but only after saving it to a JSON file, and it supports -WhatIf and -Confirm.
Restore-RunEntry: puts the value back from that JSON file.

The pattern (report first, preview with -WhatIf, back up, then change, and keep an undo) is worth copying for any script that touches the registry.

Where startup entries live

From Microsoft's startup apps page and the Run/RunOnce documentation:

Location	Scope
`HKCU\Software\Microsoft\Windows\CurrentVersion\Run`	Current user
`HKLM\Software\Microsoft\Windows\CurrentVersion\Run`	All users
`HKLM\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Run`	All users, 32-bit apps on 64-bit Windows
`...\RunOnce` (HKCU and HKLM)	Runs once at the next sign-in, then the value is deleted
`shell:startup` = `%AppData%\Microsoft\Windows\Start Menu\Programs\Startup`	Current user's Startup folder
`shell:common startup` = `%ProgramData%\Microsoft\Windows\Start Menu\Programs\Startup`	All users' Startup folder

This is not the complete list of ways a program can start (scheduled tasks and services are others), but it covers what Settings' startup list points you to.

Step 1: a read-only audit

$runKeys = @(
    'HKCU:\Software\Microsoft\Windows\CurrentVersion\Run'
    'HKCU:\Software\Microsoft\Windows\CurrentVersion\RunOnce'
    'HKLM:\Software\Microsoft\Windows\CurrentVersion\Run'
    'HKLM:\Software\Microsoft\Windows\CurrentVersion\RunOnce'
    'HKLM:\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Run'
)

function Get-StartupEntry {
    [CmdletBinding()]
    param()

    foreach ($key in $runKeys) {
        if (-not (Test-Path -LiteralPath $key)) { continue }
        $item = Get-Item -LiteralPath $key
        foreach ($name in $item.GetValueNames()) {
            if ($name -eq '') { continue }   # skip the (Default) value
            [pscustomobject]@{
                Source   = 'Registry'
                Location = $key
                Name     = $name
                # Read the raw text, so %ProgramFiles% etc. are not expanded
                Command  = $item.GetValue($name, $null, 'DoNotExpandEnvironmentNames')
            }
        }
    }

    $folders = [Environment]::GetFolderPath('Startup'),
               [Environment]::GetFolderPath('CommonStartup')
    foreach ($folder in $folders) {
        if (-not $folder -or -not (Test-Path -LiteralPath $folder)) { continue }
        Get-ChildItem -LiteralPath $folder -File |
            Where-Object Name -ne 'desktop.ini' |
            ForEach-Object {
                [pscustomobject]@{
                    Source   = 'StartupFolder'
                    Location = $folder
                    Name     = $_.Name
                    Command  = $_.FullName
                }
            }
    }
}

Run it in a normal (non-admin) PowerShell window. Reading HKLM doesn't need admin rights:

Get-StartupEntry | Format-Table -AutoSize

A few details that matter:

GetValue(..., 'DoNotExpandEnvironmentNames') returns the command exactly as stored, so %LOCALAPPDATA%\... stays as written. You want the raw text in a backup, otherwise restoring it changes the value's meaning.
[Environment]::GetFolderPath('Startup') and 'CommonStartup' resolve the two Startup folders without hard-coding paths.
This list doesn't show the On/Off state you see in Settings or Task Manager. Cross-check there before you remove anything. If an entry can be turned off with the toggle, use the toggle. It's simpler and you can undo it with one click.

Windows also exposes a combined view through WMI, which is handy as a second opinion:

Get-CimInstance -ClassName Win32_StartupCommand |
    Select-Object Name, Command, Location, User

Step 2: remove one Run value, safely

Only do this for an entry you understand: typically a leftover that points to a program you have already uninstalled. Don't remove security software, and look up anything you don't recognise first. Microsoft warns that editing the registry can have unintended consequences and says to back it up first. This function makes the backup for you, for the one value it changes.

function Disable-RunEntry {
    [CmdletBinding(SupportsShouldProcess, ConfirmImpact = 'Medium')]
    param(
        [Parameter(Mandatory)] [string] $Key,
        [Parameter(Mandatory)] [string] $Name,
        [string] $BackupDir = (Join-Path $env:LOCALAPPDATA 'StartupBackups')
    )

    $item = Get-Item -LiteralPath $Key -ErrorAction Stop
    if ($item.GetValueNames() -notcontains $Name) {
        throw "No value named '$Name' under $Key"
    }

    # Capture everything needed to put it back, before touching anything.
    $backup = [pscustomobject]@{
        Key     = $Key
        Name    = $Name
        Value   = $item.GetValue($Name, $null, 'DoNotExpandEnvironmentNames')
        Kind    = $item.GetValueKind($Name).ToString()
        SavedAt = (Get-Date).ToString('o')
    }

    if ($PSCmdlet.ShouldProcess("$Key\$Name", 'Remove startup value (backup is saved first)')) {
        New-Item -ItemType Directory -Path $BackupDir -Force -ErrorAction Stop -Confirm:$false | Out-Null
        $safeName = $Name -replace '[^\w.-]', '_'
        $file = Join-Path $BackupDir ('{0}_{1}.json' -f $safeName, (Get-Date -Format 'yyyyMMdd-HHmmss'))
        $backup | ConvertTo-Json | Set-Content -LiteralPath $file -Encoding UTF8 -ErrorAction Stop -Confirm:$false

        Remove-ItemProperty -LiteralPath $Key -Name $Name -ErrorAction Stop -Confirm:$false
        # Output the backup path so you can pass it straight to Restore-RunEntry
        [pscustomobject]@{ Removed = "$Key\$Name"; BackupFile = $file }
    }
}

Three things make this safe to run:

SupportsShouldProcess gives the function -WhatIf and -Confirm for free. Nothing inside the if ($PSCmdlet.ShouldProcess(...)) block runs during -WhatIf.
The backup is written before the delete. New-Item and Set-Content use -ErrorAction Stop, so if the backup folder or file can't be written, the function stops and Remove-ItemProperty never runs. Without -ErrorAction Stop this wouldn't hold: most write failures (access denied, disk full) are non-terminating errors, and the next line would still run.
The value kind is saved too (String vs ExpandString), so the restore writes back exactly the same type.

The inner -Confirm:$false calls stop you being asked three times when you run with -Confirm. You confirm once, at the ShouldProcess prompt.

Preview first, then run it for real:

Disable-RunEntry -Key 'HKCU:\Software\Microsoft\Windows\CurrentVersion\Run' -Name 'SomeOldUpdater' -WhatIf
Disable-RunEntry -Key 'HKCU:\Software\Microsoft\Windows\CurrentVersion\Run' -Name 'SomeOldUpdater'

The second command prints the path of the backup file. Entries under HKLM: need an administrator PowerShell window.

Step 3: undo

function Restore-RunEntry {
    [CmdletBinding(SupportsShouldProcess, ConfirmImpact = 'Medium')]
    param([Parameter(Mandatory)] [string] $BackupFile)

    $b = Get-Content -LiteralPath $BackupFile -Raw -ErrorAction Stop | ConvertFrom-Json
    if ($PSCmdlet.ShouldProcess("$($b.Key)\$($b.Name)", 'Restore startup value')) {
        New-ItemProperty -LiteralPath $b.Key -Name $b.Name -Value $b.Value `
            -PropertyType $b.Kind -Force -ErrorAction Stop -Confirm:$false | Out-Null
        [pscustomobject]@{ Restored = "$($b.Key)\$($b.Name)"; Value = $b.Value }
    }
}

Restore-RunEntry -BackupFile "$env:LOCALAPPDATA\StartupBackups\SomeOldUpdater_20261029-101500.json" -WhatIf
Restore-RunEntry -BackupFile "$env:LOCALAPPDATA\StartupBackups\SomeOldUpdater_20261029-101500.json"

What about the Startup folders?

Don't delete shortcuts there. Move them to a backup folder instead, and move them back to undo:

$backup = Join-Path $env:LOCALAPPDATA 'StartupBackups\StartupFolder'
New-Item -ItemType Directory -Path $backup -Force | Out-Null
Move-Item -LiteralPath (Join-Path ([Environment]::GetFolderPath('Startup')) 'SomeApp.lnk') -Destination $backup -WhatIf

Remove -WhatIf when the preview looks right. The all-users folder needs admin rights.

Limits, honestly

The code was tested with PowerShell 7 on Linux, using fake registry objects. It parsed cleanly, PSScriptAnalyzer reported nothing, and Pester tests of the backup/remove/restore logic passed (including a simulated failed backup write, which stops the removal). It has not been run against a real Windows registry or on Windows PowerShell 5.1. Try it with -WhatIf first, and make a restore point if you are nervous.
It only covers the Run/RunOnce keys and Startup folders, not scheduled tasks, services or per-app settings.
It removes one value at a time on purpose. Bulk "cleaners" are how people end up with a missing touchpad or audio driver panel after a restart.

Sources (Microsoft)

Configure startup applications in Windows (Settings, Task Manager impact levels, Startup folders, registry locations, back up the registry first): https://support.microsoft.com/en-us/windows/experience/startup-boot/configure-startup-applications-in-windows
Run and RunOnce registry keys: https://learn.microsoft.com/en-us/windows/win32/setupapi/run-and-runonce-registry-keys
Win32_StartupCommand class: https://learn.microsoft.com/en-us/windows/win32/cimwin32prov/win32-startupcommand
Everything you wanted to know about ShouldProcess: https://learn.microsoft.com/en-us/powershell/scripting/learn/deep-dives/everything-about-shouldprocess

Code is in my repo: https://github.com/fecklesspleb/startup-audit

This article was created with the help of AI. Not affiliated with or endorsed by Microsoft.

Fetching user-supplied URLs from a Cloudflare Worker without getting burned

Scott Williams — Thu, 29 Oct 2026 14:04:16 +0000

AI disclosure: this post was written with AI assistance (drafting, code and editing). The code comes from a project whose unit and integration tests I ran on October 28, 2026.

Any API that takes a URL and fetches it (link previews, metadata extraction, uptime checks, feed readers) is a request-forgery machine unless you put rules around it. Running on Cloudflare Workers removes some risks, since there is no cloud metadata service sitting on localhost, but not all of them. Your Worker can still be pointed at internal hostnames, at origins that only trust Cloudflare IPs, or at a 5 GB file.

These are the checks I ended up with, in the order a request hits them. The snippets are trimmed; bad, blocked and tooLarge are small helpers that build error responses.

1. Validate the URL before you touch the network

if (u.protocol !== 'http:' && u.protocol !== 'https:') throw bad('Only http and https URLs are supported.');
if (u.username || u.password) throw bad('URLs with credentials are not allowed.');
if (u.port && u.port !== '80' && u.port !== '443') throw blocked('Only ports 80 and 443 are allowed.');

Then reject hostnames that can never be public: localhost, single-label names, and suffixes such as .local, .internal, .lan, .home.arpa, .test, .invalid and .onion. Strip a trailing dot first. localhost. is a valid way to write localhost.

2. Resolve the name and check every address

A hostname can point anywhere, so check where it resolves. Workers have no dns.lookup, but DNS over HTTPS works with a plain fetch:

async function doh(name: string, type: 'A' | 'AAAA'): Promise<string[]> {
  const r = await fetch(`https://cloudflare-dns.com/dns-query?name=${encodeURIComponent(name)}&type=${type}`, {
    headers: { accept: 'application/dns-json' },
    signal: AbortSignal.timeout(5000),
  });
  const j: any = await r.json();
  const want = type === 'A' ? 1 : 28;
  return (j.Answer || []).filter((a: any) => a.type === want).map((a: any) => String(a.data));
}

Query A and AAAA in parallel and reject the request if any answer is non-public. "Any", not "the first one": an attacker controls the DNS records and can mix a public and a private address.

The IPv4 blocklist is the usual set: 0/8, 10/8, 100.64/10, 127/8, 169.254/16, 172.16/12, 192.168/16, 192.0.0/24, 192.88.99/24, 198.18/15, the three documentation ranges, multicast and 240/4. IPv6 is where most hand-rolled checks leak, because IPv4 can hide inside it. Decode and re-check the embedded IPv4 address for:

IPv4-mapped ::ffff:127.0.0.1
the deprecated IPv4-compatible form ::127.0.0.1
NAT64 64:ff9b::/96
6to4 2002::/16, where the IPv4 address sits in the second and third groups

Then block ::, ::1, fc00::/7, fe80::/10, fec0::/10, ff00::/8, 2001:db8::/32 and 2001::/23. Write unit tests for each form. Parsing :: compression and embedded dotted quads is easy to get subtly wrong.

Treat anything you cannot parse as unsafe.

3. Follow redirects yourself

fetch follows redirects by default, which skips every check above for the second hop. Turn that off and loop:

const seen = new Set<string>([url.href]);
for (let hop = 0; ; hop++) {
  await assertPublicUrl(url);            // steps 1 and 2, on every hop
  const res = await fetch(url, { redirect: 'manual', signal: AbortSignal.timeout(timeoutMs) });
  if (res.status < 300 || res.status > 399 || !res.headers.get('location')) return res;
  if (hop >= maxRedirects) throw new Error('too many redirects');
  url = new URL(res.headers.get('location')!, url);   // Location may be relative
  if (seen.has(url.href)) throw new Error('redirect loop');
  seen.add(url.href);
}

Return the redirect chain to the caller as well. It makes debugging much easier for API users.

There is a gap I have not closed: DNS rebinding. The Worker checks the name, then fetch resolves it again, and a hostile DNS server with a zero TTL can answer differently the second time. Workers do not let you pin the connection to the IP you checked, and a fetch to a bare IP breaks TLS and virtual hosting. If your threat model is serious, put the fetcher behind an egress proxy that does the resolution and the check in one step.

4. Cap time and bytes

content-length is a hint, not a promise. Reject early when it is present and too large, then enforce the cap while streaming:

const reader = res.body!.getReader();
let total = 0;
const chunks: Uint8Array[] = [];
for (;;) {
  const { done, value } = await reader.read();
  if (done) break;
  if (total + value.byteLength > maxBytes) { await reader.cancel(); throw tooLarge(maxBytes); }
  chunks.push(value);
  total += value.byteLength;
}

Gzipped sitemaps need the same treatment after decompression (DecompressionStream('gzip')), or a few kilobytes of input can expand into gigabytes. Pass every fetch an AbortSignal.timeout() too.

Also mind the plan limits. On the Workers Free plan a request gets 10 ms of CPU time and 50 subrequests, according to the pricing and limits pages. Two DoH lookups per hop count against the subrequest limit, and parsing a large PDF or sitemap does not fit in 10 ms. A small in-memory DNS cache with a 60-second lifetime helps with the first problem. The second needs the paid plan.

5. Respect robots.txt properly

If your API crawls on behalf of users, honor robots.txt. RFC 9309 standardized the rules in 2022, and a few of them differ from what older parsers do:

Matching: the most specific rule (the longest pattern) wins. When an Allow and a Disallow are equally specific, Allow should win. * matches any sequence and a trailing $ anchors the end.
Groups: use the group whose user-agent matches your product token, and fall back to * only when none does. Several groups for the same agent are merged.
Fetching robots.txt: a 4xx response means no restrictions. A 5xx response or a network failure means you must assume everything is disallowed. Follow at least five redirects, and parse at least the first 500 KiB.
/robots.txt itself is always allowed.

The matcher is short. Compile each pattern into an anchored regex, and keep the raw pattern length to compare specificity:

function compile(pattern: string): RegExp {
  const anchored = pattern.endsWith('$');
  const body = (anchored ? pattern.slice(0, -1) : pattern)
    .split('*').map((p) => p.replace(/[.+?^${}()|[\]\\]/g, '\\$&')).join('.*');
  return new RegExp('^' + body + (anchored ? '$' : ''));
}

canFetch(token: string, url: string): boolean {
  if (this.disallowAll) return false;          // robots.txt returned 5xx or was unreachable
  const u = new URL(url);
  if (u.pathname === '/robots.txt') return true;
  const path = decodeURIComponent(u.pathname) + u.search;
  let best = -1, allow = true;
  for (const r of this.rulesFor(token)) {
    if (r.rx.test(path) && (r.pattern.length > best || (r.pattern.length === best && r.allow))) {
      best = r.pattern.length;
      allow = r.allow;
    }
  }
  return allow;
}

Ignore empty Disallow: lines (they mean "allow everything"), and wrap decodeURIComponent in a try/catch, because real-world paths contain stray % signs.

Testing

Three layers were worth writing:

Pure unit tests for the robots matcher: group selection by product token, longest match, an Allow/Disallow tie, * and $ patterns, an empty Disallow, and percent-encoded paths.
Unit tests with a stubbed fetch for the network logic: a public URL that 302s to 169.254.169.254, a 301 to http://[::ffff:7f00:1]/ (127.0.0.1 written as IPv4-mapped IPv6 in hex), a robots.txt that returns 503, a robots.txt behind a redirect, and a redirect loop.
Integration tests against wrangler dev with real public URLs and blocked targets, to confirm the checks stop what they should without blocking legitimate sites.

One gap I would add next: a test for a body that is larger than its content-length header claims.

This article was created with the help of AI. The code was tested by me before publishing.

Reading Greenhouse, Lever and Ashby job boards from their public JSON endpoints

Scott Williams — Wed, 28 Oct 2026 05:13:52 +0000

AI disclosure: this post was written with AI assistance (drafting and editing). I ran every request and code sample below myself on October 28, 2026 and checked the output.

If you want a list of open jobs at a set of companies, you usually do not need a headless browser. Most tech companies host their career page on an applicant tracking system (ATS), and three of the most common ones, Greenhouse, Lever and Ashby, expose the same postings their career pages render as plain JSON. No API key, no login.

This post covers the three endpoints, the fields worth keeping, and a small reader that turns all three into one shape.

1. Find the board name

The board name is in the career page URL:

ATS	Career page looks like	Board name
Greenhouse	`boards.greenhouse.io/airbnb` or `job-boards.greenhouse.io/airbnb`	`airbnb`
Lever	`jobs.lever.co/spotify`	`spotify`
Ashby	`jobs.ashbyhq.com/ramp`	`ramp`

If a company uses its own domain, view the page source and search for greenhouse, lever.co or ashbyhq. The embed script or the apply links usually give it away.

2. The endpoints

Greenhouse (Job Board API, docs):

GET https://boards-api.greenhouse.io/v1/boards/{board}/jobs?content=true

Returns {"jobs": [...], "meta": {"total": N}}. Each job has id, title, location.name, absolute_url, updated_at, first_published and, with content=true, an HTML-escaped description in content. Drop content=true if you only need titles; the response gets much smaller.

Lever (Postings API, docs):

GET https://api.lever.co/v0/postings/{site}?mode=json

Returns a bare JSON array. Useful fields: id, text (the title), categories.location, categories.team, categories.commitment, workplaceType, hostedUrl, createdAt (epoch milliseconds). Companies on Lever's EU instance use api.eu.lever.co instead. An unknown site returns {"ok": false, "error": "Document not found"} with a 404, so check for a list before iterating.

Ashby (public job posting API, docs):

GET https://api.ashbyhq.com/posting-api/job-board/{board}?includeCompensation=true

Returns {"jobs": [...]}. Fields: id, title, department, team, location, secondaryLocations, isRemote, workplaceType, employmentType, publishedAt, jobUrl, applyUrl. With includeCompensation=true, many jobs carry compensation.compensationTierSummary, a ready-to-display string such as $211.4K – $290.6K • Offers Equity. Salary data is the main reason I reach for Ashby boards first.

3. One reader for all three

Standard library only:

import json
import urllib.request

UA = {"User-Agent": "job-board-reader/1.0 (+you@example.com)"}


def get_json(url):
    req = urllib.request.Request(url, headers=UA)
    with urllib.request.urlopen(req, timeout=20) as r:
        return json.load(r)


def greenhouse(board):
    data = get_json(f"https://boards-api.greenhouse.io/v1/boards/{board}/jobs?content=true")
    for j in data["jobs"]:
        yield {
            "source": "greenhouse", "id": str(j["id"]), "title": j["title"],
            "location": (j.get("location") or {}).get("name"),
            "posted": j.get("first_published") or j.get("updated_at"),
            "url": j["absolute_url"],
        }


def lever(site):
    for j in get_json(f"https://api.lever.co/v0/postings/{site}?mode=json"):
        cats = j.get("categories") or {}
        yield {
            "source": "lever", "id": j["id"], "title": j["text"],
            "location": cats.get("location"),
            "posted": j.get("createdAt"),  # epoch milliseconds
            "url": j["hostedUrl"],
        }


def ashby(board):
    data = get_json(f"https://api.ashbyhq.com/posting-api/job-board/{board}?includeCompensation=true")
    for j in data["jobs"]:
        yield {
            "source": "ashby", "id": j["id"], "title": j["title"],
            "location": j.get("location"),
            "posted": j.get("publishedAt"),
            "url": j["jobUrl"],
        }


if __name__ == "__main__":
    jobs = [*greenhouse("airbnb"), *lever("spotify"), *ashby("ramp")]
    print(len(jobs), "jobs")

When I ran it, it printed 456 jobs (195 Greenhouse, 89 Lever, 172 Ashby), in under a second end to end.

4. Things that bite

Dates come in three formats. Greenhouse uses ISO 8601 with a local offset, Ashby uses ISO 8601 in UTC, and Lever uses epoch milliseconds. Normalize before you sort or filter by "posted in the last N days".
Greenhouse updated_at is not the posting date. It changes when a recruiter edits the job. Prefer first_published when it is present.
Locations are free text. "Remote - USA", "United States" and "New York, NY (HQ)" all appear. Ashby's isRemote and Lever's workplaceType are more reliable than parsing strings.
IDs are only unique per ATS. Prefix them (ashby:<id>) before you dedupe across sources.
Descriptions are HTML. Greenhouse's is HTML-escaped HTML, so unescape first, then strip tags.
Detecting new jobs is just a set difference: store the prefixed IDs you have seen, and on the next run keep only IDs you have not seen.

5. Be a polite client

These endpoints exist so career pages can render, and that is the reason they are open. Treat them that way:

Send a descriptive User-Agent with a contact address.
Check robots.txt. On the day I wrote this, api.lever.co/robots.txt allowed everything with Crawl-delay: 1, and boards-api.greenhouse.io/robots.txt disallowed only /embed/.
One request per board is enough. Each endpoint returns the whole board, so polling once or twice a day covers almost every use case.
Only collect what the company has already published. Do not try to reach application data or anything behind a login.

Workday career sites are the big platform missing from this list. They also serve JSON, but through a per-tenant POST endpoint with pagination and facets, which deserves a post of its own.

This article was created with the help of AI. The endpoints, numbers and code were tested by me before publishing.

DEV Community: Scott Williams

Getting only the new DOIs from the Crossref API every day: dates, cursors and rate limits

1. Pick the right date: created, not published, not indexed

2. Always set both ends of the range

3. Page with a cursor, and you can still sort

4. The 2026 rate limits are lower than old blog posts say

5. Say who you are

6. "New" arrives late, so overlap your window

7. query is a relevance search, not a filter

8. Small normalisation traps

9. Finding the IDs to filter on

What the feed looks like

Reading Workable, Recruitee and Teamtailor job boards from their public JSON

Workable

Recruitee

Teamtailor

robots.txt and signals

A minimal reader

Field mapping at a glance

Only the new jobs: a daily Greenhouse, Lever and Ashby feed with Python and GitHub Actions

The plan

Pick a key that survives

The script

Details that break a daily diff

Run it every day on GitHub Actions

Where this stops working

Reading Workday job postings as JSON, and getting past the 2,000-result ceiling

1. From a career site URL to the API

2. Three limits worth knowing

3. Getting past the cap with facets

4. The reader

5. Things that bite

6. Be a polite client

Diagnose a stuck Windows Update with PowerShell: services, pending restarts, update cache and event-log errors, read-only first

Step 1: the services

Step 2: is a restart pending?

Step 3: the update cache and free space

Step 4: recent Windows Update errors

Step 5 (opt-in): reset the update cache

If that doesn't fix it

Limits, honestly

Sources (Microsoft)

Find what's filling your Windows drive with PowerShell: a read-only audit first, then a cleanup you can preview with -WhatIf

Step 1: free space per drive

Step 2: measure a folder without following links

Step 3: the usual suspects

Step 4: clear old TEMP files, with a preview

What about the Recycle Bin?

What not to touch by hand

Limits, honestly

Sources (Microsoft)

Audit Windows startup entries with PowerShell without breaking anything: read-only first, -WhatIf, a JSON backup and an undo

Where startup entries live

Step 1: a read-only audit

Step 2: remove one Run value, safely

Step 3: undo

What about the Startup folders?

Limits, honestly

Sources (Microsoft)

Fetching user-supplied URLs from a Cloudflare Worker without getting burned

1. Validate the URL before you touch the network

2. Resolve the name and check every address

3. Follow redirects yourself

4. Cap time and bytes

5. Respect robots.txt properly

Testing

Reading Greenhouse, Lever and Ashby job boards from their public JSON endpoints

1. Find the board name

2. The endpoints

3. One reader for all three

4. Things that bite

5. Be a polite client

7. `query` is a relevance search, not a filter