DEV Community: Siftwright

Google News RSS vs a Google News API: what you actually get (with working Python)

Siftwright — Fri, 30 Oct 2026 14:48:25 +0000

Google does not offer an official Google News API. If you need news results in code, you end up with one of two options: the undocumented Google News RSS feed, or a third-party API that returns the same kind of data as JSON. Both work. They differ in ways that only show up once you run them for a while.

This post compares them using real runs from 30 October 2026, with Python for each. We sell one of the options (the Siftwright Google News API), so we say plainly where the free RSS feed is the better choice.

The Google News RSS feed

Google News search is available as an RSS feed at a predictable URL:

https://news.google.com/rss/search?q=electric+vehicles&hl=en-US&gl=US&ceid=US:en

q is the query, hl the interface language, gl the country and ceid the country-and-language edition. A plain request returned HTTP 200 and 100 <item> elements in our run. Each item has a title, a link, a GUID, a publication date, an HTML description and a <source> element with the publisher.

Here is a minimal client using only the standard library:

import urllib.parse, urllib.request
import xml.etree.ElementTree as ET
from email.utils import parsedate_to_datetime

def google_news_rss(query, hl="en-US", gl="US", ceid="US:en"):
    url = "https://news.google.com/rss/search?" + urllib.parse.urlencode(
        {"q": query, "hl": hl, "gl": gl, "ceid": ceid})
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    with urllib.request.urlopen(req, timeout=30) as r:
        root = ET.fromstring(r.read())
    items = []
    for it in root.findall("./channel/item"):
        src = it.find("source")
        items.append({
            "title": it.findtext("title"),
            "link": it.findtext("link"),
            "source": src.text if src is not None else None,
            "published": parsedate_to_datetime(it.findtext("pubDate")).isoformat(),
        })
    return items

for i in google_news_rss("electric vehicles")[:3]:
    print(i["published"], "|", i["source"], "|", i["title"])

Real output:

2026-10-29T21:02:00+00:00 | Electrek | Toyota’s EV sales more than doubled in a year, but gas cars are getting crushed in China - Electrek
2026-10-29T22:00:00+00:00 | The Washington Post | Why Americans can’t buy these world-class EVs - The Washington Post
2026-10-27T16:00:00+00:00 | The Washington Post | Column | I went to Mexico to test-drive the cars Americans can’t buy - The Washington Post

For a hobby project, a personal dashboard or a quick prototype, this is hard to beat: it is free and needs no signup. Other languages work by changing the edition. A search for Elektroautos with hl=de&gl=DE&ceid=DE:de returned 100 German-language items, the first from Spiegel.

What the RSS feed leaves for you to clean up

Running the client above and inspecting the 100 items exposed the following.

The publisher name is glued onto the title. In our run, all 100 titles ended with - <source>, as in ... getting crushed in China - Electrek. If you show or deduplicate titles, you have to strip that suffix yourself.

Every link is a Google redirect. All 100 links were news.google.com/rss/articles/... URLs, not the publisher's address. They open the article in a browser, but if you want the publisher's domain for analysis you need to follow the redirect, or use the <source url> attribute (which gives the publisher's homepage, not the article).

The description is HTML. It contains a link to the article plus related-coverage links, not a text snippet. Parsing it for a summary is fragile.

A hard cap of 100 items. Our test query returned exactly 100, and so did the same query with when:1d and when:7d added. For broad queries, you see the newest slice, not everything. Narrow the query to go deeper.

Parameters are required. Requests without the hl/gl/ceid parameters were answered with a 302 redirect instead of a feed (we did not follow it). Build the URL with all three.

Terms. The feed carries a copyright notice stating it is made available "solely for the purpose of rendering Google News results within a personal feed reader for personal, non-commercial use". Read that before you build a commercial product on it. We are not lawyers and this is not legal advice, but it is the reason many teams look at paid APIs.

No guarantees. It is an undocumented endpoint. Nothing promises it will keep its format, stay unthrottled at volume or stay available.

The same search through a Google News API

A Google News API does the extraction and clean-up for you and returns JSON. The call below uses the public Siftwright Google News Scraper Actor on Apify, which is the same engine behind our API; pay-per-event pricing means you can try it on Apify's free plan credit:

import os, json, urllib.request

def google_news_actor(query, max_items=100, language="en", country="US"):
    url = ("https://api.apify.com/v2/acts/siftwright~google-news-scraper/"
           "run-sync-get-dataset-items?token=" + os.environ["APIFY_TOKEN"])
    body = json.dumps({"query": query, "maxItems": max_items,
                       "language": language, "country": country}).encode()
    req = urllib.request.Request(url, body, {"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=120) as r:
        return json.load(r)

for r in google_news_actor("electric vehicles", 5):
    print(r["publishedAt"], "|", r["source"], "|", r["title"][:60], "|", r["status"])

Real output:

2026-10-29T21:02:00.000Z | Electrek | Toyota’s EV sales more than doubled in a year, but gas cars  | ok
2026-10-29T22:00:00.000Z | The Washington Post | Why Americans can’t buy these world-class EVs | ok
2026-10-27T16:00:00.000Z | The Washington Post | Column | I went to Mexico to test-drive the cars Americans c | ok
2026-10-30T06:00:00.000Z | Barron's | ‘This Is a Knife Fight:’ Inside China’s Ruthless EV Blitz Up | ok
2026-10-29T23:02:00.000Z | Reuters | Chinese EVs win over some in South Korea, but Japan is provi | ok

Each record has query, title, link, source, publishedAt (ISO 8601 in UTC), snippet and status. The title comes without the source suffix, and the publisher is its own field. The same query matched the same articles as the RSS feed, which is what you would expect.

To be exact about what is and is not different:

Links are still Google redirects. The link values in our run were news.google.com/rss/articles/... URLs, same as the feed. The API does not magically resolve them. If you need publisher URLs, you still follow the redirect yourself.
The cap is the same. Asking for maxItems: 150 returned 100 records.
The sources are the same. Both read Google News. The API does not give you a different or bigger index.

So the value of an API is not extra data. It is the clean schema, the language and country parameters, an explicit status per item, and a service you call with a key and a price rather than an endpoint you hope keeps working.

Side by side

	Google News RSS	Siftwright Google News API
Cost	Free	From $29/month (10,000 results), or $1.50 per 1,000 articles on Apify
Signup	None	API key, or Apify account
Max items per query	100	100
Output	XML	JSON (or CSV on Apify)
Publisher name	Glued to the title, plus a `<source>` element	Separate `source` field
Date format	RFC 822 string	ISO 8601 UTC
Snippet	HTML with related links	Plain `snippet` field
Article links	Google redirect URLs	Google redirect URLs
Failed or empty search	Empty feed or a redirect	Empty searches are not billed
Commercial use	Feed notice says personal, non-commercial	Governed by our terms; you are responsible for lawful use
Stability promise	None	A service we operate

Which one should you use?

Use the RSS feed when:

you are prototyping, learning or building something for yourself;
you run a handful of queries and do not mind writing the clean-up code above;
you are comfortable with the feed's notice and its lack of guarantees.

Use an API when:

the data feeds something a customer pays for, and you need a clear, supported source;
you run many queries and want a consistent schema without maintaining parsing code;
you want to know what each item cost and why an item failed.

In either case, store the results and deduplicate them rather than re-reading the same feed. The pipeline in our news monitoring guide shows queries, dedupe, SQLite storage and a daily digest, and the same code works with either source if you map the fields to title, link, source and published.

Adapting the RSS client to the same schema

If you start with RSS and might switch later, normalise now. A small adapter makes the switch a one-line change:

import re

def normalise_rss(item):
    title = item["title"]
    if item["source"] and title.endswith(" - " + item["source"]):
        title = title[: -len(item["source"]) - 3]
    return {"title": title, "link": item["link"],
            "source": item["source"], "published": item["published"]}

def normalise_api(row):
    return {"title": row["title"], "link": row["link"],
            "source": row["source"], "published": row["publishedAt"]}

Every downstream step (dedupe, storage, alerts) then works on one shape. For deduplication, key on the normalised title plus source rather than the link, since a story syndicated to several outlets has several links but the same headline.

What it costs

On our plans, a result is one article that came back with status ok, and empty searches are free. Plans start at $29 per month with 10,000 successful results (see pricing). If you prefer no subscription, the Apify Actor charges $1.50 per 1,000 articles. For the 100-article test above, that is $0.15. A quick comparison for your own volume is on the pricing calculator, and our Google News API page has a live demo with no signup.

A note on testing: the RSS code and the Actor call above were run on 30 October 2026 exactly as shown. We did not re-run the /v1/google-news REST endpoint with a fresh key for this post, since the Actor returns the same record shape, and the fields in the API's response are listed in the docs.

Summary

Google has no official News API. The RSS feed is free and decent for personal projects, but it returns XML with the publisher glued to the title, HTML descriptions and Google redirect links, and its own notice limits it to personal, non-commercial use.
A News API returns the same articles with a clean schema, ISO dates, a separate source field and per-item status. It does not give you a larger index or publisher links.
Whichever you pick, normalise to one schema early, deduplicate on title plus source, and keep your queries narrow, because both stop at 100 items.

Questions or a case this post missed? Email support@siftwright.com.

YouTube subtitles to SRT and VTT from code: language tracks, cue timing and a fallback

Siftwright — Tue, 27 Oct 2026 14:51:32 +0000

You have a list of YouTube videos and you need subtitle files for them: .srt for an editor or a video player, .vtt for an HTML <track> element, maybe a second language for the same video. Doing it by hand in the YouTube UI does not scale past a handful, and the usual libraries hand you raw caption segments that you still have to turn into valid files.

This post covers the whole path: fetching captions, choosing a language track, building SRT and VTT correctly (including the one timing quirk that breaks players), and what to do when a translation request fails. The sample output below comes from real runs on 27 October 2026 against the first YouTube video ever uploaded, jNQXAC9IVRw (19 seconds, so the files stay short enough to read).

What an SRT and a VTT file actually are

Both formats are plain text lists of cues. A cue is a time range and the text to show.

SRT numbers every cue and uses a comma before the milliseconds:

1
00:00:01,200 --> 00:00:03,360
All right, so here we are, in front of the elephants

VTT starts with a WEBVTT header, drops the numbers (they are optional) and uses a dot:

WEBVTT

00:00:01.200 --> 00:00:03.360
All right, so here we are, in front of the elephants

That is the entire difference for simple captions. Which means the interesting work is not the syntax. It is getting correct timings and the right language in the first place.

Step 1: fetch the captions

The Siftwright YouTube transcript API can return srt and vtt strings directly through its outputFormats option, next to the raw segments. Requesting segments as well is worth it, because they let you build your own files when you need a different line length or style.

curl https://siftwright.com/v1/youtube-transcript \
  -H "Authorization: Bearer $SIFTWRIGHT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.youtube.com/watch?v=jNQXAC9IVRw",
       "language": "en",
       "outputFormats": ["segments", "srt", "vtt"]}'

In Python, saving one pair of files per video:

import os
import requests

API = "https://siftwright.com/v1/youtube-transcript"
KEY = os.environ["SIFTWRIGHT_API_KEY"]

def fetch(url, language="en"):
    r = requests.post(
        API,
        headers={"Authorization": f"Bearer {KEY}"},
        json={"url": url, "language": language,
              "outputFormats": ["segments", "srt", "vtt"]},
        timeout=120,
    )
    r.raise_for_status()
    return r.json()["results"][0]

item = fetch("https://www.youtube.com/watch?v=jNQXAC9IVRw")
if item["status"] == "ok":
    open(f'{item["videoId"]}.{item["language"]}.srt', "w", encoding="utf-8").write(item["srt"])
    open(f'{item["videoId"]}.{item["language"]}.vtt', "w", encoding="utf-8").write(item["vtt"])

A note on what was and was not tested. For this post we ran the same request, with the same input fields, against the public YouTube Transcript Extractor Actor, the engine behind the endpoint, and checked every output below against it. The response envelope of the endpoint (a results array, as shown in our earlier tutorial) is taken from our docs rather than from a fresh call with a new key.

The Actor returned this for the zoo video:

1
00:00:01,200 --> 00:00:03,360
All right, so here we are, in front of the elephants

2
00:00:05,318 --> 00:00:07,974
the cool thing about these guys is that they have really...

3
00:00:07,974 --> 00:00:12,616
really really long trunks

Notice cue 2 ends at 07,974, exactly where cue 3 begins. That is deliberate, and it is the quirk the next section is about.

Step 2: the overlap problem

Raw YouTube caption segments come as start and duration. For auto-generated captions in particular, a segment's start + duration often runs past the start of the next segment, because the text is being rolled up on screen while the next words arrive.

If you naively compute end = start + duration, you get cues that overlap. Overlapping cues are legal in VTT but they stack on screen, and strict SRT consumers either reject them or show two cues at once. The fix is simple: never let a cue end after the next one starts.

Two consecutive segments from the zoo video show the fields involved:

{"start": 5.318, "duration": 2.656, "text": "the cool thing about these guys is that they have really..."}
{"start": 7.974, "duration": 4.642, "text": "really really long trunks"}

Here 5.318 + 2.656 is exactly 7.974, so nothing overlaps. In auto-generated captions the rolled-up text can make a segment's computed end land after the next segment's start, so the clamp matters; it costs nothing when nothing overlaps.

Step 3: build your own files from segments

You need your own builder when you want different line lengths (broadcast guidelines often use 42 characters per line and two lines per cue), when you merge segments, or when you want to post-process text. This is the whole thing in standard-library Python:

import re

def ts(sec, sep):
    ms = round(sec * 1000)
    h, ms = divmod(ms, 3_600_000)
    m, ms = divmod(ms, 60_000)
    s, ms = divmod(ms, 1000)
    return f"{h:02}:{m:02}:{s:02}{sep}{ms:03}"

def wrap(text, width=42):
    lines, cur = [], ""
    for w in text.split():
        if cur and len(cur) + 1 + len(w) > width:
            lines.append(cur)
            cur = w
        else:
            cur = f"{cur} {w}".strip()
    lines.append(cur)
    return "\n".join(lines)

def cues(segments):
    out = []
    for i, seg in enumerate(segments):
        start = seg["start"]
        end = start + seg["duration"]
        if i + 1 < len(segments):          # never run past the next cue
            end = min(end, segments[i + 1]["start"])
        text = " ".join(seg["text"].split())
        if text and end > start:
            out.append((start, end, wrap(text)))
    return out

def to_srt(segments):
    return "\n".join(
        f"{n}\n{ts(a, ',')} --> {ts(b, ',')}\n{t}\n"
        for n, (a, b, t) in enumerate(cues(segments), 1))

def to_vtt(segments):
    body = "\n".join(
        f"{ts(a, '.')} --> {ts(b, '.')}\n{t}\n" for a, b, t in cues(segments))
    return "WEBVTT\n\n" + body

We checked this builder against the API's own srt output for the zoo video. The timings match cue for cue; the only difference is that the builder wraps long lines at 42 characters, so the first cue becomes two lines:

1
00:00:01,200 --> 00:00:03,360
All right, so here we are, in front of the
elephants

Whether that is better depends on where the subtitles are shown. For a narrow mobile player, wrapping helps. For a wide web player it is just noise, which is why we return the unwrapped version by default and give you the segments to wrap yourself.

If you already have an SRT string and just need VTT, converting it is three lines, because VTT is nearly a subset:

def srt_to_vtt(srt):
    srt = srt.replace("\r\n", "\n").strip() + "\n"
    srt = re.sub(r"(\d\d:\d\d:\d\d),(\d{3})", r"\1.\2", srt)
    srt = re.sub(r"^\d+\n(?=\d\d:)", "", srt, flags=re.M)   # drop cue numbers
    return "WEBVTT\n\n" + srt

Run on the API's SRT for the zoo video, this produced a file identical to the vtt field the API returned.

Step 4: choose the language track deliberately

A video can carry several caption tracks, and a track can be manual (written by the uploader) or auto-generated. Two things follow:

Ask for preferManual: true (the default) so a human-written track wins over a machine one in the same language.
Read availableLanguages in the response to see what actually exists instead of guessing.

For the zoo video the response lists two tracks, English and German, both manual. Asking for German with "language": "de" returned a real German track, not a machine translation:

1
00:00:00,000 --> 00:00:05,067
Also hier sind wir vor den Elefanten.

2
00:00:05,067 --> 00:00:12,467
Das Coole an den Typen ist dass sie sehr, sehr, sehr, lange Rüssel haben.

Note that the German cues have different timings from the English ones. Tracks are separate files written separately, so you cannot reuse one track's timings for another. If you ship both, ship both files.

Step 5: translation, and what happens when it fails

For languages the video does not have a track for, you can ask for a translation with translateTo (for example "translateTo": "fr"). It uses YouTube's own caption translation, so quality is machine quality, and it is a service we depend on rather than control.

We tested this, and it is worth being honest about the result. Two translation requests from our test environment, translateTo: "es" and translateTo: "fr", both came back with status: "translation_failed". The first said Translation failed (fetch failed) and the second Translation failed (HTTP 429), which is YouTube rate-limiting the request. In both cases the original-language transcript was returned instead, and the error message states that it is free of charge.

That behaviour is the useful part, and your code should expect it:

def subtitles(url, want="es"):
    r = requests.post(API, headers={"Authorization": f"Bearer {KEY}"},
        json={"url": url, "translateTo": want,
              "outputFormats": ["segments", "srt", "vtt"]}, timeout=120)
    r.raise_for_status()
    item = r.json()["results"][0]

    if item["status"] == "ok":
        return item, item["language"]            # translated (or native) track
    if item["status"] == "translation_failed":
        # original language came back, unbilled: keep it, retry the translation later
        return item, item["sourceLanguage"]
    raise RuntimeError(f'{item["status"]}: {item.get("error")}')

A failed translation is not a reason to drop the video. Store the original-language file, record that the translation is pending, and retry later with a delay. A 429 is the textbook case for backing off rather than hammering.

If you need translations you can depend on at volume, a better pipeline is to fetch the source-language segments once and send the cue texts to a translation service you control, keeping the original start and end values. That keeps timings stable and moves the failure risk to a service you can pick and pay for.

Practical checks before you ship the files

A few small checks catch most subtitle bugs:

Save as UTF-8. German, French and every non-Latin script break on a default Windows code page. Always pass encoding="utf-8".
Validate monotonic cues. Assert that each cue's start is at or after the previous cue's end. The builder above guarantees it; third-party SRTs often do not.
Name files by language. videoId.en.srt and videoId.de.srt. Players and HTML <track srclang> both want the language code.
Skip videos without captions. The API marks them with an error status and does not bill them, so a batch job should log and move on.
Respect the source. Captions belong to the video's uploader. Fine for accessibility, search and personal study; check the licence before republishing them.

What it costs

The Siftwright API plans start at $29 per month and count only results with status ok, so videos without captions and failed translations do not use your quota. If you only need a few thousand files, the pay-per-result Apify Actor is simpler. The pricing page has a calculator, and the post on pay-per-result vs subscription pricing explains where the break-even sits.

To check what a video offers before writing any code, the free YouTube transcript tool downloads TXT, SRT and VTT in the browser, no key needed.

Originally published on siftwright.com.

Track hiring signals across Greenhouse, Lever and Ashby with one Python script

Siftwright — Mon, 26 Oct 2026 19:11:02 +0000

Hiring is one of the clearest public buying signals: a company opening "Data Engineer" roles is about to spend on data tooling. The catch is that every company uses a different applicant tracking system (ATS), and each has its own API shape.

This tutorial uses the ATS Jobs Scraper Actor, which reads the public job-board feeds of Greenhouse, Lever, Ashby, SmartRecruiters and Recruitee and returns one schema. It costs $1 per 1,000 jobs returned after your filters; companies without a supported board and filtered-out jobs are free.

The script

import csv, os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("siftwright/ats-jobs-scraper").call(
    run_input={
        "companies": ["greenhouse:airbnb", "lever:palantir", "ramp.com"],
        "titleKeywords": ["data engineer", "analytics"],
        "excludeTitleKeywords": ["senior", "staff", "manager", "director"],
        "includeDescription": False,
        "maxJobsPerCompany": 20,
    },
    max_total_charge_usd=0.05,  # hard spend cap for the run
)

jobs = list(client.dataset(run.default_dataset_id).iterate_items())
with open("jobs.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["company", "ats", "title", "location", "url"])
    for j in jobs:
        w.writerow([j["company"], j["ats"], j["title"], j["location"], j["url"]])

summary = client.key_value_store(run.default_key_value_store_id).get_record("SUMMARY")
for c in summary["value"]["companies"]:
    print(f'{c["input"]:20} {c["status"]:10} {c["ats"] or "-":12} open={c["jobsOnBoard"]} matched={c["jobsMatched"]} returned={c["jobsReturned"]}')
print(len(jobs), "jobs written to jobs.csv")

Install and run:

pip install apify-client
export APIFY_TOKEN=your_token
python hiring.py

max_total_charge_usd caps what the run can spend, so a typo in a filter cannot surprise you.

What I got

I ran this on 2026-10-26 against three companies. The output:

greenhouse:airbnb    ok         Greenhouse   open=187 matched=7 returned=7
lever:palantir       ok         Lever        open=307 matched=1 returned=1
ramp.com             ok         Ashby        open=173 matched=0 returned=0
8 jobs written to jobs.csv

Eight jobs, for example "Data Engineer - Talent Acquisition Operations" (Palantir, New York) and "Software Engineer, Analytics Infrastructure" (Airbnb, Brazil). Ramp had 173 open roles and none matched these filters, so it returned nothing and cost nothing. Notice ramp.com was just a website: the Actor found the Ashby board from the careers page link. The SUMMARY record reports how each board was detected (detectedVia), so you can check it.

Limits worth knowing

Only five ATS platforms. Workday, iCIMS, Taleo and custom career sites come back as "not found" (free).
If you give a name instead of a board URL, the board slug is guessed; pass ats:board such as greenhouse:airbnb when you need certainty.
Salary is only present when the company publishes it. None of the jobs above had one.
Filters are substring matches on the title, so "analytics" also matched finance roles. Tighten titleKeywords for cleaner output.

Use it from an AI agent

The same Actor works through Apify's MCP server, so an agent can ask "which of these companies are hiring data engineers?" and pay per result.

Store page: https://apify.com/siftwright/ats-jobs-scraper. More tools: https://siftwright.com

Summarise any YouTube video with an LLM: transcript chunking that keeps timestamps

Siftwright — Fri, 23 Oct 2026 14:48:20 +0000

Pasting a YouTube transcript into a chat window works for a five-minute video. It stops working at about the time the video gets interesting. A two-hour lecture is roughly 20,000 words, which is more than you want to stuff into one prompt, and even when it fits, a single giant prompt tends to produce a summary that favours the start and the end and skips the middle.

The other thing a plain-text transcript loses is where things were said. A summary that says "the speaker explains attention" is much less useful than one that says "attention is explained at 25:10" with a link that opens the video there. This post builds that: a small Python pipeline that fetches a transcript with timestamps, splits it into chunks that respect caption boundaries, summarises each chunk, combines the notes, and rewrites every cited time into a clickable link.

We ran the transcript and chunking parts against a real video on 23 October 2026: Andrej Karpathy's "Let's build GPT: from scratch, in code, spelled out", 6,980 seconds long. The LLM calls are the one piece we could not run for you, and we say exactly where below.

Step 1: get segments, not just text

Most "transcript to text" tools return one big string. For summarisation you want the timestamped segments, because the timestamps are what let you cite. The Siftwright YouTube Transcript API returns them when you ask for segments:

curl -s -X POST https://siftwright.com/v1/youtube-transcript \
  -H "Authorization: Bearer $SIFTWRIGHT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://www.youtube.com/watch?v=jNQXAC9IVRw"], "outputFormats": ["segments"]}'

Each result carries the video's metadata plus a segments array. These are the first three segments of the first YouTube video ever uploaded, exactly as returned:

[
  {"start": 1.2,   "duration": 2.16,  "text": "All right, so here we are, in front of the elephants"},
  {"start": 5.318, "duration": 2.656, "text": "the cool thing about these guys is that they have really..."},
  {"start": 7.974, "duration": 4.642, "text": "really really long trunks"}
]

start and duration are in seconds. Two properties of caption data matter for what follows:

Segments are short. The 6,980-second lecture came back as 2,955 segments, about 7 words each. A segment is a caption line, not a sentence.
Segments can overlap, or leave gaps. In the lecture, the first segment starts at 0.199 s and lasts 4.7 s, but the second starts at 2.52 s, before the first has finished. In the zoo video above, there is a gap instead: the first segment ends at 3.4 s and the next starts at 5.3 s. Auto-generated captions are drawn with a rolling window, so never assume start + duration\ equals the next start\.

If a video has no captions, the result comes back with a non-ok status and is not billed. Check status before you use segments; the code below does.

Step 2: chunk by segment, keep the time range

The naive way to chunk is to split the text every N characters. That cuts sentences in half and throws away the timestamps. Chunking by segments keeps both: you add whole segments to a chunk until it reaches a word budget, then start the next one. The start of the first segment and the end of the last give you the chunk's time range for free.

Two small details improve the result. First, a few segments of overlap between chunks, so a thought that straddles a boundary appears in both summaries instead of being cut. Second, a time marker like [7:49] inserted roughly every 30 seconds inside the chunk text. Without markers the model has no idea when anything happened; with a marker on every segment you waste tokens. Every 30 seconds is a reasonable middle.

import json, sys

def fmt(t):
    t = int(t)
    h, m, s = t // 3600, t % 3600 // 60, t % 60
    return f"{h}:{m:02d}:{s:02d}" if h else f"{m}:{s:02d}"

def chunk_segments(segments, max_words=1500, overlap_segments=5):
    """Group caption segments into chunks of ~max_words, never splitting a segment.
    Each chunk keeps its start/end time so a summary can point back to the video."""
    chunks, cur, words = [], [], 0
    for seg in segments:
        n = len(seg["text"].split())
        if cur and words + n > max_words:
            chunks.append(cur)
            cur = cur[-overlap_segments:]          # small overlap so a sentence cut at the edge appears twice
            words = sum(len(s["text"].split()) for s in cur)
        cur.append(seg)
        words += n
    if cur:
        chunks.append(cur)
    out = []
    for c in chunks:
        out.append({
            "start": c[0]["start"],
            "end": c[-1]["start"] + c[-1]["duration"],
            # one [m:ss] marker per ~30 s keeps the prompt small but lets the model cite times
            "text": render(c),
        })
    return out

def render(seg_list, every=30):
    lines, last = [], -every
    for s in seg_list:
        if s["start"] - last >= every:
            lines.append(f"\n[{fmt(s['start'])}] ")
            last = s["start"]
        lines.append(s["text"] + " ")
    return "".join(lines).strip()

We ran this on the lecture. 2,955 segments became 15 chunks of about 1,500 words each. The time ranges are what you would hope for: the first chunk covers 0:00 to 8:01, the second 7:49 to 15:52 (the overlap is the 12-second overlap between them), the third 15:38 to 24:07, and the last 1:53:07 to 1:56:21 with 598 words. The text of a chunk starts like this:

[7:49] we're going to focus on the Transformer neural network itself okay so I created a new Google collab uh jup notebook here and this will allow me to later easily share this code ...

Note the uh and the missing punctuation. That is what auto-generated captions look like. Models cope with it fine, and the summaries they write are cleaner than the input, but do not be surprised when the raw text looks rough.

Choosing the chunk size

1,500 words is a default, not a law. Smaller chunks (500 to 800 words) give more detailed notes and more calls. Larger chunks (3,000+) are cheaper but start losing the middle of each chunk. For lectures and tutorials, 1,000 to 2,000 words is a sensible range. For a fast podcast with lots of topic changes, go smaller.

Step 3: map, then reduce

This is the classic map-reduce summarisation pattern:

Map: ask the model for short notes on each chunk, keeping the [m:ss] markers of the moments it mentions.
Reduce: give the model all the notes (not the full transcript) and ask for the final summary, again citing times.

The reduce step only sees the notes, which are a small fraction of the transcript, so it fits in a single prompt even for long videos. Here is the whole pipeline, including the fetch and the time-to-link rewrite:

import json, os, re, requests
from chunker import chunk_segments, fmt   # the chunker above, saved as chunker.py

API = "https://siftwright.com/v1/youtube-transcript"

def get_video(url):
    r = requests.post(API, headers={"Authorization": f"Bearer {os.environ['SIFTWRIGHT_API_KEY']}"},
                      json={"urls": [url], "outputFormats": ["segments"]}, timeout=300)
    r.raise_for_status()
    item = r.json()["results"][0]
    if item["status"] != "ok":
        raise LookupError(item.get("error", item["status"]))
    return item

def summarize_video(url, llm):
    """llm(prompt: str) -> str is any chat-model call you already have."""
    v = get_video(url)
    chunks = chunk_segments(v["segments"])
    notes = []
    for c in chunks:                                   # map: one call per chunk
        notes.append(llm(
            "Summarize this part of a video transcript in 3-5 bullets. "
            "Keep the [m:ss] markers of the moments you mention.\n\n" + c["text"]))
    joined = "\n\n".join(f"## {fmt(c['start'])}-{fmt(c['end'])}\n{n}" for c, n in zip(chunks, notes))
    final = llm("Combine these section notes into a summary with a short overview and "
                "5-8 key points. Cite times like [12:34].\n\n" + joined)       # reduce
    # turn [h:mm:ss] / [m:ss] into links that open the video at that moment
    def link(m):
        parts = [int(p) for p in m.group(1).split(":")]
        secs = sum(p * 60 ** i for i, p in enumerate(reversed(parts)))
        return f"[{m.group(1)}](https://youtu.be/{v['videoId']}?t={secs})"
    return v["title"], re.sub(r"\[(\d+(?::\d{2}){1,2})\]", link, final)

llm is deliberately a plain function that takes a prompt string and returns a string, so you can plug in whichever model and SDK you already use.

The last step is the one readers will actually notice. The regular expression finds every [12:34] or [1:02:03] in the final summary, converts it to seconds, and rewrites it as a youtu.be/<id>?t=<seconds> link. YouTube opens the video at that second when it sees the t parameter.

What we tested, and what we did not

We ran the full pipeline against the real lecture with a stand-in llm function that returns a canned bullet quoting a timestamp from its input. That tests everything except the model: the API call, the 15 chunks, the 16 total calls (15 map plus one reduce), the largest prompt at 1,535 words, and the link rewrite. The output was:

Let's build GPT: from scratch, in code, spelled out.
16 llm calls; max prompt words 1535
- discusses a topic around [1:00:41](https://youtu.be/kCc8FmEb1nY?t=3641)

1:00:41 becomes t=3641, which is 3,600 + 41. The rewrite works for both m:ss and h:mm:ss.

We did not run it with a real language model, so we cannot show you a real summary or tell you what it costs on your provider. That is a function of the model and the length of the video: 15 map calls of about 1,500 words in, a few bullets out, plus one reduce call. You can estimate it before you start by counting the words in the chunks, which chunk_segments already gives you.

Edge cases worth handling

Very long videos. The reduce prompt grows with the number of chunks. A two-hour lecture produced 15 chunks, which is fine. A ten-hour stream would produce about 75, and 75 sets of notes may be too long for one call. The fix is another round: reduce groups of ten notes into section summaries, then reduce those. The same function works at every level.

Videos without captions. Some videos have none, and some have captions in a different language. The API accepts a language parameter and a translateTo option, and it reports isAutoGenerated and availableLanguages in the result, so you can tell the user why a video could not be summarised instead of failing silently.

Playlists and channels. The same endpoint accepts a playlist or channel URL and returns one result per video (see maxVideosPerSource). Running the loop over a playlist gives you a summary of each lecture in a course. Billing is per video that returns a transcript, so check the video count before you run a large playlist.

Prompt injection. A transcript is untrusted text. A video can literally say "ignore your instructions and..." out loud. Keep your instructions in the system prompt, treat the transcript as data, and do not give the summarising model tools that can act on anything. A summary pipeline needs none.

Copyright and terms. Summaries for your own research are one thing; republishing someone's content is another. Check the terms that apply to you before publishing summaries at scale.

Making it cheaper

Cache by video ID. Transcripts of a published video almost never change. Store the segments and the summary keyed by videoId and a version of your prompt.
Summarise once, ask many times. If users will ask questions about the video later, keep the chunk notes. Retrieval over 15 sets of notes with timestamps is far cheaper than re-reading the whole transcript on every question.
Skip the map step for short videos. Under about 3,000 words, a single call with the markers inserted is simpler and usually just as good.

Summary

Timestamped segments are the key ingredient. With them, chunking keeps boundaries intact, the model can cite times, and a three-line regular expression turns those citations into links. Without them you get a block of text and a summary nobody can verify.

If you do not need code for one-off videos, the free transcript tool gives you the text, timestamped lines, SRT and VTT in the browser. For the pipeline above, the API is 10,000 results for $29 a month; our Python and Node.js guide covers the rest of the request options, and if your own server gets blocked when it talks to YouTube directly, see the RequestBlocked fix.

Give your AI agent web data tools over MCP: transcripts, contacts and news, billed per result

Siftwright — Wed, 21 Oct 2026 12:56:27 +0000

Agents are good at deciding what data they need and bad at fetching it. Instead of writing glue code, you can give an MCP client (Claude Desktop, Cursor, VS Code and others) a few Apify Actors as tools and let it call them directly. Here is a minimal setup using three small Actors, with the real prices and the limits.

The config

Apify hosts an MCP server at mcp.apify.com. The tools parameter limits it to the Actors you name, and your Apify API token goes in an Authorization header:

{
  "mcpServers": {
    "siftwright-data": {
      "url": "https://mcp.apify.com?tools=siftwright/youtube-transcript-extractor,siftwright/contact-details-extractor,siftwright/google-news-scraper",
      "headers": {
        "Authorization": "Bearer <APIFY_TOKEN>"
      }
    }
  }
}

Apify adds run and dataset helper tools (such as fetching dataset items) automatically, so the agent can read results. Check Apify's MCP docs for the current format; it has changed before.

The three tools

Actor	What it returns	Price
YouTube Transcript Extractor	Transcript of a video as text, SRT or VTT	$0.003 per transcript
Contact Details Extractor	Emails, phones and social links from a website	$0.002 per website
Google News Scraper	News articles for a search topic	$0.0015 per article

These are pay-per-event Actors: you are charged for results that were produced, and failed lookups are not charged. Apify also charges a tiny per-run start fee ($0.00005).

Three prompts to try

"Get the transcript of this video and give me five bullet points: "
"Find a contact email and LinkedIn page for example.com, acme.io and foo.dev."
"What are the 10 latest news stories about ? Group them by theme."

The agent picks the Actor, fills the input, runs it and reads the dataset. A single call costs a fraction of a cent.

Honest limits

Running Actors needs an Apify account and token. Without one, the server only offers search and docs tools.
YouTube blocks some server IPs, so a small share of videos can fail. Those are not charged.
Contact extraction only finds what is published on the site's own pages. It does not guess addresses.
Agents can loop. Set a spending limit on your Apify account and keep the tools list short.
Siftwright also runs its own hosted MCP server for its API plans, which needs a paid key. The Apify route above does not.

Try it without an agent first

The free browser tools at siftwright.com/tools run the same lookups with no signup, so you can see the output shape before wiring anything up.

Fix youtube-transcript-api RequestBlocked and IpBlocked on servers

Siftwright — Tue, 20 Oct 2026 14:47:40 +0000

The script worked yesterday. It fetched a transcript in half a second on your laptop, you wired it into a worker, deployed it, and now every call ends with this:

RequestBlocked:
Could not retrieve a transcript for the video https://www.youtube.com/watch?v=jNQXAC9IVRw!
This is most likely caused by:

YouTube is blocking requests from your IP. This usually is due to one of the following reasons:
- You have done too many requests and your IP has been blocked by YouTube
- You are doing requests from an IP belonging to a cloud provider (like AWS, Google Cloud Platform, Azure, etc.). ...

That is the real output of youtube-transcript-api 1.2.4, produced on 20 October 2026 from a cloud container, for the first YouTube video ever uploaded. Nothing is wrong with your code. This post explains what the error means, which fixes work, which ones to avoid, and shows a small, tested fallback you can drop into an existing pipeline.

What RequestBlocked and IpBlocked mean

The library is open source and does not use an official YouTube API. It makes the same requests a browser makes when it plays a video: it asks YouTube for the player data, finds the caption tracks in the response, and downloads the one you asked for. YouTube answers those requests differently depending on where they come from.

Both exceptions live in youtube_transcript_api._errors, and they are related:

RequestBlocked is the general case: YouTube refused to serve the player data to your address.
IpBlocked is a subclass of RequestBlocked. You can catch RequestBlocked and cover both.

The library's own error text names the two usual causes. You made a lot of requests from one address, or your address belongs to a cloud provider. Home and office connections are rarely blocked. Datacenter ranges (AWS, GCP, Azure and most VPS hosts) frequently are. That is why the same code passes on a laptop and fails the moment it is deployed.

Two practical consequences:

Retrying does not help. We ran three attempts, two seconds apart, from the blocked container. All three raised RequestBlocked. A block is a property of the address, not a transient glitch, so a retry loop only adds latency.
A block hides the real answer. When YouTube refuses you, the library cannot tell you whether the video has captions at all. A video that does not exist also raises RequestBlocked from a blocked address. You learn about the video only after you get past the block.

Not every failure is a block. The same module defines TranscriptsDisabled, NoTranscriptFound, VideoUnavailable, AgeRestricted, PoTokenRequired and others. Those describe the video or the request, and switching IPs will not change them. Your error handling should treat the two groups differently, which the fallback code below does.

Your options, honestly compared

1. Run it somewhere that is not a datacenter

If this is a personal script, run it on your own machine or a home server. It is free and it works. It stops being an answer when the job needs to run on a schedule in the cloud, or when more than one person's work depends on it.

2. Route requests through a proxy

The library has first-class proxy support. WebshareProxyConfig takes a username and password for residential proxies, and GenericProxyConfig takes any HTTP or HTTPS proxy URL:

from youtube_transcript_api import YouTubeTranscriptApi
from youtube_transcript_api.proxies import GenericProxyConfig

api = YouTubeTranscriptApi(
    proxy_config=GenericProxyConfig(
        http_url="http://user:pass@proxy.example.com:8000",
        https_url="http://user:pass@proxy.example.com:8000",
    )
)
snippets = api.fetch("jNQXAC9IVRw").snippets

We did not run this one, because we have no proxy to point it at. The signature above is from the installed library. The caveats are worth knowing before you buy anything. The proxy has to be residential or otherwise not on a blocked range, since a datacenter proxy has the same problem as your server. You pay for it by traffic or by IP. And you take over the monitoring: when the pool gets flagged, your jobs fail until you notice.

3. Authenticate with cookies

The library's own error message mentions that authenticating requests with cookies can keep things working for a while, and in the same breath warns that YouTube may eventually permanently ban the account used. Do not tie an account you care about to a scraper.

4. Call an API that does the proxying for you

This moves the problem to a service whose whole job is to keep requests working. You send a video URL and get the transcript back as JSON, from any server. The tradeoff is cost: you pay per result instead of paying for proxy traffic. Whether that is cheaper depends on your volume. A few hundred transcripts a month is usually cheaper through an API than through a proxy plan you have to configure and watch. Millions a month is a different calculation, and at that scale you should price both.

We run one of these: the Siftwright YouTube Transcript API, and a free transcript tool in the browser for one-off videos. We are biased, so here is the useful part regardless of whose API you pick: the pattern below works with any provider.

The pattern: use the library first, fall back only on blocks

You do not have to rip out a working library to fix a blocked deployment. A good compromise is to keep the free path as the default and call the paid path only when YouTube blocks you. That keeps the cost proportional to the problem.

The rules for the wrapper:

Fall back only on RequestBlocked (which includes IpBlocked).
Do not fall back on TranscriptsDisabled, NoTranscriptFound or VideoUnavailable. The answer is "no", and a second service will say the same thing and may charge you to say it.
Return which path produced the text, so you can see in your logs how often you are blocked.

import os

import requests
from youtube_transcript_api import YouTubeTranscriptApi
from youtube_transcript_api._errors import (
    IpBlocked,
    NoTranscriptFound,
    RequestBlocked,
    TranscriptsDisabled,
    VideoUnavailable,
)

API = "https://siftwright.com/v1/youtube-transcript"
KEY = os.environ["SIFTWRIGHT_API_KEY"]


def via_library(video_id):
    snippets = YouTubeTranscriptApi().fetch(video_id, languages=["en"]).snippets
    return " ".join(s.text for s in snippets)


def via_api(video_id):
    r = requests.post(
        API,
        headers={"Authorization": f"Bearer {KEY}"},
        json={
            "urls": [f"https://www.youtube.com/watch?v={video_id}"],
            "outputFormats": ["text"],
        },
        timeout=300,
    )
    r.raise_for_status()
    item = r.json()["results"][0]
    if item["status"] != "ok":
        raise LookupError(item.get("error", item["status"]))
    return item["text"]


def get_transcript(video_id):
    """Return (source, text). Only network blocks fall back."""
    try:
        return "library", via_library(video_id)
    except (RequestBlocked, IpBlocked):
        return "api", via_api(video_id)
    except (TranscriptsDisabled, NoTranscriptFound, VideoUnavailable) as err:
        raise LookupError(type(err).__name__) from err


for vid in ["jNQXAC9IVRw", "dQw4w9WgXcQ", "aaaaaaaaaaa"]:
    try:
        source, text = get_transcript(vid)
        print(vid, source, len(text.split()), "words:", text[:50])
    except LookupError as err:
        print(vid, "no transcript:", err)

Install with pip install youtube-transcript-api requests. We ran exactly this file from the same blocked container with a real key:

jNQXAC9IVRw api 39 words: All right, so here we are, in front of the elephan
dQw4w9WgXcQ api 487 words: [♪♪♪] ♪ We're no strangers to love ♪ ♪ You know th
aaaaaaaaaaa no transcript: This video is unavailable

Read the output carefully, because it shows the pattern working as designed. The first two videos were blocked by the library, so both went through the API (api). On a laptop they would print library and cost nothing. The third ID does not exist. The library was blocked on it, so it fell back, and the API gave the real answer: This video is unavailable.

That call is also a small proof of the billing rule. The request contained two URLs in an earlier test, one real and one dead, and the response reported "count": 2, "billed": 1. Videos that return no transcript are not billed.

Doing it in bulk

If you are fetching many videos, do not call the fallback once per video. The API accepts a list of URLs in urls and returns one result per video, so you can send the blocked ones in a single request. A simple structure:

Loop over IDs with the library, collecting the ones that raise RequestBlocked.
Stop trying the library once the first few calls in a run are all blocked. If three in a row fail, the address is blocked and the rest will fail too. Send everything remaining straight to the API.
Send the collected IDs in batches to the API with outputFormats set to ["text"] (or ["text", "segments"] if you need timestamps).

Step 2 matters more than it looks. In our test a blocked attempt took under a second, which sounds harmless until you multiply it: 1,000 videos against a blocked address is more than ten minutes of guaranteed failures before the real path even starts.

For the full response shape, languages, SRT and VTT output, and playlist handling, see our YouTube transcript API guide for Python and Node.js and the API docs.

Other things worth checking before you blame the IP

A few errors look similar but have a different cause:

TranscriptsDisabled or NoTranscriptFound: the creator turned captions off, or no track exists in the language you requested. Check the languages list, and consider asking for a fallback language. No proxy fixes this.
AgeRestricted: the video is age-gated, so anonymous requests cannot read it. Skip it or handle it manually.
PoTokenRequired and YouTubeDataUnparsable: YouTube changed what its player returns, and the library needs an update. Run pip install -U youtube-transcript-api before anything else, and pin a version in production so an upgrade is a choice you make.
Works for some videos only: that is a hint that the failing ones have a video-level restriction rather than an address problem.

Choosing, in one paragraph

If you run it yourself, on your own connection, at low volume: keep using the library and do nothing else. If it must run in the cloud and you have a few hundred transcripts a month, use the fallback pattern above so you only pay when you are blocked. If YouTube transcripts are central to your product and your volume is large, price a managed proxy pool against a per-result API with your real numbers before deciding. And in every case, handle the "genuinely no captions" errors separately, because they are answers and not failures.

If you want to try the API path without writing code first, the free transcript tool runs the same extraction in your browser. For code, plans start at $29/month with 10,000 results included.

Collect company social links, emails and phones from a list of websites in Python

Siftwright — Mon, 19 Oct 2026 12:54:26 +0000

Company websites already publish their social profiles and contact details; the annoying part is collecting them for a whole list of domains. Here is a small Python script that does it through an Apify Actor billed per website ($2 per 1,000 sites, and sites that fail to load are not charged).

The script

import os, requests

TOKEN = os.environ["APIFY_TOKEN"]
URL = ("https://api.apify.com/v2/acts/siftwright~contact-details-extractor/"
       "run-sync-get-dataset-items")

sites = ["https://www.python.org", "https://github.com", "https://www.mozilla.org"]
r = requests.post(URL, params={"token": TOKEN, "timeout": 90},
                  json={"startUrls": sites, "maxPagesPerDomain": 3}, timeout=120)
r.raise_for_status()

for row in r.json():
    print(row["domain"], row["status"], row["emails"], row["phones"],
          row["linkedin"], row["twitter"])

Real output

I ran exactly this on 2026-10-19. Each site returned status: "ok" with 3 pages scanned:

domain	linkedin	twitter / X	instagram	emails	phones
www.python.org	python-software-foundation	ThePSF	none	none	none
www.mozilla.org	mozilla-corporation	none	mozilla	none	none
github.com	github	github	github	none	none

Note the empty email and phone columns: these three sites simply don't publish an address on the pages scanned, and the Actor does not guess. Smaller company sites with a visible contact page are where emails and phones show up. Raise maxPagesPerDomain if the contact page is deeper in the site.

Pricing and limits

One charge per website that loads, $0.002 each; dead domains are free.
It reads public pages only: no logins, no proxies by default, so heavily protected sites may return a failed status.
Fields per site: emails, phones, facebook, twitter, linkedin, instagram, youtube, pagesScanned.

Try it: Contact Details Extractor on Apify. It also works from AI agents through Apify's MCP server. More tools: siftwright.com.

Build a daily news brief in Python with Google News search, billed per article

Siftwright — Sun, 18 Oct 2026 19:09:08 +0000

Need fresh news headlines in a script, a RAG pipeline or a media-monitoring job, without scraping Google or wiring up a paid news API? Here is a small Python brief generator built on the Google News Scraper Apify Actor. It is billed per article returned ($1.50 per 1,000), and an empty or failed search costs nothing.

Input

The Actor takes query, maxItems (1 to 200), language and country. Each result has title, link, source, publishedAt (ISO 8601), snippet and status.

The script

import os
from collections import Counter
import requests

TOKEN = os.environ["APIFY_TOKEN"]
URL = "https://api.apify.com/v2/acts/siftwright~google-news-scraper/run-sync-get-dataset-items"

def news(query, n=10, language="en", country="US"):
    r = requests.post(URL, params={"token": TOKEN},
                      json={"query": query, "maxItems": n, "language": language, "country": country},
                      timeout=180)
    r.raise_for_status()
    return [a for a in r.json() if a.get("status") == "ok"]

watchlist = ["open source LLM", "vector database"]
for q in watchlist:
    items = sorted(news(q), key=lambda a: a["publishedAt"], reverse=True)
    print(f"\n## {q} ({len(items)} articles)")
    for a in items[:3]:
        print(f"- {a['publishedAt'][:10]} {a['source']}: {a['title']}\n  {a['link']}")
    print("top sources:", Counter(a["source"] for a in items).most_common(3))

run-sync-get-dataset-items starts the run and returns the dataset in one HTTP call, which suits short jobs like this one.

Real output

I ran it on 2026-10-18 with a watchlist of two queries and 10 articles each. Sorted newest first, the top hit for "open source LLM" was a HackerNoon piece dated 2026-10-18 on GPU memory for running open LLMs locally, and for "vector database" an article dated 2026-10-13 about Elastic's serverless vector database. Your results will differ as the news changes.

Two notes from testing:

link values are Google News redirect URLs. They open the article in a browser, but if you need the publisher URL you have to resolve the redirect yourself.
snippet is often just the title plus the outlet name, so don't count on it for body text.

Variations

Change language and country (for example de and DE) to read another Google News edition.
Schedule the same Actor on Apify and send the dataset to a webhook for daily monitoring.
It also works with AI agents through Apify's MCP server, so an assistant can run the search itself.

More details and a curl example: siftwright.com/google-news-api. Questions: support@siftwright.com.

Full-page website screenshots and PDFs from Python or curl, billed per capture

Siftwright — Mon, 12 Oct 2026 20:54:31 +0000

Screenshots and PDFs of web pages are a common chore: link previews, visual regression checks, archiving a page as evidence, feeding page images to a vision model. Running your own headless browser works, but you end up maintaining Chromium, fonts and timeouts. Here is a small pay-per-result alternative, with the real output of a test run.

I am building Pageframe, an Apify Actor that captures full-page PNG/JPEG screenshots or print-ready PDFs. It costs $0.0015 per successful capture, and failed URLs are not charged.

The input

{
  "urls": ["https://example.com"],
  "format": "png",
  "fullPage": true,
  "viewportWidth": 1280,
  "viewportHeight": 800,
  "deviceScaleFactor": 1,
  "blockCookieBanners": true,
  "delaySec": 0
}

Other options: format can be png, jpeg or pdf (with pdfPageFormat A4, A3, Letter or Legal), deviceScaleFactor 2 or 3 gives retina captures, waitForSelector waits for an element before capturing, and blockCookieBanners makes a best-effort click on common consent buttons.

From Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("siftwright/pageframe-screenshots").call(run_input={
    "urls": ["https://example.com"],
    "format": "png",
    "fullPage": True,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item["status"], item["fileUrl"])

From curl

curl -X POST "https://api.apify.com/v2/acts/siftwright~pageframe-screenshots/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com"],"format":"pdf","pdfPageFormat":"A4"}'

Real output

This is the dataset item from an actual run on Apify, capturing https://example.com:

{
  "url": "https://example.com",
  "status": "ok",
  "format": "png",
  "fileUrl": "https://api.apify.com/v2/key-value-stores/iSfPTldM2Pvavg8Ac/records/capture-1791797595722-zokzr1.png",
  "sizeBytes": 71152,
  "durationMs": 289
}

The run billed one capture event (plus Apify's tiny per-run start fee). The image itself is stored in the run's key-value store at fileUrl.

Honest limits

Pages behind a login are not supported.
Cookie banner dismissal is best-effort; unusual consent widgets may still show.
Heavy pages that load content late may need waitForSelector or delaySec.

Use it from an AI agent

Every Siftwright Actor works through Apify's MCP server, so an agent can call Pageframe as a tool. There is also a free tool page at siftwright.com if you just want to try one capture first.

Feedback or a missing option? Write to support@siftwright.com.

Get YouTube transcripts in bulk: text, SRT and VTT from Python, pay only per result

Siftwright — Wed, 07 Oct 2026 12:55:17 +0000

Getting a YouTube transcript sounds trivial until you need it for 500 videos: the official Data API doesn't return caption text for videos you don't own, libraries break when YouTube changes a page, and a plain requests loop gets rate-limited fast.

This post shows a small, honest workflow: send video URLs or IDs to an Apify Actor, get back clean text, timestamped segments and ready-to-use SRT/VTT subtitles, and only pay for transcripts that were actually returned.

Try it without writing code

Paste a link into the free tools page at https://siftwright.com/tools/ or open the Actor directly: https://apify.com/siftwright/youtube-transcript-extractor

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("siftwright/youtube-transcript-extractor").call(run_input={
    "urls": ["jNQXAC9IVRw"],          # URL, Shorts link, playlist, channel or bare ID
    "language": "en",
    "outputFormats": ["text", "segments", "srt"],
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["status"], item["wordCount"])
    print(item["text"])

Real output

I ran it on the first video ever uploaded to YouTube ("Me at the zoo", 19 seconds). The result, trimmed:

{
  "videoId": "jNQXAC9IVRw",
  "status": "ok",
  "title": "Me at the zoo",
  "channelName": "jawed",
  "durationSeconds": 19,
  "language": "en",
  "isAutoGenerated": false,
  "availableLanguages": [{"code": "en"}, {"code": "de"}],
  "wordCount": 39,
  "text": "All right, so here we are, in front of the elephants the cool thing about these guys is that they have really... really really long trunks and that's cool (baaaaaaaaaaahhh!!) and that's pretty much all there is to say"
}

The srt field comes back as a ready-to-save subtitle file:

1
00:00:01,200 --> 00:00:03,360
All right, so here we are, in front of the elephants

2
00:00:05,318 --> 00:00:07,974
the cool thing about these guys is that they have really...

curl

curl -X POST "https://api.apify.com/v2/acts/siftwright~youtube-transcript-extractor/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"urls":["jNQXAC9IVRw"],"outputFormats":["text","srt"]}'

Useful options

Playlists and channels: pass a playlist or @handle URL and set maxVideosPerSource.
Translation: translateTo: "de" uses YouTube's built-in machine translation.
Prefer human captions: on by default; falls back to auto-generated.
Metadata (title, channel, views, keywords) is included at no extra charge.

Pricing and limits

Pricing is pay per event: $0.003 per transcript returned ($3 per 1,000). Videos with no captions, private videos and failures are reported in the dataset with a status and are not charged. Limits: a video needs captions (manual or auto-generated) to have a transcript, and some fields such as publish date can be empty depending on what YouTube exposes.

Using it from an AI agent

The same Actor works through Apify's MCP server, and Siftwright also runs a hosted MCP endpoint at https://siftwright.com/mcp so an agent can fetch transcripts as a tool call.

If something breaks on a video you care about, tell us at support@siftwright.com and include the video ID.

Bulk domain age, expiry and DNS audit in Python: RDAP with WHOIS fallback

Siftwright — Mon, 05 Oct 2026 19:10:36 +0000

A domain list is a surprisingly useful dataset. Which of your 500 customer domains expire this quarter? Which lead domains were registered last month (often a trust signal)? Which of your own domains have no DMARC record? All of it is public data, but getting it for a whole list means juggling RDAP, legacy WHOIS servers and DNS lookups.

This tutorial uses one Apify Actor, Domain WHOIS & DNS Lookup, to turn a list of domains into clean JSON rows: registrar, creation and expiry dates, age, nameservers, plus MX, SPF and DMARC. I'm the maker, so the pricing is stated up front: $1 per 1,000 domains, and invalid inputs, duplicates and failed lookups are free.

The code

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("siftwright/domain-whois-dns-lookup").call(run_input={
    "domains": ["stripe.com", "github.io", "https://www.bbc.co.uk/news", "spiegel.de", "this-name-should-not-exist-siftwright-test.com"],
})
for d in client.dataset(run.default_dataset_id).iterate_items():
    if not d["registered"]:
        print(f'{d["domain"]:<14} not registered (likelyAvailable={d["likelyAvailable"]})'); continue
    dns = d.get("dns") or {}
    soon = d["daysUntilExpiry"] is not None and d["daysUntilExpiry"] < 90
    print(f'{d["domain"]:<14} {d["registrar"] or "?":<30} age {d["ageDays"]} days  src={d["source"]}'
          f'{"  EXPIRES<90d" if soon else ""}{"  NO-DMARC" if not dns.get("dmarc") else ""}{"  NO-SPF" if not dns.get("spf") else ""}')

Install with pip install apify-client (3.x; on 2.x use run["defaultDatasetId"]) and set APIFY_TOKEN.

Real output

This is the output of that exact script, run on 2026-10-05 (the last row is a made-up name, so it comes back unregistered):

this-name-should-not-exist-siftwright-test.com not registered (likelyAvailable=True)
stripe.com     SafeNames Ltd.                 age 11346 days  src=rdap
github.io      MarkMonitor Inc.               age 4958 days  src=whois  NO-DMARC
spiegel.de     ?                              age None days  src=whois
bbc.co.uk      British Broadcasting Corporation age 11619 days  src=rdap

Things worth noticing:

https://www.bbc.co.uk/news was reduced to the registrable domain bbc.co.uk automatically.
.io-style and .de domains have no usable RDAP, so the Actor falls back to classic WHOIS (src=whois).
.de registries publish very little. spiegel.de has no registrar or creation date in public data, so those fields are null rather than guessed. Check for None before doing arithmetic.
Registrant names and emails are usually redacted because of GDPR. Don't build on them.

Ideas to build on it

Expiry watchlist: run weekly on your portfolio and alert when daysUntilExpiry < 60.
Lead scoring: low ageDays plus no MX can flag throwaway domains.
Email deliverability audit: list every domain missing SPF or DMARC.

Pricing and limits

Pay-per-event at $0.001 per domain looked up (registered or not). Apify's free plan includes monthly credits, so you can try it at no cost. AI agents can also call it through Apify's MCP server: https://mcp.apify.com?tools=siftwright/domain-whois-dns-lookup.

Full details and the product page: siftwright.com/domain-whois-lookup/. Questions: support@siftwright.com.

Clean a signup list in bulk: MX checks, disposable domains and typo suggestions in Python

Siftwright — Sat, 03 Oct 2026 04:53:49 +0000

Bad addresses in a signup list cost you bounces, sender reputation and wasted sends. Most checks you can run without sending a single email: syntax, whether the domain has MX records, whether it is a disposable provider, and whether it looks like a typo of a common domain. This tutorial runs those checks in bulk with the Bulk Email Validator Actor on Apify.

What it checks

Syntax and domain split
MX lookup (does the domain accept mail at all)
Disposable-provider detection
Role addresses (info@, admin@), free providers, plus-tags
Typo suggestions (gmial.com -> gmail.com)

It does not do SMTP mailbox probing, so a "deliverable" verdict means the domain can receive mail, not that the specific mailbox exists. Pricing is pay-per-event, $0.0008 per email, and it only charges for emails it actually returns.

Run it from Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("siftwright/email-validator-mx-disposable").call(run_input={
    "emails": ["jane@github.com", "test@mailinator.com", "someone@gmial.com"],
    "checkMx": True,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["email"], item["verdict"], item["reason"], item["suggestion"])

Real output

This is the actual result of a test run with those three addresses (3 emails returned, 3 charged):

email	verdict	reason	note
jane@github.com	deliverable	-	MX: github-com.mail.protection.outlook.com
test@mailinator.com	risky	disposable	has MX, but a throwaway provider
someone@gmial.com	risky	possible_typo	suggestion: someone@gmail.com, no MX

Cleaning a list

Keep deliverable, drop or review risky, and discard anything invalid:

clean = [i["email"] for i in items if i["verdict"] == "deliverable"]
review = [(i["email"], i["reason"], i["suggestion"]) for i in items if i["verdict"] == "risky"]

The typo suggestions are useful for signup forms: show "Did you mean someone@gmail.com?" instead of silently accepting a bad address.

Limits

No SMTP mailbox verification, so catch-all domains and non-existent mailboxes on valid domains pass the MX check. Use it as a cheap first filter before sending, not as a guarantee.

Actor: https://apify.com/siftwright/email-validator-mx-disposable