DEV Community: CleanScrape

Turning podcasts into text: a 52-minute NASA episode, 8,022 words, and the bugs real audio found

CleanScrape — Sat, 31 Oct 2026 17:41:45 +0000

A podcast holds a lot of useful information, and almost none of it can be searched, quoted or passed to another tool. So I built a podcast transcription tool on Apify. You paste a feed, an Apple Podcasts link or a link to an audio file, and you get the text back with timestamps and subtitle files.

The input

One link is enough. This is the run from the video, an episode of NASA's Houston We Have a Podcast:

{
  "sources": ["https://podcasts.apple.com/us/podcast/the-moon-base/id1262594123?i=1000789048885"],
  "quality": "accurate"
}

It also accepts an RSS feed, a podcast's website, or a direct link to an MP3, M4A, WAV or MP4 file. Standard quality uses the Whisper base model and Accurate uses Whisper small. Both run inside the Actor, so there's no OpenAI key to bring.

The output

The 52:18 episode became 8,022 words in 23 minutes on Accurate. Each row has the show, the title, the date, the length, the detected language, the full text and timestamped segments:

{
  "title": "The Moon Base",
  "duration": "00:52:18",
  "language": "en",
  "wordCount": 8022,
  "segments": [
    { "start": 80.5, "text": "a home for astronauts on another celestial body." }
  ]
}

You also get TXT, SRT and VTT files, so the transcript can go straight onto a video or into a search index.

What real audio taught me

The first version passed its tests on short clean clips. Real podcasts found the interesting bugs.

Memory grows with the audio. Whisper's memory use went up by about 55 MB per minute of audio. A 5-minute test looked fine, and a 5-hour file crashed a 4 GB run. The fix was to feed it 5-minute chunks, each cut at the quietest tenth of a second near the boundary so no word gets split. A 5-hour file now peaks at 1.75 GB.

The first 30 seconds can lie. Whisper detects the language from the start of the file. Podcasts often start with an ad, and one Finnish episode opened with an English ad, so the whole episode was transcribed as English. It now takes short samples across the file and lets them vote, weighted by confidence.

The same show, pasted twice, cost double. A feed link and an Apple link for one show are different links to the same audio. Episodes already in the run are now skipped, whichever link they came from.

Real speech costs more than a test sample. My first benchmark used a clean narration. Real conversation ran about 25% slower, and Finnish was the slowest of the languages I tried. That changed the price of the Accurate tier before launch.

Following a show

Give a run a watch name and put it on a schedule. The first run takes the newest episode, and each later run transcribes only what has appeared since. A check with nothing new costs $0.003.

What it can't do

It doesn't label speakers. It works on one language per file. Some podcast hosts refuse automated downloads: in a test of 134 shows, the newest episode of 127 could be reached. The rest are reported in the run and their audio isn't charged. YouTube, TikTok and Spotify links aren't supported.

Try it

https://apify.com/cleanscrape/podcast-transcription

$6 per 1,000 audio minutes on Standard and $15 on Accurate, less on paid Apify plans. The episode above cost $0.80. You can export to Excel, CSV or JSON, or call it from your own code or an AI agent through Apify's MCP server.

Niko, CleanScrape developer

An economic calendar built only from official sources: the Fed, ECB, BLS, ONS and more in one table

CleanScrape — Tue, 27 Oct 2026 16:07:47 +0000

Every number on an economic calendar has a release time set by an official agency: the Bureau of Labor Statistics for CPI and jobs, the BEA for GDP, the Fed and the ECB for rate decisions. Those agencies all publish their schedules openly, but each in its own format: an iCalendar file, a JSON feed, an HTML table, a paged API. So I built a small tool on Apify that reads 13 of those official calendars and returns one clean table.

The input

Pick economies and dates. Everything else is optional.

{
  "countries": ["US", "EA", "UK", "CA", "AU"],
  "dateRange": "next7",
  "minimumImpact": "medium"
}

That run took 26 seconds and saved 20 releases for 27 October to 2 November 2026. Wednesday had Australia's CPI at 00:30 UTC, the Bank of Canada at 13:45 and the Fed's decision at 18:00. Thursday had US GDP at 12:30 and the ECB at 13:15. The UK had no medium or high impact releases that week.

Times you can trust, and when you can't

Every row has the time in UTC and in the agency's local time, plus a status:

confirmed: the agency published the exact time (14 of the 20)
standard: a central bank's usual time, like the ECB's 14:15 Frankfurt time
provisional: a weekly schedule that can shift around holidays (US jobless claims)
date only: the agency publishes the day but not the time

Past releases come with values

For past CPI releases the tool adds the value. US CPI and euro area HICP carry the first published annual rate; the UK, Canada and Australia carry the current official series value. One run over the past year returned 76 inflation releases from all five economies. In the euro area, inflation went from 1.7% in January 2026 to 3.8% in September.

There are no consensus forecasts: those are compiled privately and are not free to reuse.

Following changes

Give the run a watch name and schedule it daily. After the first run, each run returns only what changed: new, moved, cancelled or released events.

Try it

https://apify.com/cleanscrape/economic-calendar-api

$3 per 1,000 events plus half a cent per country checked, less on paid Apify plans. The week above cost $0.08. Export to Excel, CSV or Google Sheets, or call it from your own code or an AI agent through Apify's MCP server.

Niko, CleanScrape developer

Every cut in a 15-minute film: extracting 166 scene frames from a video link

CleanScrape — Sun, 25 Oct 2026 14:21:19 +0000

If you want to send a video to an AI vision model, build a thumbnail picker, or index a lecture by its slides, you rarely want every frame. You want one frame per shot. So I built a small tool on Apify that takes a video link and returns those frames as images, with their exact times.

Here is what it does on Sintel, the 14:48 open movie from the Blender Foundation.

The input

{
  "videos": ["https://archive.org/download/Sintel/sintel-2048-stereo.mp4"],
  "mode": "scenes",
  "contactSheet": true
}

It found 166 shots and saved one frame for each. The scan took 3 minutes 22 seconds.

How it finds the cuts

In this precise mode it compares every frame with the one before. When the picture changes completely, that is a cut, and a frame is saved there. When the camera only pans or zooms, the picture changes a little at a time; that is not counted as a cut. (An earlier version did count fast camera moves, which is how I learned to require a sharp peak rather than a large change.)

When you don't need every cut

Overview mode reads only the keyframes, the full pictures a video file stores every few seconds. On the same film it returned 24 frames in 21 seconds.

Slides and screen recordings

On a 60-second lecture recording with a moving mouse pointer, precise mode with high sensitivity found all 6 slides (at 0:00, 0:10, 0:20, 0:30, 0:40 and 0:50) and ignored the pointer.

What you get

Every frame comes with its time, timecode, size and an image link (JPG, PNG or WebP), and every video gets a contact sheet: all its frames on one page. Other modes: a frame every N seconds, a set number of evenly spaced frames, or exact timestamps.

It reads MP4, MOV, WebM, MKV, AVI, MPEG-TS, OGV and animated GIF from direct links, S3 and other storage links, Dropbox and Google Drive.

Try it

https://apify.com/cleanscrape/video-frame-extractor

$2 per 1,000 frames, $0.005 per video, and in precise mode $0.01 per minute of 1080p video at 30 fps (smaller videos count less): the 166-frame Sintel run costs about $0.45, the overview run about $0.05. Use it on videos you own or may use.

Sintel © Blender Foundation, durian.blender.org, CC BY 3.0.

Niko, CleanScrape developer

Which AI companies are hiring engineers, and what they pay: 691 jobs from career pages in 9 seconds

CleanScrape — Sun, 25 Oct 2026 14:20:08 +0000

Job boards repost, duplicate and keep expired listings. The cleanest source for who is hiring is the company's own careers page, and most companies run it on one of a handful of job board systems: Greenhouse, Lever, Ashby, Workable, Recruitee, Personio or Teamtailor. So I built a small tool on Apify that reads those boards directly.

Here is one real run: engineering jobs at seven AI companies.

The input

You can paste a careers link, the company's website, or just its name. The tool finds the right board and says how it found it, so nothing is guessed.

{
  "companies": [
    "https://jobs.ashbyhq.com/openai",
    "https://job-boards.greenhouse.io/anthropic",
    "perplexity.ai",
    "ElevenLabs",
    "https://jobs.ashbyhq.com/cohere",
    "https://jobs.lever.co/mistral",
    "huggingface.co"
  ],
  "titleKeywords": ["engineer"]
}

The run took 9 seconds and saved 691 open jobs.

Who is hiring

OpenAI: 356 engineering jobs
Anthropic: 209
Perplexity 46, ElevenLabs 41, Cohere 34, Hugging Face 5
Mistral's board had no open jobs that day

What they pay

497 of the 691 jobs publish a yearly salary range in US dollars. Take the middle of each range, and half of them are above $355,500 a year.

Where

483 of the jobs (about 7 in 10) are in the United States. The rest are spread across 20 other countries.

Following hiring over time

Give the run a watch name and schedule it. The first run saves every job as a starting point; after that, each run saves only the jobs that are new, and lists the ones that closed in a separate table.

I tried it on 25 tech companies with 7,492 open jobs between them. Checked again five hours later (on a Saturday), the next run found 1 new job (a senior engineer at Cloudflare) and 1 that closed (a compliance role at Stripe). Small, but that is the point: a daily check returns only what changed.

Try it

https://apify.com/cleanscrape/career-site-jobs-scraper

$1 per 1,000 jobs plus $2 per 1,000 companies, less on paid Apify plans: the run above costs about $0.70. Export to Excel, CSV or Google Sheets, or call it from your own code or an AI agent through Apify's MCP server.

Niko, CleanScrape developer

Checking casino affiliate pages for 18+ notices and terms, page by page

CleanScrape — Mon, 19 Oct 2026 10:23:50 +0000

Casino brands work with hundreds of affiliate and review sites, and each of those pages tells players something about the brand's bonus. Whether a page shows an age notice, links to safer-gambling help and puts the terms next to the offer is something compliance teams check by hand, page by page. I built a small tool on Apify that does the reading: it searches Google from the country you choose, reads every page through a connection in that country, and returns one row per page.

No casino is named or shown in this video: brands, sites, offers, promo codes and links are blurred. It is about checking pages, not about any offer.

What one run found

The numbers are from one real run on Oct 18 2026 (United Kingdom, one brand): 10 pages on 8 sites, checked in under a minute and a half. Across those pages the brand's offer appeared in 9 different versions. All 5 affiliate pages showed an age notice and linked to safer-gambling help; 4 pages showed the offer with no terms next to it. Pages built with JavaScript are marked, so a notice they show later is not counted as missing. The rows report what each page shows and what is missing from it; they are not a legal verdict, since the rules differ by country and by who a page targets.

How it reads a page

The offer: from the title, the main heading, offer cards and comparison tables. In a table that lists several casinos, an offer counts only when it sits in your brand's row.
Notices: age notices (18+, 19+, 21+), safer-gambling links and helplines (BeGambleAware, GamCare, GAMSTOP, 1-800-GAMBLER, ConnexOntario and others), an ad or affiliate disclosure, and wording such as "risk-free".
Terms next to the offer: T&Cs, wagering, expiry or minimum deposit in the offer's box, or within about 500 characters of the offer's numbers in the page text.
Country: each page is read from the country being checked; in our tests one page in eight showed a different offer to visitors from another country.
Pages it cannot read (bot checks, files, JavaScript-only pages) are listed in the run report with the reason, and never charged.

The price is $8 per 1,000 saved pages; pages without an offer and pages that could not be read are free.

Try it: https://apify.com/cleanscrape/casino-bonus-checker

Maintained by CleanScrape. Questions or a reproducible issue: contact.cleanscrape@gmail.com

CleanScrape is independent of Apify and of every casino, affiliate site and network.

Which Kick streamers your competitors sponsor: 26 energy-drink sponsorships from 10,000 live streams

CleanScrape — Mon, 12 Oct 2026 17:15:31 +0000

Kick streamers advertise their sponsors in plain sight: a chat command in the stream title, a brand link in the channel's link panel, sometimes a promo code. Nobody collects that across channels, so I built a small tool on Apify that does: it scans the live streams on Kick, reads each channel's titles, links and the last month of broadcasts, and returns one row per streamer and brand.

Here is one real use: an energy-drink brand for gamers checking which streamers its competitors sponsor.

The input

{
  "brands": ["HOLY", "G FUEL", "Red Bull", "DZIK"],
  "maxLiveStreams": 10000,
  "minViewers": 1
}

The run looked through 10,000 live streams, picked the 26 streamers whose live title named one of the brands, and checked each channel in depth. It took about four minutes.

One row per streamer and brand

Each sponsorship row says where the brand shows up (live title, past broadcast titles, channel links, bio), how many of the streamer's last 30 broadcasts named it, how many hours those were, and any promo or affiliate codes. One example: rybsonlol, live with 20,518 viewers at the time, named HOLY in all 30 recent broadcasts, 276.6 hours.

The brand summary

HOLY: 16 streamers, 26,211 live viewers at the time, 357 broadcasts and 2,507 hours in the past month.
DZIK and G FUEL: 5 streamers each.
Red Bull: none in this scan.

A sponsorship here means the channel points viewers to the brand. It is what the channel shows publicly, not a confirmed contract.

Running it every day

Give the run a watch name and schedule it. Each row then carries isNew, so a new sponsorship (a streamer starting with a brand) stands out. Export any table to Excel or CSV.

Try it

https://apify.com/cleanscrape/kick-sponsor-tracker

About 110 brands are recognised out of the box (drinks, gear, game stores, VPNs, skin sites, sportsbooks and more), and any brand you type is tracked the same way. $3 per 1,000 sponsorships, no start fee: this run cost about $0.18. No login, public data only.

Niko, CleanScrape developer

Who sold the most Pokémon cards on Whatnot in a day: two checks of the top 50 sellers

CleanScrape — Mon, 12 Oct 2026 16:00:06 +0000

Whatnot ranks its top sellers per category every week, but the leaderboard does not say how much each seller sold or who is growing. I built a small tool on Apify that saves the top sellers with their lifetime items sold, followers, rating and rank, and compares every run with the one before.

Here is one real use: the top 50 Pokémon card sellers, checked twice, a day apart.

The input

{
  "categories": ["Pokémon Cards"],
  "topSellers": 50,
  "historyName": "Daily check"
}

The history name ties runs together: each run is compared with the last run under the same name. The first run, on Oct 11 at 14:51 UTC, saved the 50 sellers as a starting point. The second, 24.5 hours later, saved them again in 15 seconds.

What changed in a day

Each row of the second run has the change since the first: items sold since the last check, sales per day, new followers and the rank change.

The 48 sellers in the top 50 on both days sold 41,380 items between the checks.
outsidehits sold 2,633 items, gained 1,447 followers and moved from #13 to #10.
The biggest climber, pokeynpulls, went from #23 to #16.
Two sellers are new in the top 50.

Items sold come from Whatnot's own lifetime count on each profile, so "sold since the last check" is simply the difference between two checks. Whatnot can show the count a little late for sellers who are live, which is why a daily schedule works better than an hourly one.

Running it every day

Save the input as a task and schedule it daily. Every run adds a day of sales, followers and rank for each seller, and the "Changes since last check" view shows only the columns that moved. Export any view to Excel or CSV.

Try it

https://apify.com/cleanscrape/whatnot-seller-tracker

$3 per 1,000 sellers, no start fee: a daily check of 50 sellers costs about $0.15. No login, public data only.

Niko, CleanScrape developer

What a PSA 10 Charizard costs on Whatnot, from 73 live listings

CleanScrape — Sun, 11 Oct 2026 15:43:20 +0000

Whatnot is the live shopping marketplace where a lot of trading cards, sneakers and collectibles change hands, often during live streams. It has no public API for listings, so I built a small tool on Apify that turns a Whatnot search into a table, with the seller's track record on every row.

Here is one real use: what does a PSA 10 Charizard cost on Whatnot right now, and who is selling them?

The search

The input works like the filters on the website: a keyword, a category, and Whatnot's own card filters.

{
  "searchQueries": ["charizard"],
  "categories": ["Pokémon Cards"],
  "gradedOnly": true,
  "gradingServices": ["PSA"],
  "grades": ["10"],
  "buyFormat": "buy_it_now",
  "sort": "price_asc",
  "maxResultsPerSearch": 300
}

The run returned 73 listings in 16 seconds. Each row has the price, currency, quantity, category, the date it was listed and a link, plus the seller's items sold on Whatnot, followers, rating and number of reviews.

The prices

Price	Listings
Under $100	10
$100 to $499	36
$500 to $1,999	17
$2,000 and up	9

The cheapest was $15 and the most expensive $490,000, a 1999 first edition. One more listing asked $999,999, a placeholder price rather than a real offer, so I left it out of the table. The middle listing of all 73 was $300 ($292.50 without the placeholder).

Two things to keep in mind. These are asking prices: Whatnot shows sold prices only to logged-in users, and the tool never logs in. And "PSA 10" is a filter on how sellers tag their listings, so a few titles mention other grades. Read the titles before you compare.

Who is selling

The seller columns are what make this more than a price list. The cheapest PSA 10 Charizard from a seller rated 5.0 with over a thousand sales was $100, from a seller with 11,846 items sold. In Python, with the Apify client:

from statistics import median
from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("cleanscrape/whatnot-scraper").call(run_input={
    "searchQueries": ["charizard"],
    "categories": ["Pokémon Cards"],
    "gradedOnly": True, "gradingServices": ["PSA"], "grades": ["10"],
    "buyFormat": "buy_it_now", "sort": "price_asc", "maxResultsPerSearch": 300,
})
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
real = [r for r in rows if r["price"] < 999999]
print(len(rows), "listings, median", median(r["price"] for r in real))
trusted = [r for r in real if r["sellerRating"] == 5 and (r["sellerSoldCount"] or 0) >= 1000]
for r in trusted[:5]:
    print(r["price"], r["title"], r["sellerUsername"], r["sellerSoldCount"])

Live shows

Switch "What to find" to live and upcoming shows and the same category gives you the streams. The top 100 live Pokémon card shows had 6,041 viewers between them when I ran it, the biggest over a thousand. Each show row has the title, viewers, start time, format (52 Singles, 29 Surprise Sets and 19 Breaks in this run), tags and seller, and you can ask for the items for sale in each show too.

Only new listings

For deal alerts, sort by "Newly listed", turn on "Only listings or shows not saved before" and schedule the task. Each run looks at the newest listings and returns the ones earlier runs did not save, plus price drops on ones they did. The skipped ones are not charged.

Try it

Whatnot Scraper on Apify. You pay per listing saved, and the search above cost about $0.15. It works on all eight Whatnot marketplaces: the US, Canada, the UK, Germany, France, Belgium, the Netherlands and Australia.

What a data analyst earns in New York, Chicago and Austin, from 150 Google Jobs listings

CleanScrape — Wed, 07 Oct 2026 18:31:59 +0000

Google Jobs is the job search built into Google. It pulls listings from LinkedIn, Indeed, Glassdoor, company career sites and local job boards into one place, which makes it a good single source when you want to compare jobs across cities. It has no public API, so I built a small tool on Apify that turns a Google Jobs search into a table.

Here is one real use: I wanted to see what data analyst jobs pay in New York, Chicago and Austin.

The search

The input is a job title and a list of locations. Every title is searched in every location:

{
  "queries": ["data analyst"],
  "locations": ["New York, NY", "Chicago, IL", "Austin, TX"],
  "maxJobsPerSearch": 50
}

The run returned 150 jobs in 43 seconds. Most came via LinkedIn (34), Indeed (16), Dice (8) and Monster (8); the rest from company career sites and smaller boards.

Salaries as numbers

Google shows pay as text, such as "$124K–$177K a year" or "$57.70–$62.50 an hour". The tool splits that into salaryMin, salaryMax, salaryCurrency and salaryPeriod, so you can sort and filter without parsing anything yourself.

64 of the 150 jobs listed a salary. Taking the midpoint of each yearly range and the median per city:

City	Jobs with a yearly salary	Median of range midpoints
New York	21	$106,734
Chicago	19	$97,500
Austin	13	$110,000

These are small samples from one search on October 6, 2026, not a salary survey. Austin coming out on top surprised me; with 13 jobs it can easily move by the next run.

In Python, with the Apify client:

from statistics import median
from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("cleanscrape/google-jobs-scraper").call(run_input={
    "queries": ["data analyst"],
    "locations": ["New York, NY", "Chicago, IL", "Austin, TX"],
    "maxJobsPerSearch": 50,
})
jobs = list(client.dataset(run["defaultDatasetId"]).iterate_items())
for city in ["New York, NY", "Chicago, IL", "Austin, TX"]:
    mids = [(j["salaryMin"] + j["salaryMax"]) / 2 for j in jobs
            if j["searchLocation"] == city and j["salaryPeriod"] == "year"]
    print(city, len(mids), median(mids) if mids else None)

More than 10 jobs per search

Google Jobs shows 10 jobs per search. To get more, the tool runs close variations of the same search (posted today, this week, full time, remote, senior and so on) and keeps each job once. A search stops when the variations stop finding new jobs, so a narrow search ends early instead of paying for repeats.

Only new jobs

For job alerts, turn on "Only jobs not saved before" and schedule the task. Each run returns only jobs earlier runs did not save, and the skipped ones are not charged.

Try it

Google Jobs Scraper on Apify. You pay per job saved: $2.95 per 1,000 jobs since October 21, 2026, less on paid Apify plans. At that price the 150-job run above costs about $0.44. It works in 36 countries; outside English-speaking ones, type the job title in the local language.

What Chinese viewers say about coffee, second by second

CleanScrape — Tue, 06 Oct 2026 05:50:02 +0000

If you want to know what Chinese viewers watch and say about a topic, Bilibili is one of the best places to look. It is one of China's biggest video sites, and its viewers comment in two ways: ordinary comments under the video, and danmaku, the comments that fly across the video at the moment they were written.

I built CleanScrape's Bilibili Scraper to get both into a spreadsheet without a Bilibili account. I'm the developer, so this is a walkthrough of my own tool, not an independent review. Every number in it was checked against my runs.

One search term, 100 videos

For the example I searched for 咖啡 (coffee) and turned on top comments and danmaku. You need an Apify account to run the Actor, but no Bilibili login or cookies. The same input as JSON:

{
  "searchQueries": ["咖啡"],
  "maxVideosPerQuery": 100,
  "includeComments": true,
  "includeDanmaku": true,
  "maxDanmakuPerVideo": 500
}

The run took 81 seconds and saved 100 videos with views, likes, coins, favorites, shares and tags, plus 1,357 top comments and replies and 12,055 danmaku. Videos, comments, danmaku and creators each land in their own table, so an export holds one kind of row.

Why it doesn't get blocked

Bilibili limits how fast one visitor can read each kind of data, and it blocks an IP for minutes to an hour when that limit is crossed. I measured those limits field by field before building the Actor. It spreads the work over several proxy connections at once (nine in this run), keeps each one under the measured pace, and moves work elsewhere when Bilibili refuses a request. In this run, 5 of 604 requests were refused, all for danmaku, and every one was retried on another connection. No video is missing its danmaku.

What the danmaku show

The most-watched video in the results, about drinking six cups of coffee a day, has 5,108,457 views and 63,604 coins. A second run collected the 1,800 danmaku Bilibili's player loads for it (10,343 were sent in total), each with its time in the video.

Count them in 20-second steps and the busiest stretch after the opening is 0:40 to 1:00, with 136 danmaku. 21 of those are about bitterness, and the running joke is that life is more bitter than coffee:

再苦能有生活苦: "Can it be more bitter than life?"
咖啡没有我的命苦: "Coffee isn't as bitter as my life"
因为生活比咖啡还苦: "Because life is even more bitter than coffee"

That kind of moment-by-moment reaction is hard to get any other way. Comments tell you what people thought after watching; danmaku tell you what they felt at a given second.

Top comments

Without login Bilibili shows the top comments of a video (usually 3) and the first 20 replies under each, so this is a sample, not every comment. On the coffee video, the pinned top comment asks whether anyone has managed to quit coffee. It has 5,941 likes and 714 replies.

Export and cost

Open Storage > Dataset, choose Default (videos), Comments, Danmaku or Creators, and download CSV, Excel or JSON. From the API, the run object lists the four datasets under storageIds.datasets.

At base prices a video costs $1.49 per 1,000, a comment $0.49 per 1,000 and a danmaku $0.10 per 1,000, with no start fee and platform usage included. 100 videos with tags cost about $0.15. There are also ready-made examples for a keyword, one video's danmaku and a creator's profile and videos.

Besides search terms, you can paste video links, BV ids, creator links or UIDs, or the whole text the Bilibili app copies with Share > Copy link.

What it can't do without login: every comment of a video, a creator's full upload history (it uses their public collections and series), subtitles, and bilibili.tv, the international site.

Questions or a reproducible problem: use the Actor's Issues tab or email contact.cleanscrape@gmail.com. CleanScrape is independent of Bilibili and Apify.

A week of Nvidia news, past the 100-article limit

CleanScrape — Mon, 05 Oct 2026 16:05:12 +0000

A Google News search stops at about 100 articles. That's fine for a quick look, but not if you want everything the press wrote about a company in one week. A big name like Nvidia gets more than 100 articles in a single day, so a week-long search quietly leaves most of them out.

I built CleanScrape's Google News Scraper and RSS Feed Reader partly for this. I'm the developer, so this is a walkthrough of my own tool, not an independent review. Every number in it was checked against my run.

One search per day

The fix is simple: instead of one search for the whole week, run one search for each day and combine the results. The Actor does that when you turn on Search each day separately.

For the example I typed Nvidia, picked September 21 to 27, 2026, and set Maximum articles to 300. You don't need a Google account or an API key. You do need an Apify account to run the Actor. The same input as JSON:

{
  "queries": ["Nvidia"],
  "edition": "US:en",
  "startDate": "2026-09-21",
  "endDate": "2026-09-27",
  "splitByDay": true,
  "maxArticles": 300
}

That run returned 276 articles instead of about 100, each with the headline, the publisher, the publish time and a link.

What the week looked like

Counted by publish day (in UTC), Friday was the busiest, with 64 articles. By publisher, Yahoo Finance alone ran 73 of the 276.

One honest caveat: each day's search also returned Google's maximum of 100 entries, so the run report marks those days as possibly incomplete. More articles may exist. Splitting by day gets you much closer to the full picture, but for a name this busy it isn't a guarantee of every article.

Publisher links

Google News links point back to Google, not to the article. Where it can, the Actor turns them into the publisher's own link, and a link status column tells you which ones it found. In this run, 85 links were turned into the publisher's page and 191 kept the Google News link, because Google limits how many links can be looked up in a row. The Google link still opens the article; it just takes one extra hop.

Follow the topic after the first run

To keep up with a topic without getting the same headlines again, turn on Only articles not saved before and give it a Watch name. Later runs with the same watch name return only articles that weren't delivered before. Save the input as a task and schedule it daily or weekly.

{
  "queries": ["Nvidia"],
  "timeRange": "day",
  "newOnly": true,
  "watchName": "Nvidia news"
}

I wrote more about that setup in a small news watchlist that doesn't send the same headlines every day.

Export and cost

Export the Articles view as CSV or Excel. The run report shows what every search returned, including which days hit Google's limit.

The price is $0.95 per 1,000 saved articles, with no start fee and platform usage included. The 276 articles above cost about $0.26 at base prices. Check the Pricing tab for your account's rate, and set a spending limit before larger runs.

Questions or a reproducible problem: use the Actor's Issues tab or email contact.cleanscrape@gmail.com. CleanScrape is independent of Google and Apify, and Nvidia is used here as a public-news example, not an endorsement.

pytrends keeps failing with 429 errors? Here's a one-line fix

CleanScrape — Sun, 04 Oct 2026 14:04:04 +0000

If you have a Python script that pulls Google Trends data with pytrends, there's a good chance it stopped working. pytrends was archived in April 2025, and scripts using its unofficial endpoints can encounter 429 Too Many Requests.

I built trendreq for scripts using the common TrendReq methods. It returns pandas DataFrames, but the requests run on my Google Trends Actor on Apify, where the proxies and retries are handled for you. I'm the developer, so this is a walkthrough of my own tool, not an independent review. The output below is from a real run.

Why pytrends gets 429s

Google offers an official Trends API alpha, with access by application. pytrends does not use that API. pytrends talks to the same internal endpoints the website uses: one request for a short-lived token, then one per chart. Every request comes from your IP address, and Google limits how often one address can do that. Limits vary by traffic and connection; neither this wrapper nor a proxy can guarantee that requests will never be rate-limited.

The usual advice is sleeping between calls, rotating proxies yourself, or patching the library. That works for a bit, then it doesn't.

The change

# from pytrends.request import TrendReq
from trendreq import TrendReq

Install it and set your Apify token (a free Apify account includes $5 of usage a month):

pip install trendreq
export APIFY_TOKEN=your_token

The method calls stay the same. One optional line, fetch, gets several data types in one run:

pytrends = TrendReq(hl="en-US", tz=360)
pytrends.build_payload(["iced coffee", "cold brew"], timeframe="today 12-m", geo="US")

pytrends.fetch("interest_over_time", "related_queries", "interest_by_region")
over_time = pytrends.interest_over_time()
by_state = pytrends.interest_by_region(resolution="REGION", inc_geo_code=True)
related = pytrends.related_queries()

What came back

interest_over_time(), the last few weeks:

            iced coffee  cold brew  isPartial
date
2026-09-06           57         63      False
2026-09-13           59         68      False
2026-09-20           47         51      False
2026-09-27           42         49      False
2026-10-04           32         33       True

interest_by_region(), sorted by cold brew. Oregon and Washington lean hardest toward cold brew:

           geoCode  iced coffee  cold brew
geoName
Oregon       US-OR           31         69
Washington   US-WA           35         65
Kansas       US-KS           35         65
Montana      US-MT           36         64
California   US-CA           37         63

And the rising related queries for cold brew, which is where product ideas tend to show up:

                                 query  value
0      toasted coconut cream cold brew  19700
1           smores cold brew starbucks  19700
4    how to make cold brew concentrate   2050

The whole thing took one run of about 9 seconds.

One run instead of three

Each method call starts one Apify run, and each run has a small start fee. If you want several data types for the same keywords, ask for them together:

pytrends.fetch("interest_over_time", "related_queries", "interest_by_region")

After that, the three methods read from the same run. The example above did exactly that: 102 rows, about $0.36 at the base price.

What's different from pytrends

Most of it works the same, but not all of it:

Related topics come back empty. Google currently returns an empty list for automated requests (pytrends gets the same), so related_topics() gives None with a warning. Related queries work.
City results get coordinates. interest_by_region(resolution="CITY") adds latitude and longitude, because the US has several Springfields and they shouldn't merge.
US metro areas work with resolution="DMA".
gprop isn't supported yet, so YouTube, News and Shopping search raise a clear error instead of returning web data.
suggestions() and categories() aren't there yet. They raise NotImplementedError with a short explanation.
Trending searches (trending_searches, today_searches) work, and trending_now("US") adds approximate traffic and related news headlines.

The exception names are the same, so except ResponseError: blocks keep working.

Cost

You pay Apify for the Actor's rows: $0.05 per run plus $0.003 per row at the base price, less on paid plans. Two keywords over 12 months is about $0.37. Today's trending searches for one country is about $0.08. The free monthly credit covers a few dozen typical calls.

Try it

Code, docs and offline tests: github.com/CleanScrape/trendreq
The Actor it runs on: apify.com/cleanscrape/google-trends-scraper

If something you rely on in pytrends doesn't work, open an issue on GitHub. I'd rather hear about it and fix it.