DEV Community: SteadyScrape

How to get Google keyword search volume in bulk without a Google Ads account (Python)

SteadyScrape — Mon, 02 Nov 2026 00:54:28 +0000

Google Keyword Planner only shows exact search volumes if you run an active ad campaign. Without one you get ranges like "1K–10K", which is useless for deciding which article to write first.

This post shows how to get exact monthly search volume, CPC, competition and keyword difficulty for a list of keywords from Python, then save it as a CSV you can sort. No Google Ads account, no OAuth app, no developer token.

What you need

Python 3.9+
An Apify account (the free plan includes $5/month of usage) and your API token from Console → Settings → API & Integrations
pip install apify-client (this post uses v3)

The data comes from the Keyword Search Volume Actor on Apify, which wraps a Google Ads–sourced keyword data provider and caches results for 30 days.

The script

import csv, os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

keywords = [
    "crm software", "crm for small business", "free crm", "best crm",
    "crm for startups", "simple crm", "crm app", "real estate crm",
    "crm for nonprofits", "what is a crm",
]

run = client.actor("steady-scrape/keyword-search-volume").call(logger=None, run_input={
    "keywords": keywords,
    "location": "United States",
    "language": "English",
    "includeMonthlyTrend": True,
})
rows = list(client.dataset(run.default_dataset_id).iterate_items())

def trend(monthly):
    """Last 3 months vs the 3 before, as a % change."""
    v = [m["searchVolume"] or 0 for m in sorted(monthly or [], key=lambda m: m["month"])]
    if len(v) < 6 or sum(v[-6:-3]) == 0:
        return None
    return round(100 * (sum(v[-3:]) / sum(v[-6:-3]) - 1))

with open("keywords.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["keyword", "volume", "cpc_usd", "competition", "difficulty", "intent", "trend_3m_pct"])
    for r in sorted(rows, key=lambda r: r.get("searchVolume") or 0, reverse=True):
        if r.get("status") != "ok":
            continue
        w.writerow([r["keyword"], r["searchVolume"], r.get("cpc"), r.get("competition"),
                    r.get("keywordDifficulty"), r.get("searchIntent"), trend(r.get("monthlySearches"))])
        print(f'{r["keyword"]:<24} {r["searchVolume"]:>8,}  ${r.get("cpc") or 0:>6.2f}  KD {r.get("keywordDifficulty")}')

Run it:

export APIFY_TOKEN=...   # never commit this
python keyword_volume.py

trend_3m_pct compares the last three months with the three before. It's a quick way to spot keywords that are growing or fading without opening a chart.

Real output (US / English, data month 2026-09)

keyword	volume	CPC	competition	KD	intent	3-month trend
crm software	201,000	$37.13	LOW	59	commercial	+5%
what is a crm	22,200	$15.63	LOW	70	informational	+7%
free crm	5,400	$67.26	MEDIUM	38	commercial	−28%
crm for small business	4,400	$51.28	LOW	31	commercial	+59%
real estate crm	4,400	$37.34	MEDIUM	36	commercial	−4%
best crm	2,400	$63.56	MEDIUM	35	commercial	+9%
crm for nonprofits	2,400	$58.70	MEDIUM	35	commercial	−4%
crm app	1,300	$44.41	LOW	58	commercial	−9%
simple crm	590	$60.53	MEDIUM	33	commercial	−12%
crm for startups	390	$95.77	LOW	10	commercial	−8%

Two things jump out that a "1K–10K" range would hide:

"crm for small business" has 4,400 searches, a difficulty of 31 and is up 59% over three months. That's a much better first article than "crm software" (KD 59).
"crm for startups" is tiny (390/month) but has the highest CPC ($95.77) and KD 10. Advertisers pay a lot for that click, so it's worth a page even at low volume.

competition is the Google Ads competition level (how many advertisers bid). It is not SEO difficulty. Use keywordDifficulty (0–100) for organic ranking.

What it cost

I ran exactly this 10-keyword list on Nov 1, 2026. At the free-plan list price:

1 live lookup for the run: $0.12 (skipped when every keyword is already in the 30-day cache)
10 keyword results × $0.003 = $0.03
Actor start: $0.00005

Total: about $0.15 for 10 keywords, in 3.7 seconds. The fixed lookup fee is per run, so batch your keywords: 1,000 keywords in one run is about $3.12, not 100 × $0.15. Paid Apify plans get lower per-result prices.

Tips

Pass "location": "United Kingdom" (or another country) and "language" to get local volumes. The cache is per location and language.
Keywords are normalized and deduplicated by default, so "CRM Software" and "crm software " count once.
Keep includeMonthlyTrend on. The 12-month series is what makes seasonality and the trend column possible.
Rows with status other than ok (no data, invalid keyword) aren't billed as results. The script skips them.

Disclosure: SteadyScrape publishes the Actor used in this post. Numbers above are from a real run on Nov 1, 2026; volumes change monthly.

How to Get a List of Websites Using Shopify, HubSpot or Any Other Technology

SteadyScrape — Mon, 02 Nov 2026 00:38:02 +0000

If you sell a Shopify app, a HubSpot alternative or a Stripe integration, your best prospects are easy to describe: companies that already run a particular tool. The data that tells you who they are is called technographics. Most tools sell it as a subscription; this post shows a pay-per-result way to get the same kind of list in Python.

What "websites using X" data actually is

Crawlers fetch home pages and fingerprint the scripts, headers and HTML they find (the same idea as Wappalyzer). Each domain gets a list of detected technologies plus metadata: country, language, title and a "last seen" date. Two things follow from that:

Only front-end-visible tools show up. A site's Shopify theme, Klaviyo script or Intercom widget is detectable; its accounting software isn't.
It's a database, not a live crawl, so always check how recently a site was seen.

Example: US Shopify stores that also use Klaviyo

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

Find Websites Using Any Technology: Lead Lists (steady-scrape/tech-stack-lead-lists) is an Apify Actor that queries a technographics database and returns one row per company website.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("steady-scrape/tech-stack-lead-lists").call(run_input={
    "technologies": ["Shopify", "Klaviyo"],   # AND: both must be present
    "countries": ["US"],
    "lastSeenWithinDays": 180,
    "maxResults": 50,
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["domain"], row["domainRank"], row["paymentProcessors"], row["lastSeen"])

Here's what that exact input returned in our test run on November 1, 2026:

27,537 domains in the database matched (US, Shopify and Klaviyo, seen in the last 180 days). The run returned the top 50 by domain rank in about 4 seconds.
The top results were large brand stores such as shop.hulu.com and society6.com.
Median data age: 12 days. 48 of the 50 had been seen within the last 30 days; the oldest was 170 days.

Each row includes the matched technologies, the rest of the detected stack in ready-made columns (e-commerce platform, CMS, payments, marketing automation, CRM, analytics, live chat, ad pixels, reviews, Shopify apps), company social profiles, lastSeen and dataAgeDays.

Filters that make lists useful

Technology category instead of a product: live_chat, crm, payment_processors.
Country and language, for territory-based sales teams.
Domain rank range, to skip the giants and the hobby sites.
Freshness (lastSeenWithinDays), to drop stale records.
Monitor mode (trackNewDomains), which flags domains that are new since your last run. It's useful for "who just installed X" alerts on a schedule.

Limits worth knowing

Combining technologies is AND only. "Uses Shopify but not Klaviyo" needs two runs and a diff.
Many smaller sites have no known country, so a country filter returns only the ones that do.
LinkedIn URLs are rare. Plan to enrich company size elsewhere.
Contact data is company-level only. Generic role mailboxes such as sales@ and published phone numbers are opt-in. There are no named individuals.

What it costs

Event	Price
Run start (only if at least one domain is returned)	$0.015
Domain delivered	$0.005 ($5 per 1,000)

Apify also adds a $0.00005 start fee. Runs that return nothing, invalid input and provider errors aren't charged the run start or per-domain fees.

Examples:

The 50-domain test above: $0.015 + 50 × $0.005 = $0.265.
A 100-domain test list: $0.515.
A 1,000-domain list: $5.015.

Which to use

If you need a handful of sites, running Wappalyzer or a browser extension on each one is fine. For a list of thousands of companies running a tool, filtered by country and freshness, querying a technographics database is far faster, and paying per domain beats a monthly subscription when you only need a list now and then. Either way, check the lastSeen date before you pitch, and keep outreach to company-level contacts.

Try it on the Apify Store: the prefilled example input runs the same US Shopify + Klaviyo search shown above.

Finding Local Businesses With No Website (or an Unclaimed Google Profile) for Lead Lists

SteadyScrape — Sun, 01 Nov 2026 14:37:34 +0000

If you sell websites, local SEO or review management to small businesses, your best leads are easy to describe:

a plumber with no website, so customers find them only through their Maps listing,
a dentist whose Google Business Profile is unclaimed, so anyone can suggest edits,
a restaurant rated below 4 stars, or a salon with fewer than 10 reviews.

Describing them is easy. Finding them is harder, because most Maps tools return every business in an area, and you pay for all of it before throwing most away. This post shows how to build these segments in Python, where the official API falls short, and how to keep the list clean of personal data.

Option 1: the Google Places API

The Places API (New) has a Text Search endpoint that returns businesses for a query like "plumber in Austin", and you choose the fields with a field mask:

import os, requests

def places(query, page_token=None):
    r = requests.post(
        "https://places.googleapis.com/v1/places:searchText",
        headers={
            "X-Goog-Api-Key": os.environ["GOOGLE_MAPS_API_KEY"],
            "X-Goog-FieldMask": "places.displayName,places.websiteUri,places.rating,"
                                "places.userRatingCount,places.googleMapsUri,nextPageToken",
        },
        json={"textQuery": query, **({"pageToken": page_token} if page_token else {})},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()

data = places("plumber in Austin, TX")
no_site = [p for p in data.get("places", []) if not p.get("websiteUri")]
print(f"{len(no_site)} of {len(data.get('places', []))} have no website")

It's official and live, and good for small jobs. Its limits for lead lists:

A search returns up to 60 results across pages. Covering a whole metro means tiling the map into many small searches and deduping.
You filter after you fetch. If 10% of plumbers have no website, you pay for the other 90% too. Website and rating fields are on the more expensive request tiers.
No claim status. The API doesn't tell you whether a profile is claimed, which is the segment local SEO agencies care about most.
"Website" is sometimes a Facebook page. You'll want to treat social-only sites as "no website".

SOCIAL = ("facebook.com", "instagram.com", "yelp.com", "linktr.ee")

def has_real_website(url):
    return bool(url) and not any(s in url.lower() for s in SOCIAL)

Option 2: filter on the database side

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

Google Maps Leads: No Website, Unclaimed Profiles (steady-scrape/google-maps-business-segments) is an Apify Actor that runs the segment filters on the data provider's side. It queries a regularly refreshed, licensed database of Google Maps listings, so you only get (and only pay for) the businesses that match. You can filter by no website (or social-only), unclaimed, rating below X, fewer than Y reviews, or first seen in the last N days, and combine them.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("steady-scrape/google-maps-business-segments").call(run_input={
    "categories": ["plumber"],
    "coordinates": "30.2672,-97.7431",   # Austin, TX
    "radiusKm": 25,
    "website": "noWebsite",
    "claimStatus": "unclaimed",
    "maxResults": 200,
})
leads = list(client.dataset(run["defaultDatasetId"]).iterate_items())
for b in sorted(leads, key=lambda b: -(b.get("reviewsCount") or 0)):
    print(b["segmentTags"], b.get("rating"), b.get("reviewsCount"), b["googleMapsUrl"])

Here's what that input (with maxResults: 50) returned in our test run on November 1, 2026. The database query took about 1.2 seconds and the whole run about 4:

20 businesses matched in the database, all with no website and an unclaimed profile (18 plumbers and 2 businesses in related categories).
Median review count: 3.5. 17 of the 20 had fewer than 10 reviews, and 4 had none at all, which makes for a good "get more reviews" pitch.
Of the 16 with a rating, 6 were below 4 stars.
Median record age: 38 days since the database last refreshed the listing (49 days at most).

Each record has the business name and categories, the full address with coordinates, placeId and a Google Maps link, hasWebsite and websiteIsSocialProfile, isClaimed, rating with a 1–5★ distribution, review count, opening hours, status, and segmentTags such as no-website, unclaimed, no-reviews or rating-below-4.

Using coordinates, not city names

Use coordinates plus a radius rather than a city name. Addresses use the suburb's own name, so "Austin" misses Round Rock and Pflugerville. Postal codes and whole countries work too.

Doing it responsibly

Lead lists from Maps can drift into personal data, especially for sole traders. Some rules we built in, and recommend whatever tool you use:

No email addresses. The Actor drops scraped contact lists and masks emails found in descriptions. Business emails scraped from other sites are what gets cold outreach flagged as spam, or worse.
No owner names, reviewer data or photos.
Phone numbers are optional. It's the number the business published on its own Google profile. Set dropPhones: true if you target one-person businesses in the EU, UK or Quebec, where that number can count as personal data.
Check before you pitch. Coverage isn't 100% of Google Maps, and a business can have a website Google doesn't show. Open the Maps link first.
"First seen" isn't an opening date. It's when the provider first found the listing. For likely new businesses, combine first seen in the last 60–90 days with fewer than 5 reviews.

Data is a refreshed database, not a live Maps load. In our tests most active listings had been refreshed within the previous one to two months.

What it costs

Event	Price
Run start (only if a business is returned)	$0.015
Business, no segment filter	$0.002 ($2 per 1,000)
Business matching a segment filter (no website, unclaimed, rating below, few reviews, first seen)	$0.005 ($5 per 1,000)

Those are Apify Free plan prices. Paid Apify plans get 10-25% off the per-business prices automatically. Apify adds a $0.00005 start fee. Runs that return nothing, invalid input and provider errors are free.

Examples:

The 20-lead test above: $0.015 + 20 × $0.005 = $0.115 (that's what the run was billed).
100 plumbers with no website: $0.515.
1,000 dentists in a metro with no segment filter: $2.015.
A monthly refresh of no-website leads in 5 cities at ~100 each: about $2.58 a month.

Because the filter runs before delivery, you pay only for the matches, however rare they are in the area.

Which to use

For a live check of a few dozen businesses, the Places API plus the has_real_website() filter above is fine. For area-wide segment lists, and especially for claim status, filtering on the database side is cheaper and faster. Either way, keep the list to business data, and look at each listing before you send a pitch.

Try it on the Apify Store: the prefilled example input runs the same no-website, unclaimed plumber search shown above.

Give Your AI Agent Real SEO Data: Keyword Volume, Live SERPs, Backlinks and Trends Over MCP

SteadyScrape — Sun, 01 Nov 2026 00:44:24 +0000

Ask an AI assistant "where does my site rank for best crm for startups?" and it will either guess or refuse. LLMs don't have search volumes, today's Google results or backlink counts. They make them up. The fix is to give the agent tools that fetch real data, and the Model Context Protocol (MCP) is now the standard way to plug those tools into Claude, Cursor, VS Code and agent frameworks.

This post covers:

what the SEO MCP server does (8 tools),
connecting it to Claude Desktop / Cursor in one config block,
calling it from Python with the official MCP SDK,
what a typical session costs.

The 8 tools

Tool	What the agent gets
`keyword_metrics`	Google search volume, CPC, competition, keyword difficulty, intent and 12-month trend for up to 100 keywords
`google_serp`	Live Google results (top 10-100) for any country and device, your domain's position, AI Overview text and sources, People Also Ask
`domain_overview`	Organic traffic estimate, ranked-keyword count, backlinks and referring domains for up to 100 domains
`ranked_keywords`	Up to 100 keywords a domain ranks for, with position, volume and URL
`backlinks`	Up to 100 backlinks or referring domains for a domain or page
`google_trends`	Interest over time and by region for up to 5 terms
`youtube_search`	Ranked YouTube results with views, channel and date
`youtube_video`	Likes, comments, subscriber count, tags, plus the transcript if you ask

All tools are read-only. There are no logins, cookies or proxies on your side: the data comes from licensed SEO data feeds.

Connect it to Claude Desktop or Cursor

{
  "mcpServers": {
    "seo-data": {
      "url": "https://steady-scrape-1--seo-data-mcp-server.apify.actor/mcp",
      "headers": { "Authorization": "Bearer YOUR_APIFY_TOKEN" }
    }
  }
}

That's it. The server runs in Apify Standby mode: it starts on the first request (a few seconds of cold start in our tests) and stays warm while you work.

Then just ask:

"Where does example.com rank for project management software in the UK on mobile?"
"Compare organic traffic and referring domains for these 5 competitors."
"Is interest in pickleball paddles growing? Show 5-year Google Trends vs tennis racket."

From Python

import asyncio, os, httpx2
from mcp import ClientSession
from mcp.client.streamable_http import streamable_http_client

URL = "https://steady-scrape-1--seo-data-mcp-server.apify.actor/mcp"

async def main():
    headers = {"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"}
    async with httpx2.AsyncClient(headers=headers, timeout=60) as hc:
        async with streamable_http_client(URL, http_client=hc) as (r, w, *_):
            async with ClientSession(r, w) as s:
                await s.initialize()
                res = await s.call_tool("google_serp", {"keyword": "best crm for startups",
                                                        "target_domain": "hubspot.com"})
                print(res.structured_content["targetPosition"])

asyncio.run(main())

In our tests most tools answered in 1-5 seconds. A top-100 SERP takes about 15 s and Google Trends about 14 s, because both are fetched live.

What it costs

You pay per successful call. Errors, invalid arguments and empty results are free, and there's no compute charge on top.

Session	Cost (Free plan)
Rank check: 5 keywords, top 10 each	$0.035
Keyword research: 100 keywords	$0.395
Competitor audit: 10 domains + 2 ranked-keyword lists + 1 backlink list	$0.340
YouTube research: 2 searches + 5 videos, 3 with transcripts	$0.161

Per tool: $0.006 per 10 SERP results, $0.039 per 10 keywords, $0.111 per 10 domains, $0.071 per ranked-keywords call, $0.082 per backlinks call, $0.035 per Trends call, $0.006 per 20 YouTube results, $0.018 per video (+$0.018 for a transcript), and $0.005 once per session. Paid Apify plans get 5-15% off the tool prices.

Agents can loop, so the server has guardrails. It refuses calls that would push you past the maximum charge you set for a session, before fetching anything. It also limits a session to 60 calls a minute.

Try it

Disclosure: we (SteadyScrape) built this server and get paid when it's used.

SEO MCP Server: Keyword, SERP & Backlink Data (steady-scrape/seo-data-mcp-server) is on the Apify Store. Paste the config above, ask your agent an SEO question, and check the billing field in each response to see exactly what was charged.

No MCP client yet? Run the Actor normally with input like {"calls": [{"tool": "google_serp", "arguments": {"keyword": "seo mcp server"}}]} and each call becomes one dataset row. Our post-publish test of exactly that took one SERP page plus the session fee, about $0.011.

Building a Keyword Rank Tracker That Also Flags AI Overviews (Python)

SteadyScrape — Sat, 31 Oct 2026 14:49:21 +0000

Rank tracking sounds simple: search a keyword, find your site, write down the position. The useful part is everything after that. Did it move? Did a different page of yours start ranking (cannibalization)? Did Google add an AI Overview that pushed your "position 2" below the fold?

This post walks through the design of a small rank tracker: what to store per keyword, how to diff runs, and which SERP features to record. The code works with any source of SERP data. At the end there's a hosted option.

Where the SERP data comes from

Don't scrape Google results yourself. It's against Google's terms, gets blocked quickly, and results vary by IP location. In practice everyone uses a SERP API: you send keyword + location + language + device and get back parsed results. Whatever provider you choose, make sure it returns:

organic results with position and URL,
the absolute position (counting ads, AI Overview, Local Pack and other blocks),
a list of SERP features on the page.

The snapshot

Store one record per keyword, keyed by everything that changes the SERP:

def key(keyword, location, language, device):
    return f"{keyword.lower()}|{location}|{language}|{device}"

def snapshot_row(keyword, serp, target_domain):
    mine = [r for r in serp["organic"] if r["domain"] == target_domain
            or r["domain"].endswith("." + target_domain)]
    best = mine[0] if mine else None
    return {
        "keyword": keyword,
        "position": best["position"] if best else None,
        "url": best["url"] if best else None,
        "otherUrls": [r["url"] for r in mine[1:]],
        "features": sorted(serp.get("features", [])),
        "top3": [r["domain"] for r in serp["organic"][:3]],
    }

Count subdomains as you (make it a setting), and keep the URL, not just the position: most rank-tracking insights come from URL changes.

The diff

def compare(prev, cur):
    if prev is None:
        return {"changeStatus": "first_check", "positionChange": None}
    p, c = prev["position"], cur["position"]
    if p is None and c is None:
        status, change = "not_ranked", None
    elif p is None:
        status, change = "new", None
    elif c is None:
        status, change = "lost", None
    else:
        change = p - c                       # positive = moved up
        status = "up" if change > 0 else "down" if change < 0 else "same"
    return {
        "changeStatus": status,
        "positionChange": change,
        "rankingUrlChanged": bool(p and c and prev["url"] != cur["url"]),
        "newFeatures": sorted(set(cur["features"]) - set(prev["features"])),
    }

rankingUrlChanged is your cannibalization alert: two of your pages are competing for one keyword, and Google is swapping between them. newFeatures tells you when, say, ai_overview or local_pack appears on a SERP where it wasn't before. Also, mark a run not_comparable if the depth changed (top 10 vs top 100) instead of reporting false "lost" rankings.

Why SERP features matter now

An organic position doesn't tell you how visible the result is. Position 2 under an AI Overview, a Local Pack and four ads sits far lower on the page than position 2 on a plain SERP. Record the absolute position too, and flag the features that matter to you: AI Overview, People Also Ask, featured snippet, Local Pack, shopping, video and forums.

The hosted version

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

Google Rank Tracker API: Daily Keyword Positions (steady-scrape/google-rank-tracker) is an Apify Actor that implements this design on top of a licensed SERP data API. You give it a target (a domain, or an exact URL if only that page should count), up to 2,000 keywords, and a location, language and device. Each keyword comes back with its organic and absolute position, the ranking URL and any other URLs of yours on the page, the SERP features (and whether your site appears inside them), the top 3 competing domains, positions for up to 10 competitors you name, and, if you reuse the same trackerId, the change since the last run.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("steady-scrape/google-rank-tracker").call(run_input={
    "target": "apify.com",
    "keywords": ["scrape amazon prices", "instagram scraper", "web scraping api"],
    "location": "United States",
    "language": "English",
    "device": "desktop",
    "depth": "10",
    "queueMode": "standard",
    "trackerId": "apify-us-desktop",
})
for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(r["keyword"], r["position"], r["changeStatus"], r["serpFeatures"], r["aiOverviewCited"])

That's the Actor's example input. Here's what our test run of it returned on October 31, 2026:

1 of 3 keywords ranked in the top 10, at position 7 (absolute position 8, because an AI Overview sat above the organic results).
All 3 SERPs had an AI Overview. The site was cited in 2 of those 3 AI Overviews, including one for a keyword where it wasn't in the organic top 10 at all. That's exactly the kind of thing a position-only tracker misses.
The ranking URL was an individual page on apify.com, not the homepage. That's how a domain target works: any page on the site counts. Use a full URL as the target to track one page only.
The run took 43 seconds in total. The SERP checks waited 11 to 27 seconds in the standard queue.

Run it again with the same trackerId, or put it on an Apify schedule, and every row gets positionChange, changeStatus (up, down, same, new, lost and so on), previousRankingUrl and rankingUrlChanged. Set loadAiOverview: true to also load AI Overviews that Google shows asynchronously and check whether they cite you.

A few practical notes:

Depth costs more because Google serves 10 results per page. A top-100 check loads 10 pages per keyword. Most teams track the top 10 or top 30 and only go deeper for a few keywords.
Queue modes: Standard is the cheapest. In our tests top-10 checks came back in under a minute, top 30 in about 4 minutes and top 100 in about 15 minutes. Priority is faster for a small surcharge, and Live returns in seconds.

What it costs

You pay per keyword that was checked successfully. Errors, invalid keywords and timeouts are free.

Event	Price	Apify Bronze / Silver / Gold plans
Keyword checked, top 10	$0.003 ($3 per 1,000)	$0.0027 / $0.0024 / $0.00225
Keyword checked, top 30	$0.007	$0.0063 / $0.0056 / $0.00525
Keyword checked, top 100	$0.025	$0.0225 / $0.02 / $0.01875
Priority queue surcharge	$0.0015 per 10 results checked	same
Live mode surcharge	$0.003 per 10 results checked	same
AI Overview citations (only when one was loaded)	$0.002 per keyword	same
Run start (only after the first successful check)	$0.015	same

Apify also adds its own $0.00005 start fee per run.

Examples at list price:

The 3-keyword test above: $0.015 + 3 Ã— $0.003 = $0.024.
100 keywords every day at top 10: $0.315 per run, about $9.45 a month.
200 keywords weekly at top 10: about 860 checks a month Ã— $0.003 + 4.3 run starts Ã— $0.015 â‰ˆ $2.64 a month.
The same 200 keywords at top 100: 860 Ã— $0.025 â‰ˆ $21.50 a month plus run starts, which is why top 10 or top 30 is the usual choice.

You can set a maximum cost per run. The Actor works out how many keywords fit before it starts and marks the rest as skipped, free of charge.

Wrapping up

Whatever data source you use, the design is what makes a tracker useful: key by keyword + location + language + device, store the URL as well as the position, diff run over run, and record SERP features. AI Overviews are changing what "ranking #2" is worth, and a tracker that doesn't record them will report rankings that look better than they are.

Get Notified When You Gain or Lose a Backlink: A Python Monitor Without an SEO Subscription

SteadyScrape — Sat, 31 Oct 2026 00:40:12 +0000

Losing a good backlink is silent. Nobody emails you when a partner redesigns their site and drops your link, or when a competitor gets a mention in a big publication. The large SEO suites will alert you, but they're priced as monthly subscriptions, and often you only need the alert for a handful of domains.

This post covers:

where backlink data comes from (and why scraping an SEO tool's pages is a bad idea),
how to turn "all backlinks" into "what changed since last time",
what it costs, with numbers from a real run.

Where backlink data comes from

You can't check backlinks by visiting your own site. A backlink lives on someone else's page, so you need a crawler that has seen a large part of the web and kept the link graph. Building that yourself isn't realistic, and scraping a vendor's web UI breaks often and goes against their terms. What's left is to license an index through an API.

We use DataForSEO's Backlinks API. For each link it returns the linking page, anchor text, dofollow/nofollow, first and last seen dates, and 0–100 ranks for the linking domain and page.

The diff is the product

A full list of a big site's backlinks is interesting once. What you want every week is:

new: links whose first_seen is after your last check,
lost: links the crawler found missing since your last check.

The core of a DIY version is small: keep a hash of every change you've already reported, and only report what's new.

import hashlib, json, pathlib

STATE = pathlib.Path("backlink_state.json")

def change_key(row):
    # one key per (change type, linking page, target page)
    raw = f'{row["change"]}|{row["sourceUrl"]}|{row["targetUrl"]}'
    return hashlib.sha1(raw.encode()).hexdigest()

def load_seen():
    return set(json.loads(STATE.read_text())) if STATE.exists() else set()

def only_unseen(changes):
    # pure: returns changes not reported yet, writes nothing
    seen = load_seen()
    return [c for c in changes if change_key(c) not in seen]

def mark_reported(changes):
    # call only after delivery succeeded
    seen = load_seen() | {change_key(c) for c in changes}
    STATE.write_text(json.dumps(sorted(seen)))

# each run:
# changes = fetch_changes(since=last_run - timedelta(hours=48))
# fresh = only_unseen(changes)
# if fresh:
#     send_alert(fresh)      # raises on failure -> nothing is marked
#     mark_reported(fresh)

Updated Nov 2, 2026: the state is now written only after an alert is delivered, so a failed send no longer loses changes. Thanks to the reader who caught this in the comments.

Three gotchas we hit while building this:

Index lag. A link first seen yesterday may only show up in the index today. If you query strictly "since my last run", you miss it for good. Query with an overlap window instead (we use 48 hours) and drop changes you already reported, using the hashes above. That way each change shows up once, or occasionally twice if a send fails and is retried.
Big domains are noisy. Large sites gain and lose links all the time, many of them sitewide or scraper links. For big sites, monitor referring domains instead of individual links, and sort the changes by the linking domain's rank.
Filters don't behave like SQL. On the referring-domains endpoint, an OR filter that includes lost_date silently drops every row where lost_date is null, which means all new domains. We ended up running "new" and "lost" as two separate queries.

Where to keep the state: one small JSON record per (monitor, domain) in a key-value store, holding the last check time and recent change hashes. We prune hashes after 60 days so the record stays small.

What the data looks like

Here's a real run: the 100 strongest backlinks each for backlinko.com and plausible.io, sorted by the linking domain's rank.

Domain	Domain rank	Backlinks (index total)	Referring domains
backlinko.com	70	886,292	59,047
plausible.io	74	680,015	17,423

A few things in that output are worth knowing before you build on it:

The top 100 links by rank came from only 13 linking domains for backlinko.com and 4 for plausible.io. Strong sites often link many times from many pages. If you want breadth, deduplicate to one link per domain.
Only 34 of the 200 links were dofollow. High-rank sites like GitHub mark most outbound links nofollow, so filter on dofollow if that's what you care about.
Field fill rates over those 200 rows: linking URL, anchor, dofollow, first/last seen, domain rank and country were 100%; page language 99%; link attributes 90%; target status code 87%; the surrounding link text 69%.

The tool we use

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

We packaged all of the above as an Apify Actor: Backlink Checker API: New and Lost Link Monitor (steady-scrape/backlink-explorer). It has four modes: backlinks, referring_domains, summary (up to 10,000 domains in one run) and monitor.

For the weekly alert, schedule it with an input like this:

{ "domains": ["mysite.com", "competitor.com"], "mode": "monitor", "monitorId": "weekly", "firstRunLookbackDays": 7 }

Each run returns only the links gained or lost since the previous run, each with anchor, dofollow and the linking domain's rank. A domain with no changes skips the per-link lookup.

Calling it from Python:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("steady-scrape/backlink-explorer").call(run_input={
    "domains": ["competitor.com"], "mode": "backlinks",
    "dofollowOnly": True, "minSourceRank": 40, "onePerDomain": True, "maxRowsPerDomain": 200,
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["rowType"] == "backlink":
        print(row["sourceDomainRank"], row["sourceUrl"], "->", row["anchor"])

What it costs

Pricing is pay per event, with no subscription. List prices (Apify Free plan):

Event	Price
Run start (only if at least one domain returns data)	$0.03
Domain looked up in backlinks or referring-domains mode (summary row included)	$0.05
Backlink row	$0.0008 ($0.80 per 1,000)
Referring-domain row	$0.001
Domain in summary mode	$0.004
Domain checked in monitor mode	$0.05
New or lost link reported	$0.002

Apify also charges its standard $0.00005 start event. Row prices drop on paid Apify plans: a backlink row is $0.00072 on Starter, $0.00064 on Scale and $0.0006 on Business. Invalid inputs and domains with no data are free.

The run above (2 domains × 100 backlinks) works out to $0.03 + 2 × $0.05 + 200 × $0.0008 = $0.29 at list price, and it took about 6 seconds. Other examples at list price:

1 domain, 100 strongest backlinks: about $0.16
Summary metrics for 100 domains: about $0.43
Weekly monitor check of 10 domains: about $0.53, plus $0.002 per new or lost link

Most of the cost is the per-domain fee, so it pays to pull many links per domain, or to monitor only the domains you actually care about.

Caveats

Ranks are the index's own 0–100 link-graph scores. They aren't the authority score from any other SEO tool, though they tend to move in the same direction.
Every backlink index sees a different slice of the web. Compare trends over time, not absolute counts between tools.

Whatever you use to fetch the data, the monitoring rules above still apply: query with an overlap window, deduplicate changes by hash, and switch to referring domains for big sites.

Google Trends in Python Without the 429s: Momentum, Breakouts and a Weekly Trend Alert

SteadyScrape — Fri, 30 Oct 2026 14:37:47 +0000

Google Trends is one of the most useful free datasets on the internet: it shows what people are searching for, how that changes over time, and where. It's also one of the most annoying to use from code. There's no official public API, and the popular unofficial route (libraries like pytrends that call the Trends website's internal endpoints) regularly hits HTTP 429 "too many requests" errors, especially from cloud servers.

This post covers two things: what to do with Trends data once you have it (momentum, breakouts, alerts), and a way to get it reliably.

First, understand what Trends numbers mean

Trends values aren't search counts. Each series is scaled 0–100, where 100 is the peak within that request. That has consequences:

Two separate requests can't be compared. "chatgpt" at 80 and "claude" at 80 from two different queries mean nothing relative to each other. To compare terms, request them together (up to 5), so they share one scale.
The last point is partial. The current week or day is incomplete and usually looks like a drop. Leave it out of any calculation.
Small terms are noisy. Low-volume terms bounce between 0 and 100. Use longer ranges or weekly data.

Turning a series into a signal

Raw series are hard to act on. A simple momentum score is easier: compare the most recent slice of the range with the slice before it.

def momentum(points, slices=6):
    """points: [{'date': ..., 'value': int, 'partial': bool}, ...] oldest first"""
    values = [p["value"] for p in points if not p.get("partial")]
    n = max(1, len(values) // slices)
    recent, previous = values[-n:], values[-2 * n:-n]
    if not previous or sum(previous) == 0:
        return None
    pct = 100 * (sum(recent) / len(recent) / (sum(previous) / len(previous)) - 1)
    label = "rising" if pct >= 15 else "falling" if pct <= -15 else "stable"
    return round(pct, 1), label

Two other signals are worth extracting:

Peak date: when interest topped out. Is the topic past its peak?
Breakouts: Trends marks related queries that grew more than 5,000% as "Breakout". These are often the earliest signal of a new sub-topic.

Getting the data reliably

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

Google Trends API: Trend Data and Breakout Alerts (steady-scrape/google-trends-api) is an Apify Actor that returns Trends data from a licensed Google Trends API instead of scraping the Trends website, so there are no captchas, 429 errors or timed-out runs. You get one row per term with the full interest-over-time series, interest by country or region, top and rising related queries and topics, and a summary already calculated: average, latest, peak, momentum, a rising / falling / stable label and breakout lists. It uses the same "latest sixth vs the sixth before" momentum as the function above, with the partial period left out.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("steady-scrape/google-trends-api").call(run_input={
    "searchTerms": ["web scraping", "chatgpt, gemini, claude"],   # a comma line = compare on one scale
    "timeRange": "past_12_months",
})
for t in client.dataset(run["defaultDatasetId"]).iterate_items():
    if t["status"] != "ok":
        continue
    print(f'{t["searchTerm"]:<14} avg {t["averageInterest"]:>5}  momentum {t["momentumPct"]:>6}%  '
          f'{t["trend"]:<8} peak {t["peakDate"]}  breakouts: {", ".join(t["breakoutQueries"][:3])}')

Here's what a test run like that returned (worldwide, past 12 months):

Term	Average	Momentum	Trend	Peak week
web scraping	60.9	−16.5%	falling	2026-05-03
chatgpt (compared)	82	+4.5%	stable	2025-11-02
gemini (compared)	44.6	+8.8%	stable	2026-10-04
claude (compared)	14.8	+10.6%	stable	2026-06-07

The three AI assistants were requested on one line, so their numbers share a scale. Read that as "chatgpt gets roughly 5–6× the search interest of claude", not as absolute volumes. Claude was the only one of the three with breakout queries in that run, mostly new product and model names. A gibberish test term came back as no_data and wasn't charged.

The run also included the interestOverTime series, topRegions and related queries for each term, plus a googleTrendsUrl that opens the same chart on Google Trends so you can check it by eye.

Options worth knowing

region: a country or sub-region (US, US-CA, Germany).
timeRange: from past_hour to all_time (since 2004), or startDate/endDate for a custom range.
searchType: web, news, youtube, images or shopping. YouTube and Shopping trends are underrated for product research.
dataMode: Standard is cheapest and usually ready in 1–2 minutes (that 5-term run took about 70 seconds). Live returns in seconds for a higher price.

A weekly trend-change alert

Monitor mode remembers the last run for a named watchlist and flags changes: momentum past your threshold, a new breakout, or a flipped trend direction.

run = client.actor("steady-scrape/google-trends-api").call(run_input={
    "searchTerms": ["standing desk", "walking pad"],
    "region": "US",
    "monitor": True,
    "monitorName": "office-products",
    "alertThresholdPct": 25,
})
alerts = [t for t in client.dataset(run["defaultDatasetId"]).iterate_items()
          if (t.get("monitor") or {}).get("alert")]
for t in alerts:
    print(t["searchTerm"], t["monitor"]["alertReasons"])

Schedule it weekly in Apify and send alerts to Slack or email, or wire it up without code in Make, Zapier or n8n.

What it costs

Event	Price
Term with data, Standard	$0.006
Term with data, Live	$0.02
Run start (only if a term has data)	$0.005

A term inside a comparison counts as one term. No-data terms, invalid lines and errors are free. Platform usage is included.

Examples:

The 4-term test above (plus one free no-data term) cost 4 × $0.006 + $0.005 = $0.029.
A weekly monitor of 20 product terms costs about 4.3 × (20 × $0.006 + $0.005) ≈ $0.54 a month.
1,000 terms in Live mode is $20.

You can set a maximum cost per run, and the Actor only fetches what fits.

Whatever you use to fetch the data, the analysis rules above still apply: compare terms only within one request, drop the partial period, and treat momentum and breakouts as the signals, not the raw 0–100 numbers.

The Actor is Google Trends API: Trend Data and Breakout Alerts on Apify. The first run with the prefilled example input (two lines, four terms, Standard mode) cost $0.029 in our test on 2026-10-30 and took about 70 seconds.

Track Where Your Videos Rank on YouTube Search (and Pull Stats + Transcripts) Without Proxies

SteadyScrape — Fri, 30 Oct 2026 01:41:34 +0000

YouTube is the second-largest search engine, but most creators and marketers have no idea where their videos rank for the keywords they care about. YouTube Studio shows impressions, not positions. The official Data API's search.list costs 100 quota units per call (you get 10,000 per day), doesn't let you pick the signed-out ranking for a country, and doesn't return transcripts.

This post covers:

three ways to get YouTube search data, and why two of them break,
a daily "where do I rank" check in ~20 lines of Python,
adding likes, comments, subscriber counts and transcripts when you need them,
what it costs, measured.

Three ways to get YouTube search data

Official Data API: good for your own channel; quota-limited for search (about 100 searches a day), no transcripts.
Scraping youtube.com: works until it doesn't. Store scrapers in this space have public issue lists full of "403", "rate limited", "runs never finish" and "wrong view counts".
A licensed SERP feed: someone else runs the fetching at scale and sells you structured results. That's what the Actor below uses (DataForSEO's YouTube SERP API), so there are no proxies, cookies or browsers on your side.

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

The examples use YouTube Search API: Rankings, Views & Transcripts (steady-scrape/youtube-search-video-stats), an Apify Actor that returns YouTube search results in ranked order as one row per result.

A daily rank check

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("steady-scrape/youtube-search-video-stats").call(run_input={
    "searchQueries": ["python tutorial", "learn python fast", "python for beginners"],
    "maxResultsPerQuery": 50,
    "country": "US",
    "trackChannels": ["@yourchannel"],
})
summary = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("OUTPUT")["value"]
for query, pos in summary["trackedChannelPositions"].items():
    print(f"{query!r}: position {pos['position']}")

Schedule it once a day on Apify and append the output to a sheet, and you have a YouTube rank tracker. Every result row also carries position, viewCount, publishedAt, durationSec and isShort, so you can see what the videos above you have in common (age, length, Shorts vs long-form).

Stats and transcripts when you need them

Search rows already include views, channel, date and duration. Turn on addVideoDetails for likes, comment count, channel subscribers, tags and category, and addTranscripts for the captions text (optionally timestamped or translated). You can also pass a list of video URLs instead of queries.

In our test, 43 of 48 top-ranking videos had captions. Videos without captions aren't charged.

What it costs

Run	Cost at list price
1 query, top 20 results	$0.05
Daily rank check, 5 queries x 50	$0.40 per day
10 queries x 100 results	$1.52
100 video URLs with full stats	$0.62

That's $0.02 per run, plus $1.50 per 1,000 search results, $0.006 per video for full stats and $0.0065 per transcript. Paid Apify plans get 10-25% off the per-result prices. Queries with no results, missing videos and invalid input are free, and you can cap any run with a maximum charge.

Speed: a batch of queries takes about 20-25 seconds. Our 100-result test finished in 36 seconds.

Wrap-up

If you just need to know where you rank and what the top videos look like, a licensed SERP feed is the least painful route: no quota math, no proxy bills, no broken selectors. The Actor is YouTube Search API: Rankings, Views & Transcripts on Apify. The first run with the prefilled example input (two queries, top 20 each) costs about $0.08.

Turn a List of Domains Into Company Profiles (Logo, Socials, Industry) Without Scraping LinkedIn

SteadyScrape — Thu, 29 Oct 2026 15:54:43 +0000

You have a column of domains (sign-ups, a trade-show list, a CRM export) and you want a name, logo, one-line description, industry and the company's LinkedIn page next to each. Enrichment APIs sell this as a subscription. But for most companies the data is published on their own home page, in machine-readable form.

Where the data actually lives

schema.org JSON-LD. Many sites ship an Organization block for Google: name, logo, sameAs (their official social profiles), foundingDate, address, sometimes numberOfEmployees.
OpenGraph and meta tags. og:site_name, og:image and meta description are on almost every site, because they make link previews look right.
The header and footer. Social icons, mailto: and tel: links, and links to "About" and "Contact".
The about and contact pages. "Founded in 2014", the postal address and the info@ mailbox.

A minimal extractor in Python

import json, re
from curl_cffi import requests   # pip install curl_cffi lxml
from lxml import html

def profile(domain):
    r = requests.get(f"https://{domain}", impersonate="chrome", timeout=20)
    doc = html.fromstring(r.content)
    meta = {m.get("property") or m.get("name"): m.get("content") for m in doc.xpath("//meta") if m.get("content")}
    org = {}
    for s in doc.xpath('//script[@type="application/ld+json"]'):
        try:
            data = json.loads(s.text_content())
        except ValueError:
            continue
        for node in (data if isinstance(data, list) else data.get("@graph", [data])):
            if isinstance(node, dict) and node.get("@type") in ("Organization", "Corporation", "LocalBusiness"):
                org = node
    links = doc.xpath("//a/@href")
    return {
        "name": org.get("name") or meta.get("og:site_name"),
        "description": meta.get("description") or meta.get("og:description"),
        "logo": org.get("logo") if isinstance(org.get("logo"), str) else meta.get("og:image"),
        "linkedin": next((l for l in links if "linkedin.com/company/" in l), None),
        "emails": sorted({l[7:].split("?")[0] for l in links if l.startswith("mailto:")}),
    }

print(profile("shipbob.com"))

That gets you surprisingly far. The hard part is the edge cases.

The edge cases that eat your week

Names. The <title> is often a slogan ("Love your bank | Spend, save and invest"). Score each candidate (schema name, og:site_name, title segments) by how well it matches the domain, and fall back to a word in the page text that equals the domain label (N26, 3M).
Social links. Footers link to articles and share buttons too: twitter.com/intent/tweet, facebook.com/sharer, a GitHub repo mentioned in a blog post. Keep linkedin.com/company/… and drop linkedin.com/in/… (that's a person). Trust sameAs first.
Personal data. A contact page can list a named person's email. If you only want company data, keep role mailboxes on the company's own domain (info@, sales@, press@) and drop the rest.
"Founded in 1983". On an about page, that can be from a person's bio ("she co-founded … in 1983"). Ignore years near personal context.
Bot walls, parked domains and maintenance pages. These return HTTP 200 with junk. Detect them and don't count them as results.
Encodings. Not every site is UTF-8. Read the header and <meta charset> before decoding.

What fill rates to expect

From test runs on October 27, 2026 (home + about + contact pages, no browser):

	60 well-known companies	500 random domains from a top-sites list
Profile found	90%	61% (the rest were dead, parked, blocked or empty)
Name / description	100% / 100%	100% / 90%
Logo	78%	84%
LinkedIn company page	67%	32%
Instagram	85%	51%
Generic email	35%	24%
Founded year	41%	20%

Without a browser, JavaScript-only sites give you nothing. With browser-like TLS (curl_cffi), most "403 for bots" pages go away, but sites with real bot protection still block cloud IPs.

The hosted version

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

Company Data by Domain (steady-scrape/company-profile-by-domain) is an Apify Actor that does the above for lists of up to 20,000 domains per run. It adds an industry guess with a confidence label (high, medium or low), the postal address, tech hints and a per-row status, so you can see why a domain didn't return a profile. It reads only the company's own public site (home, about and contact pages), doesn't scrape LinkedIn and doesn't return personal data.

import os
from apify_client import ApifyClient   # pip install apify-client (v3)

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("steady-scrape/company-profile-by-domain").call(
    run_input={"domains": ["stripe.com", "shipbob.com", "levainbakery.com"]})
for row in client.dataset(run.default_dataset_id).iterate_items():
    print(row["domain"], row["status"], row.get("companyName"),
          row.get("industryGuess"), row.get("industryConfidence"), row.get("linkedinUrl"))

Output from a run today (October 29, 2026), about 12 seconds end to end:

shipbob.com ok ShipBob Logistics / Transportation high https://www.linkedin.com/company/shipbob
stripe.com ok Stripe Fintech / Payments medium https://www.linkedin.com/company/stripe
levainbakery.com ok Levain Bakery Food & beverage high https://www.linkedin.com/company/levain-bakery

Rows with status other than ok (unreachable, blocked, parked, empty, invalid or duplicate domains) come back with the reason, so you can retry or drop them.

What it costs

It's pay per result: $0.004 per company profile found ($4 per 1,000), plus Apify's standard $0.00005 start event per run. Dead, blocked, parked and empty domains are free, and Apify paid plans get volume discounts on the same event.

Example: you upload 1,000 domains, 700 return a profile and 300 are dead, parked or blocked. You pay 700 × $0.004 = $2.80. You can set a maximum cost per run, and the Actor stops before it goes over.

If you need revenue or headcount estimates, this isn't the tool: those come from data vendors, not from the company's website. If you want to rank the same list by size, Bulk Domain Rank & Traffic Checker adds traffic estimates, backlinks and spam score per domain.

Vetting 1,000 Domains in a Minute: Domain Rank, Backlinks, Spam Score and Traffic From Python

SteadyScrape — Wed, 28 Oct 2026 14:42:45 +0000

A lot of SEO and growth work starts with a spreadsheet of domains and the same question: which of these are worth our time?

A link builder has 800 guest-post prospects and needs to drop the spammy ones.
A sales team has a lead list and wants to sort it by how much traffic each company gets.
A marketer wants a quick size comparison of ten competitors.

Opening each domain in an SEO tool doesn't scale past a few dozen. This post walks through a scriptable version: clean the list, pull link and traffic metrics in bulk, and score the results.

Step 1: clean the list

Domain lists are always messy: full URLs, www., trailing paths, duplicates, the odd email address. Normalize before you pay for anything:

import re
from urllib.parse import urlparse

def normalize(line):
    s = line.strip().lower()
    if not s or " " in s or "@" in s:
        return None
    if "://" not in s:
        s = "http://" + s
    host = urlparse(s).hostname or ""
    host = re.sub(r"^www\d?\.", "", host)
    return host if re.fullmatch(r"[a-z0-9-]+(\.[a-z0-9-]+)+", host) else None

raw = ["https://www.HubSpot.com/blog", "backlinko.com", "apify.com/store", "not a domain", "backlinko.com"]
domains = sorted({d for d in map(normalize, raw) if d})
print(domains)   # ['apify.com', 'backlinko.com', 'hubspot.com']

Step 2: decide which metrics you need

For vetting, four numbers do most of the work:

Metric	What it tells you
Domain rank (0–100)	Overall link-graph strength
Referring domains	How many separate sites link to it. Harder to fake than raw backlinks
Spam score (0–100)	Whether its backlink profile looks manipulated
Organic traffic (estimated monthly visits)	Whether real people find it through Google

High rank with almost no traffic is a classic sign of a link-farm site. Strong traffic with a modest rank often means a real, niche site that's worth a pitch.

Step 3: pull the metrics in bulk

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

Bulk Domain Rank & Traffic Checker (steady-scrape/domain-seo-metrics) is an Apify Actor that takes up to 10,000 domains per run and returns one row per domain: domain rank, backlinks, referring domains, spam score and the date links were first seen, plus estimated organic and paid Google traffic and ranking keywords for 90+ countries. The data comes from a licensed SEO data index built for API access, not from scraping SEO tools' pages, so 1,000 domains take under a minute.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("steady-scrape/domain-seo-metrics").call(run_input={
    "domains": domains,          # from step 1
    "region": "US",
})
rows = [r for r in client.dataset(run["defaultDatasetId"]).iterate_items() if r["status"] == "ok"]

Here's a slice of a real test run from October 27, 2026 (US traffic; link metrics are global, and the counts drift a little day to day):

Domain	Rank	Referring domains	Spam score	Organic traffic / month
hubspot.com	88	400,250	4	497,381
zapier.com	77	102,674	16	1,632,175
ahrefs.com	82	106,113	23	308,744
apify.com	76	15,314	18	206,492
backlinko.com	70	61,550	2	19,941
spiegel.de	82	261,808	28	44,860

That run had 19 input lines: 17 came back with data, one domain had no data, and one line wasn't a domain at all. The last two were free. The whole run took about 5 seconds.

Notice spiegel.de: strong links, but low US traffic, because it's a German site and traffic is measured for the region you pick. Set region to the market you care about, or worldwide.

Step 4: score and filter

Now turn the numbers into a decision. A simple prospect filter:

def keep(r):
    return (
        r["domainRank"] >= 40
        and r["referringDomains"] >= 500
        and r["backlinksSpamScore"] <= 30
        and (r.get("organicTraffic") or 0) >= 1_000
    )

def score(r):
    # traffic matters more than rank for outreach; penalize spam
    return r["domainRank"] + 10 * (len(str(r.get("organicTraffic") or 0)) - 1) - r["backlinksSpamScore"]

prospects = sorted(filter(keep, rows), key=score, reverse=True)
for r in prospects:
    print(f'{r["domain"]:<25} rank {r["domainRank"]:>3}  spam {r["backlinksSpamScore"]:>3}  traffic {r.get("organicTraffic")}')

Tune the thresholds to your niche. Local and B2B sites rarely have huge traffic, and that's fine.

Going deeper on competitors

For a handful of competitors, the optional deep report adds their top keywords (with position, volume, CPC and difficulty), the ranking distribution (top 3 / top 10 / top 100), organic traffic value in USD, and their real search competitors:

run = client.actor("steady-scrape/domain-seo-metrics").call(run_input={
    "domains": ["backlinko.com"],
    "deepReport": True,
    "deepTopKeywords": 25,
    "deepCompetitors": 10,
})

In our test, backlinko.com showed 236 keywords in the top 3 and 1,113 in the top 10, with an estimated traffic value of about $27,900 a month. Its closest search competitors were other SEO publishers and tools.

Honest limits

Traffic is an estimate, built from rankings × search volume × click-through rate. Every SEO tool works this way. It's good for comparing sites, but it won't match a site's own analytics.
Domain rank is this index's own score. It isn't Moz DA or Ahrefs DR, though it moves in the same direction.
Very new or tiny sites may have no data. Those rows are free.

What it costs

$4 per 1,000 domains with data ($0.004 each) on Apify's free plan, dropping to $3 per 1,000 on higher Apify plans. No-data, invalid and error rows are free.
$0.05 per run, charged only if at least one domain returns data.
Deep report: $0.10 per domain, only when the report comes back non-empty.
Apify's $0.00005 start fee per run.

So vetting 1,000 prospects costs $4.05, and the 17-domain test above cost about $0.12. A deep dive on five competitors adds $0.50. You can set a maximum cost per run, and the Actor won't fetch beyond it.

The cleaning and scoring code above works with any data source. Even if you get metrics elsewhere, normalize first and filter on referring domains and spam, not on rank alone.

Monitoring Competitors' Google Ads With the Ads Transparency Center (and Getting Alerts for New Ones)

SteadyScrape — Tue, 27 Oct 2026 21:53:51 +0000

If you work in marketing, someone eventually asks: "What ads are our competitors running right now?" Since 2023, Google has had a public answer: the Ads Transparency Center (adstransparency.google.com). It lists the ads every verified advertiser runs across Search, YouTube, Shopping, Maps and Play, with first-shown and last-shown dates.

It's a great resource for one-off checks. This post is about turning it into data you can work with: a table of a competitor's creatives, the signals worth pulling out of it, and a daily alert when a competitor launches something new.

What you can see manually

Search an advertiser name or a domain, pick a region, and you get a grid of creatives. Click one and you see:

the format (text, image or video) and a rendered preview,
first shown and last shown dates,
the advertiser's verified name and its location.

The UI is fine for looking at one competitor once. It gets slow when you want to compare five competitors, sort by run time, or notice new creatives the day they appear. There's no export button, and no official API for the ad library.

The signals that matter

Before collecting anything, decide what you'll do with it. Three signals carry most of the value:

Days running. Advertisers stop paying for ads that don't work. A creative that has run for two years is probably a winner, so study its angle and offer.
Active now. "Last shown" within a day or two means it's live. Comparing active ads across competitors tells you who's pushing hardest right now.
New creatives. A batch of new ads usually means a launch, a promotion or a new market.

Getting it as data

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

Google Ads Transparency Scraper (steady-scrape/google-ads-transparency) is an Apify Actor that returns Transparency Center data as one row per creative. You can query by domain (hubspot.com), by advertiser ID, or by company name (it finds the advertiser accounts for you). It doesn't use a browser, proxies or a login. The data comes from a licensed ads-data API, so runs don't break when Google changes the page.

Here's a script that pulls a competitor's US ads and summarizes them:

import os, statistics
from collections import Counter
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("steady-scrape/google-ads-transparency").call(run_input={
    "targets": ["hubspot.com"],
    "region": "US",
    "maxAdsPerTarget": 40,
})
ads = [a for a in client.dataset(run["defaultDatasetId"]).iterate_items() if a.get("creativeId")]

print(Counter(a["format"] for a in ads))
print("active now:", sum(a["isActive"] for a in ads))
print("median days running:", statistics.median(a["daysRunning"] for a in ads))
for a in sorted(ads, key=lambda a: -a["daysRunning"])[:5]:
    print(a["daysRunning"], a["format"], a["adUrl"])

In our test run, that returned 40 ads in about 14 seconds. Google estimated around 2,000 ads in total for the domain:

28 text, 11 video, 1 image, all 40 shown in the last 48 hours.
Median days running: about 370. Half of the returned creatives had run for more than a year, and the longest for 1,828 days, roughly five years.
Not every ad was HubSpot's own. A domain query returns every ad that points to that domain, so a few came from partners. The advertiserName column shows who paid. To see one company's own ads only, query its advertiser ID or use a name search.

Each row has advertiserName, advertiserId, verified, creativeId, format, previewImageUrl or previewUrl, firstShown, lastShown, daysRunning, isActive and adUrl (a link to the creative in the Transparency Center). The longest-running ads are the ones to open first.

Finding the advertiser by name

Don't know a competitor's domain or ID? Search by name and, optionally, pull ads for the best match in the same run:

run = client.actor("steady-scrape/google-ads-transparency").call(run_input={
    "advertiserSearch": ["mailchimp"],
    "fetchAdsForFoundAdvertisers": True,
    "region": "US",
})

In our test, the "mailchimp" search returned one advertiser account with its related domains (mailchimp.com, mailchimpsites.com).

A daily new-ads alert

Monitor mode remembers which creatives you've already received and returns only new ones:

run = client.actor("steady-scrape/google-ads-transparency").call(run_input={
    "targets": ["competitor-one.com", "competitor-two.com"],
    "onlyNewAds": True,
    "lastShownWithinDays": 2,
})
new_ads = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Schedule that daily on Apify (or call it from cron), and send new_ads to Slack, email or a sheet. You can also connect it without code through Make, Zapier or n8n.

Honest limits

120 creatives per query. That's how much the Transparency Center returns, newest first. With format all, the Actor queries text, image and video separately, which lifts this to 360 unique creatives per target. Very large advertisers are covered by their most recently shown ads.
No ad copy text yet, and no regional impression ranges. For text and image ads, the preview image shows the rendered copy.
Old date ranges can come back empty. Google keeps fewer historical creatives.
Counts like totalAdsApprox are Google's rounded estimates.

What it costs

Event	Price
Ad row	$0.001 ($1 per 1,000)
Advertiser from a name search	$0.003
Monitor check, per target after the baseline run	$0.008
New ad found in monitor mode	$0.003

Errors, "no ads" results and filtered-out ads are free, and Apify's $0.00005 start fee applies per run.

Examples:

The 40-ad pull above costs $0.04.
A full 360-ad snapshot of one competitor costs $0.36.
A daily monitor on 5 competitors costs 5 × $0.008 × 30 = $1.20 a month in checks, plus $0.003 per new creative. Twenty new ads in a month adds $0.06.

You can set a maximum cost per run, and the Actor only fetches what fits in that limit.

The ad library has been public for years, but most teams still check it by hand, if they check it at all. Even a weekly table of "ads running over 180 days" for each competitor tells you a lot about what's working in your market.

How to Check Whether ChatGPT and Perplexity Recommend Your Brand

SteadyScrape — Tue, 27 Oct 2026 02:54:56 +0000

More and more buying research starts with a question to an AI assistant instead of a list of blue links: "best CRM for a small business", "easiest invoicing tool for freelancers", "Notion alternatives for a 10-person team". The assistant answers with a short list of names. If your product isn't on that list, that buyer may never see you.

This post covers three things: why this is worth tracking, how to check it by hand in about an hour, and how to automate the check once a spreadsheet stops being enough. The example data is from a real test run on October 26, 2026.

Why AI answers are worth tracking

A few numbers, each from its original source:

ChatGPT reach. OpenAI says more than 1.2 billion people use ChatGPT each week (OpenAI, October 7, 2026).
Fewer clicks when an AI summary appears. Pew Research Center looked at the browsing data of 900 U.S. adults. When a Google search showed an AI summary, users clicked a traditional result in 8% of visits, compared with 15% when there was no summary. They clicked a link inside the summary in just 1% of visits (Pew Research Center, July 2025). That study covers Google's AI summaries, not ChatGPT, and it shows a correlation, not cause and effect. Still, the pattern is clear: when the answer is on the page, fewer people click through to compare.
Analysts expected the shift. In 2024 Gartner predicted that traditional search engine volume would drop 25% by 2026 as AI chatbots and virtual agents take over queries (Gartner press release, February 19, 2024). That was a forecast, not a measurement, but it explains why marketing teams started paying attention.

The practical takeaway is simple. For the questions your buyers ask, you want to know three things:

Are you mentioned at all?
Where do you appear (first pick, or fourth in the "also consider" list)?
Which sources does the assistant cite? Your own site, a review site, or a competitor's comparison page?

The third one matters more than it seems. Citations show you which pages the assistant actually trusts for your category, and those pages are where your content and PR effort will pay off.

The manual way (about an hour)

You don't need any tooling for a first look.

1. Write 10–20 buyer prompts. Use the words a buyer would use, not your brand name. Mix in:

category questions: "best CRM for small business"
use-case questions: "easiest CRM for a 5-person sales team"
alternative questions: "cheaper alternatives to Salesforce"

Keep your own brand out of most prompts. If you ask "is HubSpot good?" the answer will mention HubSpot, and that tells you nothing about organic visibility.

2. Ask each engine. Use ChatGPT and Perplexity in a private window or a fresh chat, so your history and memory don't steer the answer. Turn web search on if it isn't automatic.

3. Log the answer in a spreadsheet, one row per prompt and engine:

Prompt	Engine	We're mentioned?	Our position	Competitors mentioned	Our site cited?	Cited domains

4. Repeat it a few days later. This is the step people skip, and it's the important one. AI answers are not stable. In one of our own tests, we sent the same 6 prompts to Perplexity twice on the same day and compared the answers. Across the brands we tracked, 4 mentions disappeared and 2 new ones showed up between the two runs. One answer is an anecdote. You need a sample, repeated over time.

A small helper for the "position" column

Reading 40 answers and counting who came first gets old fast. If you paste answer text into a file, this short function finds which tracked brands are mentioned and in what order. It removes links first, so a cited URL like hubspot.com/pricing doesn't count as a mention:

import re

def brand_ranks(answer, brands):
    """brands: {"HubSpot": ["HubSpot", "HubSpot CRM"], ...} -> {brand: rank or None}"""
    text = re.sub(r"\[[^\]]*\]\([^)]*\)|https?://\S+", " ", answer)  # drop markdown links and bare URLs
    first = {}
    for brand, names in brands.items():
        hits = [m.start() for n in names
                for m in re.finditer(rf"(?<!\w){re.escape(n)}(?!\w)", text, re.I)]
        if hits:
            first[brand] = min(hits)
    order = sorted(first, key=first.get)
    return {b: (order.index(b) + 1 if b in first else None) for b in brands}

print(brand_ranks(answer, {
    "HubSpot": ["HubSpot", "HubSpot CRM"],
    "Pipedrive": ["Pipedrive"],
    "Zoho CRM": ["Zoho CRM", "Zoho"],
    "Salesforce": ["Salesforce"],
}))
# {'HubSpot': 2, 'Pipedrive': 1, 'Zoho CRM': None, 'Salesforce': None}

That output is from a real ChatGPT answer to "easiest CRM for a 5-person sales team". The answer did recommend Bigin, which is a Zoho product, but it never said "Zoho". The lesson: list product names as aliases, not just the company name. Otherwise you'll undercount.

Where the manual approach breaks down

It works for a first look. It gets painful when you want:

more than a couple dozen prompts, or several markets
both engines, every week, with the same settings each time
the cited URLs, not just the answer text
share of voice, meaning how often you're mentioned compared with each competitor across all prompts
a diff against last week: which prompts you gained, lost or moved on

That is the point where it makes sense to script it or use a hosted tool.

Automating it with the AI Visibility Tracker

Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.

We packaged this workflow into an Apify Actor: AI Visibility & Share of Voice Tracker (steady-scrape/ai-brand-visibility-tracker). For every prompt × engine it:

asks the engine through its official API with live web search (no browser automation, no scraping of chatgpt.com)
records, for your brand and each competitor: mentioned or not, mention count, rank, list position, a simple sentiment hint and the surrounding text
saves every cited URL and domain, and flags whether your own domain was cited
computes share of voice and visibility rate, overall and per engine
compares the run with your previous run if you give it a trackingId, so a weekly schedule becomes a tracker

An honest caveat first. The answers come from the developer APIs, not the consumer apps. The chatgpt engine uses an OpenAI model (by default gpt-6-luna, the model family OpenAI says powers ChatGPT's Free and Go tiers) with OpenAI's own web search, called through OpenRouter or OpenAI's API. The perplexity engine uses Perplexity's Agent API with web search. The apps add their own system prompts, personalization, memory and location, so a logged-in user can see a different answer. Treat the results as a repeatable, comparable sample, not a screenshot of what any one person sees.

Example input

This is the input from our real test run (2 prompts × 2 engines = 4 checks). maxChecks and maxCostUsd are safety caps:

{
  "prompts": [
    "best CRM for small business",
    "easiest CRM for a 5-person sales team"
  ],
  "brands": [
    { "name": "HubSpot", "aliases": ["HubSpot CRM"], "domains": ["hubspot.com"] }
  ],
  "competitors": [
    { "name": "Salesforce", "domains": ["salesforce.com"] },
    { "name": "Zoho CRM", "aliases": ["Zoho"], "domains": ["zoho.com"] },
    { "name": "Pipedrive", "domains": ["pipedrive.com"] },
    { "name": "Freshsales", "domains": ["freshworks.com"] }
  ],
  "engines": ["chatgpt", "perplexity"],
  "trackingId": "real-pricing-check",
  "maxChecks": 4,
  "maxCostUsd": 0.3
}

(HubSpot and friends are just an example category here. We have no connection to any of them.)

Example output (real run, trimmed)

All 4 checks succeeded in about 12 seconds. The interesting part is the second prompt, where the two engines disagreed. Here's the ChatGPT row, trimmed:

{
  "prompt": "easiest CRM for a 5-person sales team",
  "engine": "chatgpt",
  "model": "openai/gpt-6-luna",
  "answerSnippet": "**My pick: Pipedrive Lite** if you want a straightforward, sales-focused CRM your reps can learn quickly. ...",
  "brandMentioned": true,
  "bestBrandRank": 2,
  "brandDomainCited": true,
  "competitorsMentioned": ["Pipedrive"],
  "citedDomains": ["pipedrive.com", "zoho.com", "hubspot.com"],
  "brands": [
    { "brand": "HubSpot",   "mentioned": true,  "rank": 2, "listPosition": 2, "sentiment": "neutral", "domainCited": true },
    { "brand": "Pipedrive", "mentioned": true,  "rank": 1, "listPosition": null, "sentiment": "neutral", "domainCited": true },
    { "brand": "Zoho CRM",  "mentioned": false, "rank": null, "domainCited": true }
  ],
  "searchCalls": 2,
  "charged": true,
  "chargeEvent": "chatgpt-check"
}

And the Perplexity row for the same prompt:

{
  "prompt": "easiest CRM for a 5-person sales team",
  "engine": "perplexity",
  "answerSnippet": "**Pipedrive** is my pick for a five-person sales team: its visual deal pipeline is easy to understand ...",
  "brandMentioned": false,
  "bestBrandRank": null,
  "brandDomainCited": false,
  "competitorsMentioned": ["Pipedrive"],
  "citedDomains": ["capterra.com", "lessannoyingcrm.com"],
  "charged": true,
  "chargeEvent": "perplexity-check"
}

Same question, two engines: ChatGPT put HubSpot second and cited hubspot.com, while Perplexity didn't mention it at all. Perplexity built its answer from a Capterra review page and a smaller competitor's pricing page. That's exactly the kind of gap you'd want to know about, and it's invisible if you only check one engine.

The run also writes a summary row with share of voice across all 4 checks (trimmed):

{
  "recordType": "summary",
  "checksSucceeded": 4,
  "chargedUsd": 0.18,
  "shareOfVoice": [
    { "brand": "Pipedrive", "shareOfVoice": 0.3889, "visibilityRate": 1.0,  "avgRank": 2,    "citationRate": 0.5 },
    { "brand": "HubSpot",   "shareOfVoice": 0.3333, "visibilityRate": 0.75, "avgRank": 1.33, "citationRate": 0.75 },
    { "brand": "Zoho CRM",  "shareOfVoice": 0.2222, "visibilityRate": 0.5,  "avgRank": 2,    "citationRate": 0.75 },
    { "brand": "Freshsales","shareOfVoice": 0.0556, "visibilityRate": 0.25, "avgRank": 4,    "citationRate": 0 },
    { "brand": "Salesforce","shareOfVoice": 0,      "visibilityRate": 0,    "avgRank": null, "citationRate": 0.25 }
  ],
  "topCitedDomains": [
    { "domain": "hubspot.com", "checksCiting": 3 },
    { "domain": "zoho.com", "checksCiting": 3 },
    { "domain": "pipedrive.com", "checksCiting": 2 }
  ]
}

Two prompts is far too small a sample to conclude anything about these brands. It's here to show the format. Salesforce is a good example of why the citation column matters: it was never mentioned, but one answer still cited a salesforce.com page.

Running it from code

Any Apify Actor can be called over the Apify API. This runs the tracker and returns the dataset rows in one request:

curl -X POST \
  "https://api.apify.com/v2/acts/steady-scrape~ai-brand-visibility-tracker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d @input.json

For a tracker, save the input as an Apify Task and put it on a weekly schedule. Keep the same trackingId and each run's summary includes a changes block against the previous run.

Pricing

It's pay per event, and you only pay for checks that return an answer:

Event	Price
ChatGPT check (one prompt answered by ChatGPT and analyzed)	$0.06
Perplexity check (one prompt answered by Perplexity and analyzed)	$0.03
Actor start	$0.00005 per run (Apify's standard fee)

Some examples:

The test run above (2 prompts × 2 engines) costs $0.18.
10 prompts on both engines cost $0.90 per run, so a weekly schedule comes to about $3.90 a month.
Failed, skipped and mock-mode checks are never charged. Set "mockMode": true to see the full output format for free before you spend anything.
Your Max charge per run setting is respected: the Actor only plans the checks that fit under it.

Wrapping up

Start by hand. Ten unbranded buyer prompts, two engines, one spreadsheet, and a second pass a few days later will tell you a lot about where you stand. Pay particular attention to the cited domains: they show you which review sites and comparison pages the assistants lean on for your category.

Once you want dozens of prompts, both engines, citations and week-over-week changes, automate it. You can write your own script, or use the AI Visibility Tracker if your time is worth more than $0.03–$0.06 a check.

Have you seen your brand show up (or vanish) in AI answers? We'd like to hear which prompts surprised you. Please share them in the comments.