DEV Community: SerpApi

Ecommerce Web Scraping: How to Track Competitor Prices, Products, and Stock

Srujana Maddula — Thu, 29 Oct 2026 11:01:45 +0000

Almost all major e-commerce players already scrape each other’s sites, and they feed that data straight into pricing intelligence, inventory stocking and supply chain optimization. If you don't do it, you’re behind.

In this guide, I’ll show you how ecommerce web scraping works and which data is worth scraping. Even if you can’t pull entire catalogs, you’ll learn the two signals everyone should collect at a minimum. You’ll also get a working script that scrapes a popular e-commerce site in Python. And if you have another marketplace like Amazon or Walmart in mind, I have a solution for that too.

What Ecommerce data is worth scraping?

You can scrape almost anything from an online store. The question is what’s worth the effort. In this section, the first two give you a rich dataset, and the last two are the minimum every store should be scraping.

Product detail pages

Scrape product pages

By scraping single product pages, you get title, brand, category, price, availability, customer reviews, product description, and much more about an item.

Start with how they sell their best sellers. That means compare the descriptions of their top products with their slow movers. Note which features they lead with and which claims they repeat. Then test those same moves on your own listings.

The product detail page also carries the customer reviews. Reviews tell you which features buyers care about, what breaks or disappoints them, and whether they felt the price was fair. If the same phrase keeps showing up, like “too expensive for what it is,” a better-priced alternative could win those buyers. If reviews keep praising a product as “great value for the price,” you learn what shoppers consider fair and can position your own products around it.

And here’s the clever part. You’ll never see a competitor’s real sales numbers. But review counts are a decent stand-in. When a product that usually gets a handful of reviews a week suddenly gets dozens, something has happened: a sales spike, a promotion or a product going viral on TikTok. Spotting it early tells you what’s about to be in demand.

You’ll also see which category/department each product belongs to. Compare them with your own catalog to fix misclassified products and to spot the categories and products you’re missing.

Search results listings

Scrape ecommerce search results

When shoppers search for a product on a marketplace, the results page shows what the platform thinks is relevant to their query. Those listings are worth scraping.

You can collect product titles, images, prices, rankings, and other listing details for a search query. This shows how competitors position their products for different search terms, so you can benchmark your own listings, sharpen your titles and descriptions, and spot gaps in your assortment.

For example, say Amazon's top results for "cordless drill 20V" include bestseller brands or models that Walmart doesn't have. That could be a buying opportunity: stock those products or consider a comparable alternative under your own brand.

Search results also tells you how a marketplace interprets a query. If the top results for "air fryer" mostly feature large-capacity models, that's a clue about what shoppers may expect to find for that search. If your site keeps surfacing irrelevant products for the same query, you identified to fix a search-relevance problem.

The above two broad categories: product pages and search results are the two richest datasets. But if you're a smaller retailer and don't need that much data, or don't have the budget to build a comprehensive competitor dataset yet? Start with the price and stock data fields that are useful to almost every e-commerce business.

Product prices

Price is the first thing a shopper weighs before buying, and it’s the most tracked field. In a June 2024 letter to Walmart and Amazon, Senator Sherrod Brown pointed out that some retailers using pricing algorithms change their prices more than 2,000 times a day.

So scrape their prices regularly, and decide in time whether to match a drop, offer a better deal, or hold your price because your margins matter more. Over time, that price history also shows you how they run promotions, so you can plan your own pricing instead of only reacting to theirs. If you only get to track one thing, track this.

Stock availability

When a competitor runs out of a product people want, those shoppers look somewhere else. If you’re watching, that somewhere else can be you. Restock it, feature it, or sharpen your price while they’re empty, and you catch the demand they couldn’t serve.

Why is Ecommerce web scraping difficult?

Ecommerce sites are some of the most heavily defended sites on the web. Basic defenses like IP rate limiting and User-Agent blocking are common across the web. But ecommerce sites often go further, using advanced browser fingerprints and other signals to spot automated traffic.

Big retailers rely on bot-protection services like Akamai, DataDome and Cloudflare, which look far beyond your IP. They check whether your connection really looks like Chrome, whether your browser has the quirks a normal one would, and whether you scroll, click and pause like a person. A script that loads page after page at perfect intervals stands out right away.

And even when you get that far, a CAPTCHA can stop you in your tracks. Sites use these challenges when they suspect a visitor is a bot, sometimes alongside other checks that run in the background.

So a generic automation script won't help pass such sophisticated defense systems. Even if you try to build the first class scraper yourself, you have to spend on proxies and IPs. To remove that hassle, you can try the below simple API call approach to get any e-commerce data regularly.

Scraping an Ecommerce website in Python

You can send any public URL to SerpApi's Extract API and get the page back as structured JSON or Markdown. In this example, you’ll start from a Best Buy search results page, find the products on it, and pull each product’s title, price and stock status into clean JSON.

Prerequisites

You’ll need:

Python 3.8 or later
A code editor like VS Code, or a Jupyter notebook
A SerpApi account, which comes with 250 free searches per month.

Set up SerpApi

Install the SerpApi Python client:

pip install serpapi

Then save your API key as an environment variable, so it never ends up in your code:

export SERPAPI_API_KEY="your_api_key"

Choose your target site and the data you want

In this case, it is Best Buy’s search results for the search term "samsung galaxy": https://www.bestbuy.com/site/searchpage.jsp?id=pcat17071&st=samsung%20galaxy

The search page shows a grid of products, but the details you want live on each product’s own page. So the script works in two passes:

Fetch the search page and collect the product links.
Visit each product page and pull out the data.

For each product, you’ll collect:

Title
Product URL
Price
Stock availability

Unlike CSS-selector based ecommerce data scraping, you don’t need to open DevTools and hunt for class names. The Extract API reads the page for you, including the structured product data most big stores embed for Google.

Build the Ecommerce web scraper

Create a file called best_buy_scraper.pyand add the below code we discuss in each step.

1. Import the packages and set your target

import html
import json
import os
import sys
from urllib.parse import urlsplit, urlunsplit

import serpapi


PRODUCT_LIMIT = 3
SEARCH_PAGE_URL = "https://www.bestbuy.com/site/searchpage.jsp?id=pcat17071&st=samsung%20galaxy"

PRODUCT_LIMIT sets how many products you want back. SEARCH_PAGE_URL is your search term result page of Best Buy. To track another keyword listings, change st=samsung%20galaxy to any search term.

2. Filter the product links

def bestbuy_product_links(data):
    links = []
    seen = set()
    headings = {
        item.get("position"): html.unescape(item["title"])
        for item in data.get("headings", [])
        if isinstance(item, dict)
        and isinstance(item.get("position"), int)
        and isinstance(item.get("title"), str)
    }
    for item in data.get("inline_links", []):
        if not isinstance(item, dict) or not isinstance(item.get("link"), str):
            continue
        parsed = urlsplit(item["link"])
        hostname = (parsed.hostname or "").lower()
        if parsed.scheme not in ("http", "https"):
            continue
        if hostname != "bestbuy.com" and not hostname.endswith(".bestbuy.com"):
            continue
        if not parsed.path.lower().startswith("/product/"):
            continue

        link = urlunsplit((parsed.scheme, parsed.netloc, parsed.path, parsed.query, ""))
        if link not in seen:
            seen.add(link)
            links.append({
                "link": link,
                "title": item.get("title") or headings.get(item.get("position")),
            })
    return links

Here’s what each check does:

parsed.scheme: drops anything that isn’t a regular web link, like mailto: or javascript: links.
hostname: keeps only links on bestbuy.com or its subdomains, so ads pointing to other sites are out.
parsed.path.startswith("/product/"): keeps only product pages. Every Best Buy product URL follows this pattern.
urlunsplit(...) and seen: strips the #fragment from each URL and skips duplicates. The same product often appears twice on a page, once on its image and once on its title.
headings: if a link has no title of its own, the function borrows the title from the nearest heading on the page.

If you adapt this script for another store, this is the function to change. Amazon product URLs, for example, contain /dp/ instead of /product/.

Since the Extract API is a generic API for all web pages, you need to parse Best Buy URLs to find product links and then parse the JSON output to extract the required fields.

However, if you want to scrape major retailers, SerpApi provides dedicated Walmart , Amazon , and eBay APIs, along with specialized APIs for other platforms. A single call to any of these retail APIs returns structured JSON with fields like price, title, product ID, and much more, making the data much easier to work with.

SerpApi's 250 free searches per month give you access to all these APIs. Check out these guides to scraping Walmart and Amazon data.

3. Pull out price and stock

This function takes the Extract API’s response for one product page and returns the output:

def extract_prices(data, result):
    if (
        data.get("error")
        or data.get("search_metadata", {}).get("status") == "Error"
    ):
        raise RuntimeError(
            data.get("error") or "SerpApi Extract request failed"
        )

    product = next(
        (
            item for item in data.get("json_ld", [])
            if isinstance(item, dict) and item.get("@type") == "Product"
        ),
        {},
    )
    offer = product.get("offers") or {}
    if isinstance(offer, list):
        offer = offer[0] if offer else {}
    if not isinstance(offer, dict):
        offer = {}

    page_offer = next(
        (
            item for item in data.get("price_offers", [])
            if isinstance(item, dict)
            and item.get("extracted_price") is not None
        ),
        {},
    )
    current_price = offer.get("price")
    if current_price is None:
        current_price = page_offer.get("extracted_price")

    currency = offer.get("priceCurrency") or page_offer.get("currency")
    if current_price is not None and currency:
        amount = str(current_price)
        if amount.endswith(".00"):
            amount = amount[:-3]
        symbol = "$" if currency == "USD" else f"{currency} "
        current_price = f"{symbol}{amount}"

    availability = offer.get("availability") or page_offer.get("availability")
    if availability:
        availability = str(availability).rsplit("/", 1)[-1]
        availability = {
            "instock": "In stock",
            "outofstock": "Out of stock",
        }.get(availability.lower(), availability)

    return {
        "title": (
            product.get("name")
            or (data.get("page_information") or {}).get("title")
            or result.get("title")
        ),
        "url": result["link"],
        "regular_price": current_price,
        "availability": availability,
    }

The function looks for the price in two places

json_ld: most large stores embed structured product data in their pages so Google can show prices in search results. The function finds the item marked "@type": "Product" and reads the price, currency and stock status from its offers. This is the most reliable source, because it’s built for machines to read.
price_offers: if the structured data has no price, the function falls back to the prices the Extract API found in the visible page content.

4. Handle failed pages and extract successful product pages

When you scrape many pages, a few may fail: a timeout, a removed product, a blocked request. Handle them like below:

def error_detail(error):
    if isinstance(error, serpapi.HTTPError):
        return f"HTTP {error.status_code}: {error.error}"
    return str(error) or "request timed out"


def extract_product(client, product):
    try:
        data = client.search({"engine": "extract", "url": product["link"]})
        return extract_prices(data, product)
    except (serpapi.HTTPError, serpapi.TimeoutError, RuntimeError) as error:
        print(f"Skipping {product['link']}: {error_detail(error)}", file=sys.stderr)
        return None

extract_product() fetches each product page with one Extract API call per product. If anything goes wrong, it prints which URL failed and returns None, so the main loop can skip it.

5. Run the ecommerce scraper

Run the scraper: python best_buy_scraper.py

Here’s the output:

You can each product's title, url, price, and availability, ready to save to a file, load into a spreadsheet or feed into a price tracker. Run it on a schedule and compare each day’s results with the last, and you have a basic competitor price and stock monitor.

Key takeaways

Ecommerce data scraping is now the cost of staying competitive. Dynamic pricing is a baseline, catalogs adapt to customer demand, and shoppers have plenty of alternatives, so working from stale competitor data costs you sales.

The hard part isn’t writing the scraper, it’s the infra to maintain. Ecommerce sites are among the hardest to scrape, handling proxies, blocks and parsers that break whenever a page changes cost you engineering overload and resource drains.

As the code in this guide shows, you send a simple request to SerpApi and get structured data back into your pipeline, with none of that infrastructure to maintain. It includes 250 free searches a month, so try it out on your competitor site.

How to Scrape Booking.com Hotel Listings and Prices (2026 Tutorial)

Noraina Nordin — Tue, 27 Oct 2026 02:42:49 +0000

Say you want to track what a three-night stay in New York costs, or compare how hotels and vacation rentals are priced across a city. On Booking.com, you'd type a destination, pick your dates, and scroll. Manually extracting this data is slow and difficult to scale, especially across multiple destinations and date ranges. Replicating this process in code usually requires building and maintaining your own Booking.com scraper: rendering JavaScript pages, getting past anti-bot checks, fixing your parser when the HTML changes, and dealing with results that Booking.com ranks differently for every visitor.

SerpApi’s Booking.com Search API handles that for you and returns the results as structured JSON or Markdown. The same way you would do on manual search, you can search for a destination based on your travel dates, number of guests, and preferred currency. You’ll receive a list of properties with prices, review scores, room options, policies, and the filters Booking.com offers for that search. This data can be used to build a price tracker, market research dashboard, or travel assistant.

Booking.com interface

In this tutorial, you'll make your first request with cURL, Python, and JavaScript, then save the results to CSV, page through more properties, and filter by price and amenities.

What data can you retrieve with the Booking.com Search API?

Each search returns a properties array with the places to stay that Booking.com shows for your destination and dates. The most useful data includes:

Property details: name, type (such as Hotel or Apartment), and a link to the listing on Booking.com.
Prices: price for the whole stay, price_per_night, taxes_and_fees that aren’t included in the price, and the currency they’re in.
Reviews: the review score in rating (out of 10) and the number of reviews.
Location: location (district and city) and gps_coordinates with latitude and longitude.
Policies: whether the property offers free_cancellation, plus the rooms recommended for your guests in rooms.
Filters: the filters array lists the filters Booking.com offers for your search, which you can reuse in your next request (more on that below).

Prices depend on dates. If you don't set check_in_date and check_out_date, Booking.com returns properties without prices.

SerpApi's Booking.com Search API documentation

For the full list of fields and parameters, see the Booking.com Search API documentation.

How to scrape Booking.com hotel listings

You can call the Booking.com Search API from the playground, a plain GET request, cURL, Python, or JavaScript. Every method returns the same results.

Try it in the playground

The quickest way to explore the API is the SerpApi interactive playground. This example searches New York for a three-night stay:

engine: booking
q: New York
check_in_date: 2026-11-22
check_out_date: 2026-11-25

SerpApi's Booking.com search playground

Run the search and look at the properties array: compare price with price_per_night and taxes_and_fees, open the rooms of a few properties, and scroll down to filters to see which neighborhoods, property types, and amenities you can filter by.

💡 Swap in your own destination and future dates as you experiment. The q parameter accepts a city, region, district, landmark, or airport (such as Paris, Tuscany, Manhattan, or Eiffel Tower). However, Booking.com doesn't return results for most street addresses.

Get your API key

To call the API from your own code, register for a SerpApi account and copy your private API key from the API key page in your dashboard.

The examples below read the key from an environment variable. Set it in your terminal, replacing the placeholder with your real key:

export SERPAPI_API_KEY="YOUR_SERPAPI_API_KEY"

You should store your API key in a safe location if you're sharing or publishing your code. If it's ever leaked, you can generate a new one from the dashboard.

Make a GET request

The API endpoint is https://serpapi.com/search with engine=booking. Here's the same New York search as a plain GET request. JSON is the default output:

https://serpapi.com/search?engine=booking&q=New+York&check_in_date=2026-11-22&check_out_date=2026-11-25&api_key=YOUR_SERPAPI_API_KEY

To get Markdown instead, add output=md:

https://serpapi.com/search?engine=booking&q=New+York&check_in_date=2026-11-22&check_out_date=2026-11-25&api_key=YOUR_SERPAPI_API_KEY&output=md

Markdown output is a token-efficient format with tables, Markdown links, and YAML frontmatter, which works well when you pass results to an LLM or AI agent. Use JSON when your code needs exact field access, such as reading extracted_price or gps_coordinates.latitude. The rest of this tutorial uses JSON.

cURL implementation

Here's the equivalent request in cURL, using the environment variable you set earlier:

curl --get https://serpapi.com/search \
 -d engine="booking" \
 -d q="New+York" \
 -d check_in_date="2026-11-22" \
 -d check_out_date="2026-11-25" \
 -d api_key="YOUR_SERPAPI_API_KEY"

Here's what each parameter does:

engine=booking selects the Booking.com Search API.
q is the destination to search. It's required.
check_in_date and check_out_date set the stay in YYYY-MM-DD format. They're optional, but if you use them, they must be used together to get prices. The check-out date can be up to 90 nights after check-in.
api_key is your private SerpApi key. It's required.

By default, the search is for 2 adults in 1 room. You can change that with adults, children, children_ages, and rooms, all documented in the Booking.com Search API documentation.

Python implementation

We'll use the requests library. Install it first:

pip install requests

Create a file named main.py:

import os
import sys

import requests

SERPAPI_API_KEY = os.environ.get("SERPAPI_API_KEY")

if not SERPAPI_API_KEY:
    sys.exit("Set the SERPAPI_API_KEY environment variable first.")

params = {
    "engine": "booking",
    "q": "New York",
    "check_in_date": "2026-11-22",
    "check_out_date": "2026-11-25",
    "api_key": SERPAPI_API_KEY,
}


def fetch_properties(params):
    try:
        response = requests.get("https://serpapi.com/search", params=params, timeout=60)
    except requests.RequestException as error:
        # Only print the error type: the full message can contain the request URL and API key.
        sys.exit(f"Request failed: {type(error).__name__}")

    try:
        data = response.json()
    except ValueError:
        sys.exit(f"Unexpected response (HTTP {response.status_code}).")

    if response.status_code != 200 or data.get("error"):
        sys.exit(f"SerpApi error (HTTP {response.status_code}): {data.get('error', 'Unknown error')}")

    return data


data = fetch_properties(params)
properties = data.get("properties", [])

if not properties:
    print("The search succeeded, but no properties were returned.")
    sys.exit()

info = data.get("search_information", {})
print(f"Destination: {info.get('destination')} ({info.get('total_results', 'unknown')} properties found)\n")

for prop in properties[:5]:
    print(f"{prop.get('position')}. {prop.get('name')} ({prop.get('type')})")
    print(f"   Location: {prop.get('location')}")
    print(f"   Review score: {prop.get('rating', 'n/a')} {prop.get('rating_text', '')} from {prop.get('reviews', 0)} reviews")
    print(f"   Stay price: {prop.get('price', 'n/a')} ({prop.get('price_per_night', 'n/a')} per night)")
    print(f"   Free cancellation: {prop.get('free_cancellation', False)}\n")

Run it:

python main.py

The script prints the destination and the first five properties. A few notes on what you're seeing:

price is the price of the whole stay. By default, it's in each property's currency, which is returned in currency.
price_per_night is Booking.com's average nightly rate. It can exclude charges included in price, so it isn't necessarily the stay price divided by the number of nights.
rating is the review score out of 10, and reviews is the number of reviews behind it.

First five properties found in New York

The script uses .get() for each field because optional fields such as price, rating, or deals aren't returned for every property. It also keeps three outcomes separate: a failed request or API error stops the script with a message, a successful search with no properties says so, and a successful search prints results. If a search fails, SerpApi returns a message in the error field.

Export results to CSV

Property listings fit nicely into rows. To save them, add import csv to the imports at the top of main.py, then add this function below fetch_properties:

def save_csv(properties, filename="booking_properties.csv"):
    fieldnames = [
        "position", "property_id", "name", "type", "rating", "reviews",
        "price", "extracted_price", "currency", "free_cancellation",
        "latitude", "longitude", "room_name", "deals", "link",
    ]
    with open(filename, "w", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(file, fieldnames=fieldnames)
        writer.writeheader()
        for prop in properties:
            gps = prop.get("gps_coordinates", {})
            rooms = prop.get("rooms", [])
            writer.writerow({
                "position": prop.get("position"),
                "property_id": prop.get("property_id"),
                "name": prop.get("name"),
                "type": prop.get("type"),
                "rating": prop.get("rating"),
                "reviews": prop.get("reviews"),
                "price": prop.get("price"),
                "extracted_price": prop.get("extracted_price"),
                "currency": prop.get("currency"),
                "free_cancellation": prop.get("free_cancellation"),
                "latitude": gps.get("latitude"),
                "longitude": gps.get("longitude"),
                # Keep only the first recommended room and join deals into one cell.
                "room_name": rooms[0].get("name") if rooms else None,
                "deals": "; ".join(prop.get("deals", [])),
                "link": prop.get("link"),
            })
    print(f"Saved {len(properties)} properties to {filename}")

Finally, add this line at the end of the file, after the for loop:

save_csv(properties)

Run python main.py again to create booking_properties.csv. The nested gps_coordinates object becomes two columns, the deals array is joined into one cell, and only the first room from the rooms array is kept. Use extracted_price instead of price when you want to sort or compare numbers in a spreadsheet.

Properties found in New York in CSV format

JavaScript implementation

For Node.js, we'll use SerpApi's official JavaScript library. Install it:

npm install serpapi

Create a file named main.cjs. The .cjs extension tells Node.js to treat it as a CommonJS module, so require works without extra configuration:

const { getJson } = require("serpapi");

const apiKey = process.env.SERPAPI_API_KEY;

if (!apiKey) {
  console.error("Set the SERPAPI_API_KEY environment variable first.");
  process.exit(1);
}

async function main() {
  let data;
  try {
    data = await getJson({
      engine: "booking",
      q: "New York",
      check_in_date: "2026-11-22",
      check_out_date: "2026-11-25",
      api_key: apiKey,
    });
  } catch (error) {
    // Redact the key in case the error message includes the request URL.
    console.error("Request failed:", String(error.message).replaceAll(apiKey, "[REDACTED]"));
    process.exit(1);
  }

  if (data.error) {
    console.error("SerpApi error:", data.error);
    process.exit(1);
  }

  const properties = data.properties || [];
  if (properties.length === 0) {
    console.log("The search succeeded, but no properties were returned.");
    return;
  }

  const info = data.search_information || {};
  console.log(`Destination: ${info.destination} (${info.total_results ?? "unknown"} properties found)\n`);

  for (const property of properties.slice(0, 5)) {
    console.log(`${property.position}. ${property.name} (${property.type})`);
    console.log(`   Location: ${property.location}`);
    console.log(`   Review score: ${property.rating ?? "n/a"} ${property.rating_text ?? ""} from ${property.reviews ?? 0} reviews`);
    console.log(`   Stay price: ${property.price ?? "n/a"} (${property.price_per_night ?? "n/a"} per night)`);
    console.log(`   Free cancellation: ${property.free_cancellation ?? false}\n`);
  }
}

main();

Run it:

node main.cjs

You'll see the same fields as the Python version. If you’d rather pass results straight to an LLM, use the output=md GET request shown earlier.

Other languages and no-code options

SerpApi has libraries for more programming languages, listed on the integrations page. Since the Booking.com Search API is a regular GET request with engine=booking, any HTTP client also works.

If you prefer no-code tools:

SerpApi's Make.com app has native modules for some APIs and a Universal "Make an API Call" module where you set the request path and each parameter yourself. Pagination isn't automated in the Universal module.
SerpApi's verified n8n node launched with 22 popular APIs. For APIs that aren't in the node, the announcement shows how to use n8n's HTTP Request node with the engine parameter, for example engine=booking.

How to paginate the results

A search returns 25 properties by default. Here's how to get more:

offset skips a number of results. With the default page size, offset=25 is page 2 and offset=50 is page 3. It can go up to 1,000; beyond that, Booking.com returns the same properties.
limit sets the number of properties per page, from 1 to 100. Booking.com sometimes returns a few extra properties, such as sponsored ones.
session_token keeps the same Booking.com visitor between requests. Booking.com ranks properties differently for each visitor, so without it, later pages can repeat properties from earlier ones. It's returned in serpapi_pagination.session_token.
serpapi_pagination.next is a ready-made link to the next page. It already includes the next offset, the session_token, and your original parameters.

For example, the second page of the New York search looks like this, where SESSION_TOKEN_FROM_PAGE_1 is the serpapi_pagination.session_token value from the first response:

https://serpapi.com/search?engine=booking&q=New+York&check_in_date=2026-11-22&check_out_date=2026-11-25&offset=25&session_token=SESSION_TOKEN_FROM_PAGE_1&api_key=YOUR_SERPAPI_API_KEY

In code, the simplest approach is to follow the next link and add your API key to it. This paginate.py script fetches up to three pages, stops early when there’s no next page or a page comes back empty, and saves all properties to a JSON file:

import json
import os
import sys

import requests

SERPAPI_API_KEY = os.environ.get("SERPAPI_API_KEY")

if not SERPAPI_API_KEY:
    sys.exit("Set the SERPAPI_API_KEY environment variable first.")

MAX_PAGES = 3  # Each page is a separate search request.

params = {
    "engine": "booking",
    "q": "New York",
    "check_in_date": "2026-11-22",
    "check_out_date": "2026-11-25",
    "api_key": SERPAPI_API_KEY,
}

url = "https://serpapi.com/search"
all_properties = []

for page in range(1, MAX_PAGES + 1):
    try:
        response = requests.get(url, params=params, timeout=60)
    except requests.RequestException as error:
        sys.exit(f"Request for page {page} failed: {type(error).__name__}")

    try:
        data = response.json()
    except ValueError:
        sys.exit(f"Unexpected response on page {page} (HTTP {response.status_code}).")

    if response.status_code != 200 or data.get("error"):
        sys.exit(f"SerpApi error on page {page} (HTTP {response.status_code}): {data.get('error', 'Unknown error')}")

    properties = data.get("properties", [])
    all_properties.extend(properties)
    print(f"Page {page}: {len(properties)} properties")

    next_link = data.get("serpapi_pagination", {}).get("next")
    if not properties or not next_link:
        break

    # The next link already carries offset, session_token, and the original filters.
    url = next_link
    params = {"api_key": SERPAPI_API_KEY}

with open("booking_properties.json", "w", encoding="utf-8") as file:
    json.dump(all_properties, file, indent=2, ensure_ascii=False)

print(f"Saved {len(all_properties)} properties to booking_properties.json")

Run it with python paginate.py. Each page is a separate search and can use a search credit, so keep MAX_PAGES small while you're testing. If you need more properties per request, raise limit (up to 100) instead of fetching many small pages.

Customize or filter the results

The Booking.com Search API supports the sorting and filters you'd use on Booking.com itself. Here are three useful ones:

sort_by: Changes the order of results. The default is popularity ("Our top picks"). Other options include price (lowest first), bayesian_review_score (top reviewed), review_score_and_price, and distance_from_search (distance from downtown).
min_price and max_price: Limit the price per night, in the currency of the results. They must be used together with a currency code, such as USD or EUR. See the Booking.com currencies page for supported codes.
nflt: Applies any other Booking.com filter, such as amenities, meals, neighborhoods, or brands. It's a semicolon-separated list of filter values. You don't need to guess them: take the value from filters[].options in a previous response. For example, the documentation uses di=1127 for London's Theatreland neighborhood.

Here's a New York search for stays between 100 and 300 USD per night, sorted by lowest price, limited to properties with a filter value taken from the filters array:

https://serpapi.com/search?engine=booking&q=New+York&check_in_date=2026-11-22&check_out_date=2026-11-25&currency=USD&min_price=100&max_price=300&sort_by=price&nflt=roomfacility%3D38&api_key=YOUR_SERPAPI_API_KEY

In this example, roomfacility=38 is the "Private bathroom" option from the documentation's New York sample. Note that nflt values need URL encoding (= becomes %3D and ; becomes %3B); requests and the JavaScript library handle that for you.

Other handy filters include free_cancellation=true, rating for a minimum review score (9, 8, 7, or 6), hotel_class for star ratings (for example, 4,5), and property_types (for example, 201,220 for apartments and vacation homes). You can also localize results with hl for the language and currency for prices, or set device=mobile to get Booking.com's mobile results, which can include mobile-only prices. See the Booking.com Search API documentation for the full list.

Conclusion

You can now scrape Booking.com hotel listings for any destination and dates, read their prices, review scores, rooms, and policies, save them to CSV, page through more results with session_token, and narrow searches with sorting and filters. You can also switch to Markdown output with output=md when the results are going to an LLM.

To compare Booking.com prices with other sources, see how to scrape Google Hotels data, scrape Airbnb data, and scrape Tripadvisor.

Frequently asked questions

Which parameters are required?

Only engine=booking, q (the destination), and api_key. Everything else is optional, but you'll usually want check_in_date and check_out_date, because Booking.com returns properties without prices when dates aren't set.

Why are prices in different currencies?

By default, currency is hotel_currency, so each property's price is in its own currency (for example, EUR in Paris). Set currency to a code such as USD to get all prices in one currency. The currency field on each property tells you which one was used.

Can I search flexible dates?

Yes. Set flex_window to 1, 2, 3, or 7 to let the check-in and check-out dates move up to that many days earlier or later for the best price of each property. Each property then includes the check_in_date and check_out_date of its price. flex_window must be used with both dates.

Is this the official Booking.com API?

No. The Booking.com Search API is a SerpApi API that returns Booking.com's public search results (the same properties, prices, and filters you see on Booking.com) as structured data. It's a separate product from the APIs Booking.com offers its own partners, and you only need a SerpApi API key to use it.

Does the Booking.com Search API support Markdown output?

Yes. Add output=md to get Markdown, which is useful for LLMs and AI agents. You can also use the /search.md endpoint or the Accept: text/markdown request header. See the Markdown output page for details.

From JSON to Markdown: Reducing AI SEO Analysis Token Costs by 90%

Magenta Qin — Sun, 25 Oct 2026 10:33:25 +0000

AI SEO analysis pipelines usually run into two problems quickly.

First, the more data you analyze, the more input tokens you send to the model. Adding more keywords, markets, competitors, pages, or result rows makes every prompt longer. That may not matter for a one-off analysis, but it matters when a pipeline runs thousands of times a day.

Second, cutting the input down can make the analysis less useful. When the full dataset does not fit into the model context, you might often split it into batches, summarize each batch, or rely on agent memory. Those techniques help, but they can not replace the original data. If positions 1-10 and positions 21-30 land in different batches, the model may never compare them side by side.

The core question is simple: Can we help the model see more SEO evidence in the same token budget?

In this post, I will compare JSON and Markdown as API response formats. I will first reproduce a token benchmark for Google Shopping results. Then I will run a practical SEO analysis demo to see whether the larger sample changes the model's recommendation. The demo code is available here.

Why build an AI SEO analysis pipeline?

Existing SEO tools already provide rich datasets, dashboards, reports, and AI features. For many workflows, that is enough.

But internal SEO questions often depend on business context that a general-purpose SEO platform does not have. For example:

Why did a competitor's product enter the top results for this query?
Which ranking changes should the team handle today?
Does this search result change matter for revenue, conversion rate, inventory, or a product launch?

When you build your own AI SEO analysis pipeline, you can choose the market, device, timestamp, query set, and data source. You can also combine search data with internal business data before asking the model to reason over it.

The goal is not to rebuild a full SEO platform. The goal is to add a decision layer that matches your company's workflow.

The hidden cost of API response format

Most APIs return JSON by default. JSON is excellent for code. It gives programs exact fields, predictable structure, and a reliable contract. But when the next consumer is an AI model, the response format also affects context usage. Consider a simplified Google Shopping result in JSON:

{
  "shopping_results": [
    {
      "position": 1,
      "title": "Trail Runner X",
      "price": "$89",
      "rating": 4.7
    },
    {
      "position": 2,
      "title": "Aero Sprint",
      "price": "$74",
      "rating": 4.5
    }
  ]
}

The same information can be represented as a Markdown table:

## Shopping Results

| Position | Title | Price | Rating |
|---:|---|---:|---:|
| 1 | Trail Runner X | $89 | 4.7 |
| 2 | Aero Sprint | $74 | 4.5 |

The JSON response repeats keys such as position, title, price, and rating for every item. It also includes quotes, commas, braces, and nesting. The Markdown table keeps the field names in the header and uses rows for the values. Both formats describe the same result set, but Markdown often uses fewer tokens. That means the same model context can fit more search results before batching becomes necessary.

This is why Markdown is moving beyond documentation. In April 2025, Cloudflare added a Browser Rendering API endpoint that converts webpages into Markdown. In February 2026, Notion documents page content access as Markdown in its data APIs. In August 2026, SerpApi supports Markdown output for search results by using output=md.

The shared pattern is clear: API responses are no longer consumed only by programs. They may go directly into an AI agent's context.

Benchmark: How many input tokens can Markdown save?

SerpApi published a Markdown output benchmark comparing JSON and Markdown responses across several APIs. The result depends on the API, query, returned fields, and tokenizer, but the largest savings appear in more complex result sets.

In that benchmark:

These numbers compare input tokens only. If a model provider charges the same price per input token, reducing input tokens reduces that part of the cost. The total cost of a full AI pipeline will still depend on model choice, output length, orchestration, retries, caching, and other API calls. I wanted to reproduce the Google Shopping result with my own code.

Reproducing the Google Shopping token benchmark

The reproduction changes only one variable: response format. I prepared 12 Google Shopping queries. For each query, I used the same location, language, and device settings. Then I requested the same Google Shopping results twice through SerpApi: once as JSON and once as Markdown.

I sent the raw API responses to the same DeepSeek model and read usage.prompt_tokens from the DeepSeek API response. That value reports the actual number of input tokens the model received. I disabled thinking mode because this experiment measures input representation, not reasoning quality. I did not ask DeepSeek to solve an SEO problem in this benchmark. I used it as the token accounting endpoint.

The calculation was:

saving = 100 * (1 - markdown_tokens / json_tokens)

Let's execute it:

0:00
/0:18
1×

Across the 12 queries, Markdown reduced input tokens by an average of 89.55%. Individual queries ranged from 86.59% to 91.20%. That closely matches the published Google Shopping benchmark of about 90%.

The run saves intermediate evidence so the result can be checked later:

Output	Purpose
`raw/.json` and `raw/.md`	Original API responses
`deepseek-usage/*.json`	DeepSeek token usage receipts
`results.csv` and `results.json`	Per-query comparison data
`summary.md`	Final summary table

For this sample, Markdown compressed the Google Shopping inputs to roughly one-tenth of the JSON token usage.

The next question is more important for AI SEO: if Markdown lets the model see more results in the same budget, can that change the analysis?

A practical SEO demo: Trail running shoes

For the second experiment, I used a practical Google Shopping question: I sell trail running shoes. To show up higher in Google Shopping, should I optimize titles, lower prices, or get more reviews?

I fixed the search setup:

Setting	Value
Query	`trail running shoes`
Market	New York, United States
Language	English
Device	Desktop
Fields kept	position, title, price, rating, reviews, source
Input budget	1,550 prompt tokens
Model	Same DeepSeek model and prompt for both formats

SerpApi returned the same Google Shopping snapshot with 30 products. Python then measured how many products could fit into the 1550-token budget for each format.

Let's run it:

0:00
/0:15
1×

The result:

Format	Products that fit	Prompt tokens used	Position coverage
JSON	16	1,548	1-16
Markdown	30	1,515	1-30

With the same token budget, Markdown let DeepSeek see 1.88x more products.

The larger sample changed the recommendation

The JSON sample only covered positions 1-16. In that sample, the average query-term coverage in product titles was:

Sample	Query-term coverage
Top positions 1-5	27%
Lower positions 12-16	20%

That difference was small. DeepSeek treated the title signal as weak and did not see titles as a clear explanation for position.

The Markdown sample covered all 30 products. With the full sample, the title pattern became clearer:

Sample	Query-term coverage
Top positions 1-10	47%
Lower positions 21-30	17%

With the additional 14 products, DeepSeek gave a more specific recommendation: test title changes for lower-position products by adding terms such as "trail running" and gender-specific modifiers. Price and review count did not show meaningful rank relationships in this snapshot:

Metric	JSON sample	Markdown sample
Spearman position vs. price	0.001	0.155
Spearman position vs. reviews	0.118	-0.031

So the supported next step was not to cut prices or chase more reviews for ranking. The better next step was to run a title A/B test.

This does not prove that title changes cause higher Google Shopping positions. A single search snapshot can only generate hypotheses. But it does show how a compact response format can expose a signal that a smaller sample hides.

Markdown did not make the model smarter. It gave the model more original evidence to compare in the same context budget.

JSON and Markdown both have a role

This is not a case for replacing JSON everywhere. JSON remains the better format when code needs a strict contract: tool arguments, IDs, status fields, saved state, validation, and orchestration logic. Markdown is useful when the next step is model understanding. If an AI agent needs to read, compare, summarize, or reason over search results, Markdown can preserve a clear structure while reducing token overhead.

A practical agent workflow can use both:

Role	Better fit
Control data	JSON
Tool arguments	JSON
IDs and state	JSON
Search result context for the model	Markdown
Documents, citations, and comparison tables	Markdown

What I learned

This experiment led to three takeaways.

First, building an AI SEO pipeline gives teams control over what the model sees and how recommendations get tested. The value comes from connecting search data with the business question, not from rebuilding a full SEO platform.

Second, Markdown can reduce input token usage substantially for search result payloads. In our Google Shopping benchmark reproduction, Markdown reduced input tokens by 89.55% on average.

Third, a more compact input format can change the model's answer by increasing evidence coverage. In the Shopping demo, JSON fit 16 products while Markdown fit all 30. The larger sample changed the title signal and led DeepSeek to recommend testing title relevance first.

If you are building an AI agent or AI SEO pipeline, test both formats in one real workflow. Compare:

Actual input tokens reported by the model API
How much source data fits into the prompt
Whether the larger context changes the final recommendation

File size alone is not enough. What matters is how much useful evidence the model can see when it makes the decision.

Try Markdown output with SerpApi

SerpApi supports Markdown responses for search APIs with output=md. If your workflow sends search results into an LLM or AI agent, this can reduce formatting overhead and help the model analyze more results in one context.

You can try SerpApi with a free account, which includes 250 searches per month. If you have questions or feedback about Markdown responses, contact us at contact@serpapi.com.

AI #SEO #markdown

RubyLLM 2.x: What's new for your agents

Josef Strzibny — Sat, 24 Oct 2026 10:24:10 +0000

RubyLLM 2.0 and 2.1 are out and come with new APIs for video, speech, OCR, reranking, batches, conversation controls, expanded list of supported providers and even better Rails integration. But that's not all. If you build chat-based agents in Rails, tools can now wait for a human to approve them, you can run the agent loop one step at a time, and the agents can live in the database and survive a restart. So let's have a look on how to upgrade our old agents and give them new capabilities.

💡 This post is mainly about RubyLLM 2.0 and 2.1 features for agents. For the full list, see also the official What's New in 2.0 and 2.1 guides.

A quick look on the changes

The short version of both releases include the following main additions:

Tool approvals. Mark a tool with requires_approval and the agent pauses until a human says yes or no.
Caller-controlled loop. ask_later, generate, run_tools, and step let you run the agent loop one piece at a time.
Durable agents. Agents can be backed by your Chat model and resumed across requests, jobs, and restarts.
Provider-hosted tools. Web search, code execution, and remote MCP servers through one provider_tools API, with citations.
Batches. Submit many chats at once at the provider's batch rates and collect results later.
Fallbacks, prompt caching, and cost tracking. with_fallbacks, with_caching, and normalized tokens and cost on every response.
MCP client. Connect agents to Linear, GitHub, Notion, or any MCP server with a class, the same way you describe a tool.
Tool progress. Tools report what they're doing while they run, and your UI can show it.
Tool search. Mark tools as deferred and the model searches for the ones it needs instead of receiving every definition.
Evaluations. Run your agent against a dataset of reference answers and get a graded report.
Cache lifetimes per boundary. cache_until_here(ttl: "1h") for the prefix that never changes.
Faster by default. Streaming parses incrementally, replies stop retaining requests, and connections can be reused.

New agent capabilities

Tool approvals

One of the biggest additions to RubyLLM is the ability to approve tool calls. The obvious problem is that nobody wants a language model repricing the catalog on its own. RubyLLM 2.0 solves this with requires_approval:

class UpdatePrice < RubyLLM::Tool
  description "Updates the price of a product in our catalog"
  requires_approval

  param :sku, desc: "Product SKU"
  param :price, desc: "New price in USD"

  def execute(sku:, price:)
    product = Product.find_by!(sku: sku)
    product.update!(price: price)
    "Updated #{product.name} to $#{price}"
  end
end

Add it to the agent's tools and ask for a change:

class PriceMonitorAgent < RubyLLM::Agent
  # ...
  tools LookupProduct, SearchGoogleShopping, UpdatePrice
end

agent = PriceMonitorAgent.new
agent.ask "Lower the Chemex 6-Cup to match the market average."
agent.awaiting_approval? # => true

The agent still runs LookupProduct and SearchGoogleShopping on its own. When it gets to UpdatePrice, it stops. The tool call stays pending and ask returns. You can inspect what the model wants to do before anything touches the database:

agent.pending_approvals.each do |call|
  puts "#{call.name} #{call.arguments.inspect}"
end
# update_price {"sku"=>"CHX-6", "price"=>44.97}

Then approve or deny and let the agent finish:

agent.approve(agent.pending_approvals.first)
agent.complete

deny works the same way. The model gets told its request was rejected and can respond accordingly, usually by explaining what it would have done.

This is the single most useful change in 2.0 for anyone running agents against production data. Read-only tools backed by SerpApi can run freely. Tools that adjust live prices might need a human in the loop, and now that's possible.

Controlling the agent loop

In RubyLLM 1.x, ask ran the whole thing: send the message, run tools, send results back, repeat until the model answers. That's convenient until you want to be more specific and choose the number of rounds, log every step, or run each turn in a separate background job.

RubyLLM 2.0 breaks the loop into 4 parts:

ask_later stages the question without sending it.
generate asks the model for one response.
run_tools executes any pending tool calls.
step advances by one generation or one tool execution, whichever is next.

So the previous loop can be run one step at the time:

agent = PriceMonitorAgent.new
agent.ask_later "Which of our coffee products are priced more than 15% above market?"

agent.generate  # model decides to call LookupProduct and SearchGoogleShopping
agent.run_tools # the SerpApi searches happen here
agent.generate  # model reads the results and answers

Nothing new happens compared to ask, but you can now put a counter, a timeout, or a job boundary between any two lines. For a pricing agent that might run dozens of Google Shopping searches, being able to say "stop after ten tool rounds" is a feature, not a limitation.

Rails durable agents

Previously we could persist chats with acts_as_chat. Now we can go further and lets the agent itself be the record. Point the agent at your chat model with chat_model:

class PriceMonitorAgent < RubyLLM::Agent
  chat_model Chat
  model "gpt-5.4-mini"

  instructions <<~PROMPT
    You are a pricing analyst for the merchandising team.
    Always show specific numbers when comparing prices.
  PROMPT

  tools LookupProduct, SearchGoogleShopping, UpdatePrice
end

Then use create! instead of new:

chat = PriceMonitorAgent.create!
chat.ask "Lower the Chemex 6-Cup to match the market average."
chat.awaiting_approval? # => true

This is where approvals get practical. The request that triggered the agent has long returned. The merchandiser sees the proposed price change in the UI, maybe hours later, and clicks approve. Your controller or job just reloads the agent and continues:

chat = PriceMonitorAgent.find(params[:chat_id])
chat.approve(params[:tool_call_id])
chat.complete

The transcript, the pending tool call, and the decision all live in your database. If a job dies halfway, you can resume from the last saved step.

💡 One caveat from the docs: an interrupted operation may run again, so tools should tolerate retries. UpdatePrice is fine since setting a price twice is harmless. A tool that sends an email should check first.

Your app still owns the chats and messages tables. RubyLLM 2.0 takes ownership of the supporting tables for the model registry, tool calls, usage, and batches, so it can evolve them without you maintaining the models.

Provider tools

The next useful addition is a shared API for tools hosted by the provider. Things like web search, code execution, and remote MCP servers. You enable them with with_provider_tools on a chat or provider_tools on an agent, and the results come back with citations:

class PriceMonitorAgent < RubyLLM::Agent
  model "claude-sonnet-5"
  provider_tools :web_search
  tools LookupProduct, SearchGoogleShopping, UpdatePrice
end

response = PriceMonitorAgent.new.ask "Any recent news about Chemex pricing?"
response.citations.each { |c| puts "#{c.title} - #{c.url}" }

Provider web search is great starting point for open-ended questions. The tool returns answers with links.

Provider web search vs. SerpApi tools

Previously, I showed you how to build RubyLLM agents with the examples of wrapping SerpApi SDK to give your agents access to 100+ search engines. While it's now easier than ever to just use the default search, a SerpApi-backed tool gives you structured data from a very specific engine.

For example, SerpApi's Google Shopping API always returns a title, a price, and a source every single time. You pick the engine, so swapping to Amazon, Walmart, or eBay is a one-line change. You control the parameters, so you can pin the location or currency. And you can see every query the agent ran in the search inspector.

This gives you accuracy and flexibility the default search cannot.

Batch processing

Checking one product price in a chat is one thing. But what about checking the whole catalog every night? RubyLLM now adds batch processing on top of ask_later which lets you use providers batch rates for more affordable queries.

An example of this could be submitting a chat question in bulk:

chats = Product.where(category: "coffee").map do |product|
  market = SerpApi.search(q: product.name, engine: "google_shopping")
                  .shopping_results.first(5)
                  .map { |r| "#{r[:source]}: #{r[:price]}" }
                  .join(", ")

  RubyLLM.chat(model: "gpt-5.4-mini")
    .with_instructions("You are a pricing analyst. Answer in one sentence.")
    .ask_later("Our #{product.name} is $#{product.price}. Market: #{market}. " \
               "Are we more than 15% above market?")
end

batch = RubyLLM.batch(chats)

Save batch.id somewhere and another background job can check on it later:

batch = RubyLLM::Batch.find(batch_id, provider: :openai)

if batch.complete?
  batch.messages.each { |message| puts message.content }
end

Results come back in submission order, so zipping them with the products list is trivial. The Batches guide covers failures, embedding batches, and how tool calls behave inside a batch if you'd rather let the model call the tool itself.

Fallbacks, caching, and cost

Finally let's go over three smaller (but neat) additions that matter once the agent runs on a schedule rather than in a console.

Fallbacks. When the primary model fails with a transient error, try another one:

chat = RubyLLM.chat(model: "gpt-5.4-mini")
             .with_tools(LookupProduct, SearchGoogleShopping)
             .with_fallbacks("claude-sonnet-5")

The conversation keeps its tools and settings, so pick a fallback that supports what you use. You need credentials for both providers.

Prompt caching. Our agent sends the same instructions on every request. Mark them as a cacheable prefix:

chat = RubyLLM.chat(model: "claude-sonnet-5").with_caching
chat.with_instructions(File.read("pricing-policy.txt")).cache_until_here

response = chat.ask "Is the Aeropress competitively priced?"
response.tokens.cache_read

Providers still decide the minimum prefix length and lifetime, and a cache hit isn't guaranteed. But for a long pricing policy that doesn't change, it adds up.

Cost tracking. Every response now reports normalized tokens and cost, including the failed attempts behind a fallback:

response.tokens.input
response.tokens.output
response.cost.total
chat.cost.total

Unknown pricing stays nil rather than showing as zero. In Rails, usage is recorded in its own ledger, so changing model prices later doesn't rewrite history. If you've ever been asked "how much does this agent cost us per day?", this is the answer.

Upgrading from 1.x

Install or upgrade the gem with the following Bundler call:

bundle add ruby_llm --version 2.0.0

Applications on 1.x go to 2.0 first, then 2.1. From 2.1 on, each release ships the upgrade from the release before it:

bundle update ruby_llm --version 2.1.0

The 2.0 step is mostly renames, with one name per concept across the library. For tools, param became parameter, desc: became description:, and params do ... end became parameters do ... end.

A 1.x tool:

class LookupProduct < RubyLLM::Tool
  description "Looks up a product in our catalog by name"
  param :name, desc: "Product name to search for"

  def execute(name:)
    # ...
  end
end

Becomes:

class LookupProduct < RubyLLM::Tool
  description "Looks up a product in our catalog by name"
  parameter :name, description: "Product name to search for"

  def execute(name:)
    # ...
  end
end

The agent class itself needs no changes. Other renames you'll hit first:

max_tokens is now max_output_tokens.
with_params and provider-specific options go through with_provider_options.
response.input_tokens and friends live under response.tokens.
Callbacks follow Rails naming: on_tool_call is before_tool_call, on_end_message is after_message.
Tools can no longer halt the loop; use approvals or the caller-controlled loop instead.
Tool results that return a Hash or Array are sent as JSON text instead of Ruby's inspect output.

In Rails, version 2.0 moves the model registry, tool calls, usage, and batches into RubyLLM-owned tables. bin/rails generate ruby_llm:upgrade creates the migrations in phases, with cleanup in a later deploy, and an optional --mode copy keeps a route back to 1.16 while you verify. Then run the generator again on 2.1 to add the MCP credentials table and the new columns.

The official upgrade guide has the full list.

Also new

Here are some other things I skipped because they don't change how agents work, but which you might want to start using too:

RubyLLM.animate is now available for requesting video
RubyLLM.speak for speech
RubyLLM.transcribe for transcription with speaker labels and timestamps
RubyLLM.ocr for PDFs and scans
RubyLLM.rerank for search results
RubyLLM.research for provider-hosted deep research
RubyLLM.workflow for grouping steps in your instrumentation

Plus there are now four new providers too: Cohere, Deepgram, ElevenLabs, and Ollama Cloud.

Conclusion

RubyLLM 2.x makes agents safer to run against real data. Approvals keep humans in charge of writes, loop control lets you bound what an agent does, and durable agents mean a pending decision can wait in your database until someone makes it.

SerpApi fits in the same place it did before: as an agent tool that gives your agent structured, inspectable search data from Google Shopping, Amazon, Walmart, and other engines. Give it a try with 250 free searches a month.

How to Fit More Web Search Data into a Local LLM

Tomás Murúa — Tue, 20 Oct 2026 01:52:22 +0000

A local model can only use the web search results that fit in its context window. When we built the app for How to Connect Your Local LLM with Web Search Data, a single SerpApi response was often bigger than the whole context window available. So we trimmed every response with the JSON Restrictor, which keeps only the fields the model needs and drops the rest on SerpApi’s servers before it reaches the LLM.

SerpApi's new Markdown output changes that budget. The same data takes fewer tokens in Markdown than in JSON, so the restrictor can keep more of it. For most searches, the full response now fits with no restrictor at all.

This blog will show you how to apply the JSON Restrictor to Markdown output, and when the full response is worth the wait. We'll use the same open-source app that gives a local model running in LM Studio internet access. The app is built with TanStack Start, and we upgraded it to Start 1.0 along the way.

GitHub - serpapi/local-llm-web-search: Give a local LLM real-time web search via SerpApi function calling.

What’s New Since the First Blog

SerpApi's Markdown output returns curated results as a Markdown document instead of JSON. You add output=md to a request, and the results come back as headings, lists and tables without the JSON keys, quotes and braces. It works across SerpApi's100+ APIs, and a Markdown search costs the same as a JSON search. The launch blog Introducing SerpApi’s New Markdown Output covers the format and its token savings across our engines.

The app now lets you choose what the model reads after each search. The benchmark compares four outputs, and a model only ever reads three of them.

Output	What the model reads	Parameters
JSON (Raw)	The full JSON response. The benchmark measures it, but no model reads it	No restrictor
JSON (Restrictor)	JSON trimmed to the fields the model needs, as in the first blog	`json_restrictor`
Markdown (Restrictor)	Markdown with more results and more fields, at about the same size. This is the app's default	`json_restrictor` and `output=md`
Markdown (Raw)	The full Markdown document	`output=md`

The setup changed too. The first blog ran Qwen3.5-9B on a MacBook Pro M4 Max with 36 GB of memory. This time I ran four of LM Studio's staff picks on a MacBook Pro M5 Pro with 48 GB, using LM Link to connect both MacBooks. They are:

How We Tested

Every model ran with a 32K context window, the same size as in the first blog. LM Studio loads a model with an 8K window by default, which is too small once a chat carries search results. A 32K window is enough for a long conversation, and it fits a 27B model in memory on a 36 GB Mac.

What the Benchmark Measures

The app's Benchmark page measures two things, and this blog uses both.

Payload size: The same seven searches run once for each of the four outputs, with no model involved. There's one search per tool, coveringSearch,Finance,News,Maps,Flights,Hotels andShopping. This shows what each output costs before a model reads anything. These sizes come from OpenAI's tiktoken library with its cl100k_base encoding, which splits text into tokens. Each local model splits text its own way, so treat these sizes as estimates for comparing formats.

End to end: Thirteen questions run through the whole app on each model, once for each of the three outputs a model can read. Seven need live data from SerpApi, one search per tool. Two need two searches at once, such as comparing AAPL with TSLA. The other four need no search, such as "What is 15% of 240?", so they show how the model behaves on its own. We used it to compare how each model handles each output, how long it takes, and what it answers.

How Much Does Markdown Save?

To see what the format alone saves, we kept the restrictor strings from the first blog and sent the same seven searches as JSON and as Markdown. The seven results took 4,668 tokens in JSON and 3,412 in Markdown, 27% fewer. Here's how each tool changed:

Tool	JSON	Markdown	Change
Flights	823	439	−47%
Finance	521	324	−38%
News	678	422	−38%
Search	785	586	−25%
Hotels	567	433	−24%
Maps	497	449	−10%
Shopping	797	759	−5%
Total	4,668	3,412	−27%

The saving depends on how deeply the data is nested. A flight option holds legs, and each leg holds airports and times, so JSON repeats keys, quotes and braces at every level. Markdown turns lists of results into tables and writes each column name once. Shopping results are flat, with a title, a price and a store per product, so there is little to remove.

So instead of keeping that saving, we spend it. In Markdown, the same token budget can carry more fields and more results, so the Markdown output gets a looser restrictor.

Apply the JSON Restrictor to Markdown Output

There's no separate restrictor for Markdown. You send the same json_restrictor parameter together with output=md, and SerpApi filters the sections first, then renders what's left as Markdown.

In the app, each tool has two restrictor strings. JSON sends the original one, and Markdown sends a looser one with more results and more fields. The Markdown request is the same search with output=md added:

// Pick the restrictor for the output mode. Markdown (Raw) sends none.
function restrictorFor(name: RestrictorTool, output: ToolOutput): string | null {
  if (output === "markdown-full") return null
  return output === "markdown" ? MARKDOWN_RESTRICTORS[name] : RESTRICTORS[name]
}

// Same parameters as the JSON request, plus output=md.
async function serpApiMarkdown(params: Record<string, unknown>): Promise<string> {
  const url = new URL("https://serpapi.com/search")
  for (const [key, value] of Object.entries(params)) {
    if (value !== undefined && value !== null) url.searchParams.set(key, String(value))
  }
  url.searchParams.set("output", "md")
  url.searchParams.set("api_key", process.env.SERPAPI_API_KEY ?? "")

  const res = await fetch(url)
  return res.text()
}

Each tool spends the saving on different extras:

Tool	JSON (Restrictor) keeps	Markdown (Restrictor) keeps
Search	Answer box, AI Overview (6 blocks), 3 related questions, knowledge graph, 5 web results	5 web results, answer box with its lists, AI Overview (8 blocks), knowledge graph, 3 top stories, 3 related questions
Finance	Price, key stats, 3 news items	Price, key stats, company description, 5 news items
News	5 stories, 1 related article each	7 stories, 2 related articles each
Maps	5 places	6 places, with opening hours and a description
Flights	3 best and 3 other options	4 best and 4 other options, with aircraft, legroom, overnight layovers and price insights
Hotels	5 hotels	7 hotels, with a description and a location rating
Shopping	5 products, plus 3 inline results	7 products, plus 3 inline results, with second-hand condition

For example, here are both restrictor strings for Maps, from restrictors.ts:

// JSON (Restrictor): five places.
export const RESTRICTORS = {
  google_maps_search: [
    "local_results[0:5].{title,rating,reviews,type,address,price,open_state,phone,website,gps_coordinates}",
    "place_results.{title,rating,reviews,type,address,price,open_state,phone,website}",
    "error",
    "search_metadata.google_maps_url",
  ].join(","),
  // ...one restrictor per tool
}

// Markdown (Restrictor): six places, with opening hours and a description.
export const MARKDOWN_RESTRICTORS = {
  google_maps_search: [
    "local_results[0:6].{title,rating,reviews,type,address,price,open_state,hours,phone,website,description}",
    "place_results.{title,rating,reviews,type,address,price,open_state,hours,phone,website,description}",
    "error",
    "search_metadata.google_maps_url",
  ].join(","),
  // ...one restrictor per tool
}

The extra data shows up in the answers. Asked for the best barbecue in Austin, Qwen 3.8 27B listed five places with JSON and six with Markdown. The new one, Good BBQ Company, was one of the two highest rated. The Markdown answer also described Terry Black's as "4th-generation pitmasters, old-school smoked meats," from the place descriptions only the Markdown restrictor keeps.

What Happens When You Skip the JSON Restrictor

The bigger change is the full response, which I called Raw in the app. In the first blog, sending a raw SerpApi response to a local model wasn't an option. The seven raw JSON responses added up to 326,949 tokens, almost 10 times the context window. The same seven responses in Markdown came to 71,244 tokens, 4.6 times smaller, and none of them was bigger than the window.

Fitting in the window doesn't make it fast. With Markdown (Raw), the models read two to three times as many tokens as with Markdown (Restrictor), and took up to 2.7 times as long to answer.

Two full pages in one question can still overflow the context window. On one of the four models, the prompt comparing AAPL and TSLA went over 32K tokens. LM Studio cut part of it without an error, and the answer got Apple’s daily change wrong.

Which Output Should You Use?

Most local chats should start with Markdown using the restrictor, and the other outputs fit narrower cases.

Markdown (Restrictor): The default. Use it whenever a model reads the results.
JSON (Restrictor): Use it when your code reads the results, for example to sort by price.
Markdown (Raw): Use it when an answer needs the whole page and can wait up to three times longer. Avoid it for questions that need two big searches, such as comparing two stocks.
JSON (Raw): Don't send it to a local model.

If your hardware has the memory, you can raise the 32K window with the app’s LMSTUDIO_CONTEXT_LENGTH setting.

How the Four Models Compare

All four models finished every question. The times below are the average per search question in Markdown (Restrictor):

Qwen 3.8 27B is the app's default model when it's installed. In long test conversations, it turned follow-ups like "around there" and "which of those" into the right searches. It took about 42 seconds per question.
Meta Muse Glimmer 30B splits text into fewer tokens than the other three, so the same search results take less of its window. It was the slowest, about 62 seconds per question.
Prism Bonsai 27B is a 27B model that LM Studio lists at about 4 GB, and it took about 24 seconds per question.
Gemma 4 12B QAT is the smallest model here, at 12B parameters, and the fastest, at about 16 seconds per question, or about 43 with Markdown (Raw). In long conversations it had the most problems, such as giving a price that wasn’t in the search results.

With less memory than the 48 GB used here, the app may load a smaller window, so less search data fits. New open models also come out almost every week, so the best choice changes fast. The staff picks inLM Studio's model search are a good place to start, because LM Studio matches them to your hardware.

When Search Results Contradict What the Model Knows

Every model stops learning at its training cutoff, so it doesn’t know what happened after that date. On September 1, 2026, John Ternus became Apple's CEO and Tim Cook became executive chairman, after 15 years as CEO. So "Who is the current CEO of Apple?" puts a model's training against the search results.

Here's what Qwen 3.8 27B thought before its first search:

The user is asking who the current CEO of Apple is. This is a factual question. As of my knowledge, Tim Cook has been Apple's CEO since 2011. However, since this is a question about the current situation, I should verify with a search to make sure there haven't been any changes. Let me search.

That part is right. The model knows its knowledge can be out of date, and it checks. The harder part comes when the results disagree with it. In an earlier run, the AI Overview in Qwen's first results already named John Ternus, and Qwen didn't accept it.

This is a significant claim - it suggests Tim Cook stepped down and John Ternus took over. Today's date is 2026-10-02, so this would be about a month ago. Let me verify this with a news search to make sure this is accurate and not a hallucination.

We can read all of this because the models run locally. LM Studio writes each model's reasoning to its server log, word for word. Hosted reasoning models usually show you a summary instead. When OpenAI launched o1, itchose not to show the raw chain of thought, and other AI labs have done the same recently. One competitive risk is knowledge distillation, where a smaller model learns to copy a bigger model's outputs, and full reasoning is valuable material for that.

Conclusion

In this blog we added SerpApi’s Markdown output to the app and benchmarked it on four local models. For the same fields, Markdown took 27% fewer tokens than JSON, so the restrictor could keep more results for about the same budget. Even without a restrictor, all seven full Markdown responses fit in the 32K window.

The same approach works for any SerpApi engine. Add output=md to the request, keep one restrictor per engine, and spend the tokens Markdown saves on the fields your questions need. To try it, clone the repo and add your SerpApi API key.A free account includes 250 searches a month.

When the next model you want to try shows up in LM Studio, run it through the app's Benchmark page and compare it with these four.

How to build a web scraper: A beginner's guide

Hilman Ramadhan — Sat, 17 Oct 2026 17:34:05 +0000

The simple definition of web scraping is the automated process of extracting data from websites. It also usually involves saving that data into a structured format, like a database, CSV or JSON file, or text document. That way, you can parse large collections of pages (or entire websites) in a much more efficient way than a person browsing the pages and manually copying the desired information.

A web scraper first needs to make requests to a web server, and then fetch the contents of the requested page, just like a person using a web browser normally would. Then, it needs to parse the page into whichever format is required. Both of those steps can be tricky at scale—many servers will throttle or block IP addresses making bulk requests, and parsers have to understand various page layouts and rendering methods.

In this guide, we'll go over how to build your own web scraper in Python, JavaScript, and Ruby, and how to use the easier option—the Extract API from SerpApi.

Building your own web scraper

Web scraping can be done using a variety of programming languages and tools. Whether you're comfortable with JavaScript, Ruby, Go, or another language, libraries and frameworks are available to help you scrape websites.

The first step in web scraping is to fetch the HTML content of the web page you want. This is similar to how your web browser works when you visit a website—it sends a request to the server, and the server responds with the HTML, CSS, JavaScript, and other resources needed to display the page.

In web scraping, you'll use a programming language or tool to send an HTTP request to the website's server. This can be done using various methods:

Programming Libraries : Languages like Python, JavaScript, Ruby, and others have libraries that make it easy to send HTTP requests.
Built-in Functions : Some languages have built-in functions for making web requests.

By requesting the HTML content, you're retrieving the raw data that makes up the web page, which you can then process and analyze.

Once you've obtained the HTML content, the next step is to parse it to extract the information you're interested in. Web pages are structured documents that use HTML tags to define elements like headings, paragraphs, links, images, and more.

Some common methods of navigating through HTML pages are CSS selectors or XPath. For example, if you wanted to find all links in a page with "product" in the destination URL, you could use a CSS selector like this:

a[href*="product"]

Libraries like Beautiful Soup for Python, jsdom or Cheerio for Node.js, Nokogiri or Nokolexbor for Ruby, and others can parse HTML and XML documents. That way, you can navigate the document elements using standard selectors. Another option is using an AI model to parse the HTML.

Once you've identified the correct elements, you can extract the text, attributes, or other relevant information. That gives you the data for analyzing trends, feeding into another application, building a dataset, or other tasks.

Practice scraping a real website

It's time to build a web scraper! We will use the Hacker News home page as the example site, with the goal of extracting the title, external link, and visible link label for each post.

First, we need to figure out where the links are in the page's HTML. Try right-clicking one of the links, then clicking the "Inspect Element" or "Inspect" menu option.

🧭 Using Safari on Mac? You may need to enable development features first. From the top menu bar, select Safari > Settings, click the Advanced tab, then check the box for "Show features for web developers."

This will open the HTML page's structure, and jump to where that link element is located.

As of the time of writing, each submission is a <tr> table row element with a athing CSS class. In each of those table rows, there's a .titleline element containing everything we need: a link element (a) with the external link and post title, and a .sitestr tag with the visible link label.

Now it's time to scrape that information!

Web scraping using Python

If you want to build a web scraper in Python, you can start with the Requests and Beautiful Soup libraries. For more complex web pages beyond this example, you can try Selenium or Scrapy.

First, install Python on your computer if you haven't already, then create the folder you want to use for this project. Next, open the Terminal (Mac/Linux) or PowerShell (Windows) application, and change the current directory to that project folder:

cd ~/Desktop/scraper

📁 Not sure how to find a folder's path? Type cd and a space, then drag the folder into your Terminal or PowerShell window.

Next, create a Python virtual environment for the project, so all dependencies and other data stays within the folder:

python -m venv venv

💻 Python command not found? Try "python3" instead. The name of the executable depends on how it was installed.

Now, activate the Python virtual environment. On a Mac with the default bash/zsh shell, run this:

source venv/bin/activate

On a Windows PC in PowerShell, run this:

venv\Scripts\Activate.ps1

Next, install the Beautiful Soup and Requests libraries in the project:

pip install requests beautifulsoup4

Now it's time to start writing code! Create a new Python script called main.py in your project with the following contents, then try running it with the python main.py command:

import requests

URL = 'https://news.ycombinator.com/'
response = requests.get(URL)

html_content = response.text
print(html_content)

This imports the Requests library, sends a GET request to the URL, and stores the HTML content in a variable.

Now, let's parse the HTML to extract the data. Add the following import statement to the top of the script:

from bs4 import BeautifulSoup

Replace the print(html_content) statement with this line, which creates a Beautiful Soup object for the parsed HTML content:

soup = BeautifulSoup(html_content, 'html.parser')

Next, the script needs to get a list of all table rows with the athing class, and extract each row's link URL, inner text, and visible link. You can do this with a soup.find_all function and a for loop at the end of the script:

rows = soup.find_all('tr', class_='athing')

for row in rows:
    titleline = row.find('span', class_='titleline')
    link_tag = titleline.find('a', href=True)

    title = link_tag.text
    link = link_tag['href']

    visible_link_tag = row.find('span', class_='sitestr')
    # This avoids an error when the visible_link_tag is None
    visible_link = visible_link_tag.text if visible_link_tag else "No visible link"

    print("Title:", title)
    print("Link:", link)
    print("Visible Link:", visible_link)
    print('---')

Running the entire script should give you an output like this:

Title: StreetComplete on iOS is now in public beta
Link: https://github.com/streetcomplete/StreetComplete/issues/5421
Visible Link: github.com/streetcomplete
---
Title: RIP, vector database
Link: https://turbopuffer.com/blog/rip-vector-database
Visible Link: turbopuffer.com
---
Title: How to speed up the Rust compiler in September 2026
Link: https://nnethercote.github.io/2026/09/30/how-to-speed-up-the-rust-compiler-in-september-2026.html
Visible Link: nnethercote.github.io
---

The website has multiple pages. If we want to grab all the data, we must first find the pattern. In this case, this is what the patterns look like:

Page 1: https://news.ycombinator.com/?p=1
Page 2: https://news.ycombinator.com/?p=2
Page n: https://news.ycombinator.com/?p=n

You can check each page by wrapping the entire script in another for loop that counts from 1 to 3, and then runs the GET request with that number as the page count in the URL. The full script should now look like this:

import requests
from bs4 import BeautifulSoup

URL = 'https://news.ycombinator.com/'
max_pages = 3 # I gave the limit to only 3 pages

for i in range(1, max_pages+1):
    response = requests.get(f'{URL}?p={i}')
    html_content = response.text
    soup = BeautifulSoup(html_content, 'html.parser')

    rows = soup.find_all('tr', class_='athing')

    for row in rows:
        titleline = row.find('span', class_='titleline')
        link_tag = titleline.find('a', href=True)

        title = link_tag.text
        link = link_tag['href']

        visible_link_tag = row.find('span', class_='sitestr')
        visible_link = visible_link_tag.text if visible_link_tag else "No visible link"

        print("Title:", title)
        print("Link:", link)
        print("Visible Link:", visible_link)
        print('---')

The output will now include results from the first three pages.

You probably want to save the data somewhere other than the terminal output. Python's built-in CSV library can be used to write a CSV spreadsheet file, which can then be opened by Google Sheets, Microsoft Excel, LibreOffice Calc, and many other applications.

Here's how you could write a CSV file with the title, link, and visible link as the columns, and each post as a row:

import requests
from bs4 import BeautifulSoup
import csv

URL = 'https://news.ycombinator.com/'
max_pages = 3

# Open a CSV file in write mode
with open('data.csv', mode='w', newline='') as file:
    writer = csv.writer(file)

    # Write the header row
    writer.writerow(['Title', 'Link', 'Visible Link'])

    for i in range(1, max_pages+1):
        response = requests.get(f'{URL}?p={i}')
        html_content = response.text
        soup = BeautifulSoup(html_content, 'html.parser')

        rows = soup.find_all('tr', class_='athing')

        for row in rows:
            titleline = row.find('span', class_='titleline')
            link_tag = titleline.find('a', href=True)

            title = link_tag.text
            link = link_tag['href']

            visible_link_tag = row.find('span', class_='sitestr')
            visible_link = visible_link_tag.text if visible_link_tag else "No visible link"

            # Write the data row to the CSV file
            writer.writerow([title, link, visible_link])

Now, running the script will create a data.csv file in your project folder with all the data. The file looks like this when opened in a spreadsheet application:

You now have a basic web scraper running in Python! The below resources can help you take on more complex scraping tasks:

Web scraping with CSS selectors using Python : A guide on using CSS selectors to find and parse HTML elements.
Selenium web scraping : Web pages that load with JavaScript or other dynamic elements won't work with only GET requests and Beautiful Soup, but you can parse them with Selenium.
Web scraping in Python: A more in-depth guide on building scrapers with Python.
The documentation for Beautiful Soup, Requests, and Python's CSV library.
13 ways to scrape public data: More methods for parsing complex web pages in Python.

You can also skip to the Extract API for a simpler method of web scraping.

Web scraping using JavaScript

JavaScript is also an option for web scraping projects, though not in the same way you might use the language in front-end development. Web browsers will block access to most external domains with CORS restrictions, which is required for scraping, so you need to use Node.js or another non-browser runtime. Those runtimes typically don't have a built-in Document Object Model (DOM), so common functions for parsing HTML like querySelector or getElementById() are unavailable unless you add additional libraries.

For this web scraping project, we'll use the Node.js runtime with the jsdom library. Node.js has a built-in Fetch API for retrieving web content, so additional libraries for that purpose (like Axios) are no longer required.

First, you'll need Node.js and the NPM package manager installed on your computer. Then create a folder somewhere for your project, open a Terminal window (Mac/Linux) or PowerShell prompt (Windows), and navigate to that folder. For example, this will change the current directory to a "scraper" folder on your Desktop:

cd ~/Desktop/scraper

📁 Not sure how to find a folder's path? Type cd and a space, then drag the folder into your Terminal or PowerShell window.

Next, we need to install the jsdom library in the project, with this command:

npm install jsdom

Next, create a new JavaScript file called main.js in the project folder, and start with this code that fetches the HTML for the home page and displays it in the console:

const response = await fetch("https://news.ycombinator.com");
const text = await response.text();
console.log(text);

You can run it with the node main.js command. Next, we can parse the HTML by importing the jsdom library, passing the text to a JSDOM constructor, and making a for loop that goes through every .athing table row:

import jsdom from "jsdom";

const response = await fetch("https://news.ycombinator.com");
const text = await response.text();
const dom = new jsdom.JSDOM(text);

// Get every table row representing a post
for (let el of dom.window.document.querySelectorAll("tr.athing")) {
    // Get the title, link, and visible link
    const title = el.querySelector(".titleline a").textContent;
    const link = el.querySelector(".titleline a").href;
    const visibleLink = (el?.querySelector(".sitestr")?.textContent || "No visible link");
    // Display the data
    console.log(`Title: ${title}\nLink: ${link}\nVisible Link: ${visibleLink}\n---`);
}

The website has multiple pages. If we want to grab all the data, we must first find the pattern. In this case, this is what the patterns look like:

Page 1: https://news.ycombinator.com/?p=1
Page 2: https://news.ycombinator.com/?p=2
Page n: https://news.ycombinator.com/?p=n

This is a modified version of the script that fetches the first three pages, and then displays all links from all pages:

import jsdom from "jsdom";

const baseUrl = "https://news.ycombinator.com"

for (let i = 1; i <= 3; i++) {
    const response = await fetch(`${baseUrl}/?p=${i}`);
    const text = await response.text();
    const dom = new jsdom.JSDOM(text);
    // Get every table row representing a post
    for (let el of dom.window.document.querySelectorAll("tr.athing")) {
        // Get the title, link, and visible link
        const title = el.querySelector(".titleline a").textContent;
        const link = el.querySelector(".titleline a").href;
        const visibleLink = (el?.querySelector(".sitestr")?.textContent || "No visible link");
        // Display the data
        console.log(`Title: ${title}\nLink: ${link}\nVisible Link: ${visibleLink}\n---`);
    }
}

You might want to save the scraped data outside of the terminal output. We can write the results to a CSV file by importing Node's fs library and constructing the CSV format:

import jsdom from "jsdom";
import fs from "node:fs/promises";

const baseUrl = "https://news.ycombinator.com"
let csvText = `"Title","Link","Visible Link"`

for (let i = 1; i <= 3; i++) {
    const response = await fetch(`${baseUrl}/?p=${i}`);
    const text = await response.text();
    const dom = new jsdom.JSDOM(text);
    // Get every table row representing a post
    for (let el of dom.window.document.querySelectorAll("tr.athing")) {
        // Get the title, link, and visible link
        const title = el.querySelector(".titleline a").textContent;
        const link = el.querySelector(".titleline a").href;
        const visibleLink = (el?.querySelector(".sitestr")?.textContent || "No visible link");
        // Add the data as a new row
        csvText += `\n"${title}","${link}","${visibleLink}"`
    }
}

await fs.writeFile("data.csv", csvText);

Now, running the script will create a data.csv file in your project folder with all the data. The file looks like this when opened in a spreadsheet application:

You now have a basic web scraper running in Node.js with JavaScript! The below resources can help you take on more complex scraping tasks:

CSS selectors and combinators: Learn about using CSS selectors to find elements, which also work in jsdom and other HTML parsing methods.
Web scraping a dynamic website: How to use Puppeteer with Node.js to scrape a website that uses a dynamic layout instead of static HTML.
Web scraping with JavaScript and Node.js: A more detailed guide on web scraping techniques using JavaScript.

You can also skip to the Extract API for a simpler method of web scraping.

Web scraping using Ruby

Ruby is another language option for building a web scraping tool. You can use libraries like Net::HTTP or HTTParty for making GET requests to web pages, Nokolexbor or Nokogiri for parsing HTML and XML elements, and Selenium for scraping with an automated web browser.

First, you need Ruby installed on your computer. You can download it from the official website or use a version manager like rbenv or RVM. Then create a folder somewhere for your project, open a Terminal window (Mac/Linux) or PowerShell prompt (Windows), and navigate to that folder. For example, this will change the current directory to a "scraper" folder on your Desktop:

cd ~/Desktop/scraper

📁 Not sure how to find a folder's path? Type cd and a space, then drag the folder into your Terminal or PowerShell window.

Next, create a plain text file called Gemfile in your project folder to keep track of dependencies, with source 'https://rubygems.org' as the only line. You can do that in a few different ways:

Use a plain text editor (make sure not to add a .txt extension to the file)
For Windows PowerShell, run this command in the project folder: Add-Content Gemfile "source 'https://rubygems.org'"
For Mac/Linux, run this command in the project folder: "source 'https://rubygems.org'" >> Gemfile

We will use HTTParty for fetching web pages, Nokolexbor for parsing HTML, and csv for writing the data to CSV files. The CSV gem is usually bundled with Ruby, but it should still be defined as a dependency. Make sure they are installed with this command:

bundle add httparty nokolexbor csv

Next, create a Ruby file called main.rb in your project folder. The first version fetches the home page for Hacker News using the httparty gem:

require 'httparty'

url = 'https://news.ycombinator.com/'
response = HTTParty.get(url)
html_content = response.body
puts html_content

You can run it with the ruby main.rb command. Next, we need to add a require line for nokolexbor and then use it to parse each .athing table row as a post:

require 'httparty'
require 'nokolexbor'

url = 'https://news.ycombinator.com/'
response = HTTParty.get(url)
html_content = response.body
parsed_page = Nokolexbor::HTML(html_content)

parsed_page.css('tr.athing').each do |el|
    # Get the title, link, and visible link
    title = el.at_css('.titleline a').text
    link = el.at_css('.titleline a')['href']
    visible_link = el.at_css('.sitestr') ? el.at_css('.sitestr').text : 'No visible link'
    # Display the data
    puts "Title: #{title}\nLink: #{link}\nVisible Link: #{visible_link}\n---"
end

The website has multiple pages. If we want to grab all the data, we must first find the pattern. In this case, this is what the patterns look like:

Page 1: https://news.ycombinator.com/?p=1
Page 2: https://news.ycombinator.com/?p=2
Page n: https://news.ycombinator.com/?p=n

Modifying the script to parse the first three pages isn't too difficult. We just need a loop that adds the current iteration to the page URL, like this:

require 'httparty'
require 'nokolexbor'

url = 'https://news.ycombinator.com/'
max_pages = 3

(1..max_pages).each do |i|
    response = HTTParty.get("#{url}?p=#{i}")
    html_content = response.body
    parsed_page = Nokolexbor::HTML(html_content)

    parsed_page.css('tr.athing').each do |el|
        # Get the title, link, and visible link
        title = el.at_css('.titleline a').text
        link = el.at_css('.titleline a')['href']
        visible_link = el.at_css('.sitestr') ? el.at_css('.sitestr').text : 'No visible link'
        # Display the data
        puts "Title: #{title}\nLink: #{link}\nVisible Link: #{visible_link}\n---"
    end
end

You might want to save the scraped data for future use. We can write the results to a CSV file by adding a require line for csv, creating a header row and a row for each post, and then writing it to a file:

require 'csv'

URL = 'https://news.ycombinator.com/'
max_pages = 3

# Open a CSV file in write mode
CSV.open('data.csv', 'w') do |csv|
  # Write the header row
  csv << ['Title', 'Link', 'Visible Link']

  (1..max_pages).each do |i|
    response = HTTParty.get("#{URL}?p=#{i}")
    html_content = response.body
    parsed_page = Nokogiri::HTML(html_content)

    rows = parsed_page.css('tr.athing')

    rows.each do |row|
      titleline = row.at_css('span.titleline a')
      title = titleline.text
      link = titleline['href']

      visible_link_tag = row.at_css('span.sitestr')
      visible_link = visible_link_tag ? visible_link_tag.text : 'No visible link'

      # Write the data row to the CSV file
      csv << [title, link, visible_link]
    end
  end
end

puts 'Data successfully written to data.csv'

Now, running the script will create a data.csv file in your project folder with all the data. The file looks like this when opened in a spreadsheet application:

You now have a Ruby-powered web scraper! Check out these resources for help with more complex scraping tasks:

CSS selectors and combinators: Learn about using CSS selectors to find elements, which are used by Nokolexbor, Nokogiri, and other HTML parsing methods.
Selenium documentation: For sites that load dynamically instead of providing static HTML, you need to use Selenium or another browser automation tool.
Web scraping with Ruby: A more complete guide on web scraping methods while using Ruby.

You can also skip to the Extract API for a simpler method of web scraping.

Challenges with web scraping

Basic web scraping, with pages already rendered in HTML with consistent element names, isn't too complicated in many programming languages and environments. However, you will inevitably run into problems as you try parsing different websites, or try requesting many pages.

These are some of the common challenges with web scraping at scale:

Identifying data sources : Determining where and how data is stored can be complex. Websites may store data across different pages or make requests over APIs, making it difficult to pinpoint the exact source.
Handling CAPTCHA tests : Websites often use a CAPTCHA, short for "Completely Automated Public Turing test to tell Computers and Humans Apart," to block automated access. Bypassing these tests without violating terms of service requires sophisticated methods, such as using CAPTCHA-solving services, which adds cost and complexity.
Managing IP blocks and rate limits : Frequent requests from the same IP can lead to blocks or bans. Using proxies to rotate IPs and implementing smart throttling mechanisms can help mitigate this but require careful management and additional infrastructure.
Legal and ethical considerations : Scraping must be done in compliance with laws and website terms of service. Ignoring these can lead to legal actions and ethical issues.
Data structure changes : Websites often update their layout and structure, which can break scrapers. Regularly maintaining and updating scraping scripts is crucial to handle these changes.
Dynamically-rendered pages: Many websites will render the page with JavaScript, instead of sending a complete HTML structure over the initial request. This requires fetching the page with an automated web browser, like Selenium or Puppeteer, instead of a simple GET request.

The easy web scraping method

If you don't want to build and maintain your own web scraper, you can use our Extract API. It can parse any public web page and return its contents as structured JSON or Markdown, including automatic detection of headers, meta tags, inline links, and the complete page text. There's also a Map mode for finding all pages on a given website, and Crawl mode for finding and parsing all of a website's pages.

If you request a page on an engine or platform that has a dedicated SerpApi API, the results will be automatically routed to that API. For example, fetching a YouTube video URL with the Extract API will return all the data available through the YouTube Video API, like the video description and all recommended videos.

If the built-in parsing from the Extract API isn't enough, you can still use it to retrieve the fully-rendered HTML, and then build your own parser using that input.

Getting started with SerpApi

You need a free SerpApi account before you can use the Google Play Store APIs. You can upgrade to a paid account in the future if you need more search capacity, faster speeds, or other features.

First, create an account and verify your email and other details. After that, get the API key from your account dashboard.

You should store your API key in a safe location if you are sharing or publishing your code. If the key is leaked or stolen, you can make a new one from the SerpApi account dashboard.

Install the SerpApi library (optional)

You can use SerpApi's official libraries for Python, JavaScript, Ruby, Java, and other languages. They aren't required, but they provide a simple wrapper around the API requests.

The APIs can also be used with cURL, fetch() in Node.js, or anything else that can send GET requests and process a JSON response.

Extract API with a GET request

Here's how you could fetch the Hacker News home page with the Extract API, using a simple GET request:

https://serpapi.com/search.json?engine=extract&url=https://news.ycombinator.com&api_key=YOUR_KEY_GOES_HERE

The default response is in JSON format, with basic page information in the page_information object, a list of all included links in the inline_links object, and so on.

If you want the response in Markdown format, instead of JSON, just add output=md to the request:

https://serpapi.com/search.json?engine=extract&url=https://news.ycombinator.com&output=md&api_key=YOUR_KEY_GOES_HERE

If you need the page's rendered HTML, for building your own parser or some other use case, you can set output=html for the request:

https://serpapi.com/search.json?engine=extract&url=https://news.ycombinator.com&output=html&api_key=YOUR_KEY_GOES_HERE

Extract API with JavaScript

Here's how you can use the Extract API with the SerpApi JavaScript library to display all posts from the first Hacker News home page:

import { getJson } from 'serpapi';

let search = await getJson({
  engine: "extract",
  api_key: YOUR_KEY_GOES_HERE,
  url: "https://news.ycombinator.com/",
  mode: "scrape"
});

for (let link of (search?.inline_links || [])) {
  // Skip internal links, and the "API" link at the bottom of the page
  if (link.internal || link.link === "https://github.com/HackerNews/API") {
    continue;
  }
  // Display the title, link, and visible link text
  console.log(`Title: ${link.title}\nLink: ${link.link}\nVisible Link: ${link.displayed_link}\n----`)
}

The Extract API fetches the page contents and parses it, so all that is required is going through the list of extracted links (search.inline_links) and filtering the results to external URLs.

Here's a more advanced version that scans the first three pages, and saves the results to a CSV file:

import { getJson } from 'serpapi';
import fs from "node:fs/promises";

const baseUrl = "https://news.ycombinator.com"
let csvText = `"Title","Link","Visible Link"`

let settings = {
  engine: "extract",
  api_key: YOUR_KEY_GOES_HERE,
  mode: "scrape"
}

for (let i = 1; i <= 3; i++) {
  settings.url = `${baseUrl}/?p=${i}`;
  let search = await getJson(settings);
  for (let link of (search?.inline_links || [])) {
    // Skip internal links, and the "API" link at the bottom of the page
    if (link.internal || link.link === "https://github.com/HackerNews/API") {
      continue;
    }
    // Add the data as a new row
    csvText += `\n"${link.title}","${link.link}","${link.displayed_link}"`;
  }
}

await fs.writeFile("data.csv", csvText);

The crawl mode is helpful for analyzing many pages in a group. This example will crawl the UniFi store, up to 50 pages, to find every product page:

// import { getJson } from 'serpapi';

let search = await getJson({
  engine: "extract",
  api_key: YOUR_KEY_GOES_HERE,
  url: "https://store.ui.com/us/en",
  mode: "crawl",
  max_pages: 50
});

let productUrls = [];

// Iterate through all crawled pages
for (let page of search.pages) {
  // Iterate through all inline links
  for (let result of (page?.inline_links || [])) {
    // Filter out non-product links, and links already added to the list
    if (result.link.includes("/products/") && !productUrls.includes(result.link)) {
      productUrls.push(result.link);
    }
  }
}

// List all products
for (let product of productUrls) {
  console.log(product);
}

More information

You can learn more from the documentation pages for the Extract API, Map mode, and Crawl mode.

7 Best Rank Tracking API Options Compared

Noraina Nordin — Thu, 15 Oct 2026 09:11:51 +0000

Once you track more than a few dozen keywords, checking positions by hand stops working. You need data you can load into a dashboard, a client report or your own app. Picking the best rank tracking API starts with one decision. You can buy a rank tracker that has an API, like Semrush, AccuRanker or Ahrefs. Or you can build on a SERP API, which returns the raw results page and lets you work out positions yourself.

Rank trackers give you dashboards, history and alerts out of the box. SERP APIs give you more control over location, device and schedule, and usually cost less per keyword check. This guide compares 7 options across both camps on cost, accuracy, mobile support and AI Overview parsing.

🚦 Disclosure: this is SerpApi's blog, and SerpApi is one of the 7 options. We list its limitations alongside everyone else's.

Best rank tracking APIs at a glance

Prices and features below were checked on each provider's pricing and documentation pages in October 2026.

Provider

Type

Starting price

Free tier

Location & device targeting

Update frequency

AI Overview parsing

SerpApi

SERP API

$25/mo for 1,000 searches

250 searches/mo

City, coordinates or uule; desktop, tablet, mobile

Real time, on your schedule

Yes

|
|

DataForSEO

SERP API

$0.60 per 1,000 SERPs (standard queue)

$1 credit + sandbox

Location code, name or coordinates; desktop, mobile

About 5 min (standard) to about 6 sec (live)

Yes

|
|

Serpent API

SERP API

$0.60 per 1,000 calls

Free calls on sign-up

Country and language only

Real time, on your schedule

Yes (Deep Search)

|
|

cloro

SERP API

$30/mo for 37,500 credits

500 credits/mo

Country, city or uule; desktop, iOS, Android

Real time, on your schedule

Yes (+2 credits)

|
|

Semrush Position Tracking

Rank tracker API

$549/mo (Advanced) + API units

No API access on free plan

Down to postal code; desktop, phone, tablet

Daily

Filter by AI Overview

|
|

AccuRanker

Rank tracker API

$764/mo (Expert, for API)

Not stated

Country or local; device not stated

Daily + on-demand refresh

Not stated

|
|

Ahrefs Rank Tracker

Rank tracker API

$129/mo (Lite)

Not stated

Country, state or city; desktop, mobile

Weekly

Yes

Pick by what you need most:

SerpApi if you want real-time results, city-level and tablet targeting, and no charge for failed or cached searches.
DataForSEO if cost per keyword matters more than speed and you can work with a task queue.
Serpent API if you only track country-level desktop rankings and want the lowest entry cost.
cloro if AI Overview tracking is a core need and you want iOS and Android as separate devices.
Semrush if your team already works in Semrush and wants rankings flowing into its reports.
AccuRanker if you need on-demand refreshes and an unlimited read API at enterprise scale.
Ahrefs if weekly data is enough and you already pay for Ahrefs.

SERP API or rank tracker API?

Both return a position for a keyword. The difference is who does the work between the search and the number.

A rank tracker API sits on top of a finished product. You set up a project in Semrush, AccuRanker or Ahrefs, add keywords, and the tool checks them on its own schedule. The API then lets you read that stored history. You get clean data and no parsing. You also accept the tool's update frequency, its locations and its price per tracked keyword.

A SERP API returns the full results page as JSON every time you call it. You decide when to check, from where and on which device. You then find your domain in the results and store the position yourself. That takes some code, but it removes per-seat pricing. It also lets you track things a dashboard may not show, like who holds the featured snippet.

A rough rule of thumb:

Choose a rank tracker API if you need reports for non-technical stakeholders and your keyword list fits the plan.
Choose a SERP API if you track many locations or devices, need checks on your own schedule, or want rankings inside your own product.

The r/SEO_LLM discussion on SEO API providers shows the same split. Some people want finished metrics, while others want raw SERP data they can shape themselves.

There is also a middle path. Open source trackers give you a dashboard but run on a SERP API you pay for directly. SerpBear is the best known. It is MIT licensed, runs in Docker, and can use SerpApi or several other scrapers.

We built our own take on this idea. SerpTrail, our self-hosted rank tracker, runs in Docker with Rails and SQLite. It tracks Google rankings and AI Overviews across multiple locations on a daily, weekly, biweekly or monthly schedule. It can check up to five pages of Google results, and the source code is on GitHub. It runs on your SerpApi key, so the free plan's 250 searches are enough to try it.

SerpTrail self-hosted rank tracker dashboard

What makes an accurate rank tracker?

An accurate rank tracker is one that reports the results page your customers actually see. There is no single "true" position for a keyword. Google builds each page from the searcher's location, device, language and the moment of the search. A tracker is accurate when it controls all four and keeps them the same from one check to the next.

Location, device, language and time

Location matters most for anything with local intent. A national check and a check from a specific city can return different pages, so set the location at city level when it matters. If you target several countries, our guide to global SEO strategies covers how results shift by market and language. Match the language (hl) and country (gl) to your audience too.

Device is the next big factor. On October 6, 2026, we ran "best running shoes for women" through SerpApi from Austin, Texas, once on desktop and once on mobile. Desktop returned 10 organic results and mobile returned 9. Only 6 pages appeared in both. Outside Online ranked #3 on desktop but #7 on mobile, and GearJunkie's #5 desktop result didn't appear on mobile at all.

Desktop vs mobile Google organic results for the same keyword and location

Time of check matters because rankings move during the day. Check at the same hour each day so changes reflect Google, not your schedule.

Why tools and humans see different results

A common complaint is that a tool's ranking doesn't match a manual search. Usually the tool isn't wrong. This is caused by the person searching being logged in, having search history, or sitting a few miles from the location the tool uses. Accuracy comes from consistent parameters more than from any one provider.

The num=100 change

In September 2025, Google stopped supporting the num=100 parameter, which many tools used to fetch 100 results in one request. Tracking the top 100 now takes about 10 paginated requests per keyword. Ask any provider how it handles depth and how it bills for it.

Check against Search Console and pixel position

Google Search Console is a useful sanity check. Its average position blends every location and device, so it won't match a single check. But a large, lasting gap between the two is a sign to recheck your tracker's settings.

Position number alone also hides how visible a result is. A #1 organic result can sit below an AI Overview, ads and a local pack. Pixel ranking measures how far down the page a result appears, which tells you whether it is seen without scrolling. For more on how AI Overviews changed what a ranking means, read rank tracking in the age of AI Overviews.

Organic result #1 pushed below the fold by an AI Overview

Mobile rank tracking through an API

Google uses mobile-first indexing, which means it uses the mobile version of a page for indexing and ranking. That doesn't mean desktop and mobile results are the same. The mobile page has a narrower layout, shows SERP features in a different order, and often places local packs higher. As our Austin test showed, the set of pages can differ as well as their order.

For most sites, a large share of searches happen on phones. If you only track desktop, you may be looking at rankings that most of your visitors never see.

Mobile doubles the call count

Every device is a separate search. Tracking 500 keywords daily on desktop and mobile means 1,000 checks a day, not 500. With a SERP API, that doubles your bill. Rank trackers handle it differently. Some count a mobile keyword as a separate tracked keyword, while others include both devices. Ahrefs, for example, says mobile rankings are available on all Rank Tracker plans. Check this before comparing prices.

What to check before you pick a provider

A device parameter. The API should accept a mobile device value. Some APIs have separate options for iOS and Android tracking, and a tablet setting.
Location depth on mobile. City-level targeting matters even more on mobile, where local results take up more of the screen.
SERP features in the mobile response. Make sure local packs, AI Overviews and other features are parsed for mobile, not only desktop.
How mobile is billed. Find out whether a mobile check costs the same as desktop or counts against a separate limit.

One of the seven APIs in this guide, Serpent API, has no device parameter in its documentation. That rules it out for mobile rank tracking.

The 7 best rank tracking APIs

We start with the four SERP APIs, then cover the three rank tracker APIs. For each one, we cover targeting, data freshness, AI Overview support, cost and where it falls short.

1. SerpApi

SerpApi is a real-time SERP API. Each request runs a live Google search and returns the parsed results page as JSON or Markdown. It also covers Bing, YouTube, Google Maps and dozens of other engines through the same key.

Type: SERP API
Best for: teams that need precise location and device control, or that want rankings inside their own product
Location and device targeting: location at city level, uule, or lat and lon with a radius. Set gl, hl and google_domain for country and language. device accepts desktop, tablet or mobile.
Refresh and scheduling: real-time on every call, on whatever schedule you set. Results are cached for an hour, and cached searches are free. Add no_cache to force a fresh search. SerpTrail adds daily, weekly, biweekly or monthly scheduling if you don't want to write your own.
AI Overview parsing: returned in an ai_overview block. Sometimes Google loads it separately, and the response includes a page_token you pass to the Google AI Overview API.
Starting price and free tier: 250 free searches a month. Paid plans start at $25 for 1,000 searches, then $75 for 5,000 and $150 for 15,000. Only successful searches count, so errors and failed searches are free. See every plan on the SerpApi pricing page.

SerpApi Google Search Playground

A few extras matter for rank tracking. The Pixel Position API (output=json_with_pixel_position) returns the on-page coordinates of each result, so you can measure pixel rank as well as position. Production plans and above include the U.S. Legal Shield, which covers legal questions around scraping public search results. SerpApi also offers an MCP server, so AI assistants can run searches directly.

Limitation: SerpApi costs more per search than queue-based APIs. At 30,000 searches a month, it is $275 against $18 on DataForSEO's standard queue. There is also no way to fetch 100 results in one request. You page through with start, and each page is one search.

2. DataForSEO

DataForSEO sells SERP data by the request, with three speed tiers. You post a task and collect results later, or pay more to get them live.

Type: SERP API
Best for: high-volume tracking where cost per keyword matters more than speed
Location and device targeting: location_code, location_name or location_coordinate (latitude, longitude and radius). device is desktop or mobile, and os sets Windows, macOS, Android or iOS.
Refresh and scheduling: the standard queue returns in about 5 minutes. Live mode returns in up to about 6 seconds. depth goes up to 700 results, billed per page of 10.
AI Overview parsing: returned as an ai_overview item. Setting load_async_ai_overview fetches ones Google loads separately, for an extra $0.0006 per task.
Starting price and free tier: $0.0006 per SERP on the standard queue, $0.0012 on priority and $0.002 live. New accounts get $1 of credit and a free sandbox.

DataForSEO is the cheapest per search of the SERP APIs here. If you track tens of thousands of keywords a day and can wait a few minutes, that gap adds up quickly.

Limitation: the cheapest tier uses a post-then-fetch workflow, which means more code than a single synchronous call. There is no tablet device option.

3. Serpent API

Serpent API is a low-cost SERP API that covers Google, Bing, Yahoo, DuckDuckGo and Brave. It is pay as you go, with discounts for larger deposits.

Type: SERP API
Best for: country-level desktop tracking on a tight budget, or tracking several search engines at once
Location and device targeting: country and language only. The docs list no device or city parameter.
Refresh and scheduling: real time on each call. num accepts 1 to 100 and pages accepts 1 to 10.
AI Overview parsing: include_aio on Deep Search requests, plus include_ai_mode for AI Mode. A paid pixel_position option is also available.
Starting price and free tier: $0.60 per 1,000 calls pay as you go, with a $10 minimum deposit. A $100 deposit unlocks 10× lower pricing and $500 unlocks 20× lower pricing. New accounts get free calls.

Serpent's own blog puts 1,000 keywords tracked daily at about $18 a month on the default tier. Paid credits never expire, which suits irregular usage. Promotional credits expire after 30 days.

Limitation: with no device or city targeting, Serpent API can't do mobile rank tracking or local rank tracking. It fits national desktop tracking only.

4. cloro

cloro is a SERP API with a credit system and a focus on AI search results, including AI Overviews.

Type: SERP API
Best for: tracking AI Overview presence next to classic rankings at a mid-range price
Location and device targeting: country (or gl), plus location by canonical name or uule for city level. hl sets language. device accepts desktop, ios, android or mobile.
Refresh and scheduling: real time on each call. pages fetches 1 to 10 pages.
AI Overview parsing: set include.aioverview for an extra 2 credits per search.
Starting price and free tier: 500 free credits a month. Lite is $30 for 37,500 credits and Hobby is $100 for 250,000. Higher tiers bring the rate down to $0.31 per 1,000 credits.

The math for search credits is a bit complicated here. A Google search costs 3 credits, each extra page adds 2, and AI Overview adds 2. Synchronous requests add 2 more. A synchronous search with AI Overview therefore costs 7 credits.

Limitation: the headline price per 1,000 credits is not a price per search. Unused credits also don't roll over to the next month.

5. Semrush Position Tracking

Semrush Position Tracking is a full rank tracker inside the Semrush suite. Its API lets you create campaigns, add keywords and pull ranking reports into your own tools.

Type: rank tracker API
Best for: agencies and in-house teams already reporting from Semrush
Location and device targeting: locations down to postal code. Devices are desktop, phone and tablet.
Refresh and scheduling: daily updates, tracking up to the top 100 positions.
AI Overview parsing: you can filter tracked keywords by SERP features, including AI Overview.
Starting price and free tier: API access starts on the Advanced plan at $549 a month, billed monthly. There is no API access on the free plan.

API calls use Semrush API units on top of the subscription. Creating a campaign, adding a keyword and requesting a report each cost 100 units. Check how many units your plan includes before you plan a large integration.

Semrush's strength is everything around the rankings. Keyword volumes, competitor visibility and site audits live in the same account, so the ranking data connects to the rest of your SEO work.

Limitation: the price of entry is high if you only need ranking data. API units are a second cost to track, and you are tied to Semrush's daily schedule.

6. AccuRanker

AccuRanker is a dedicated rank tracker known for speed and on-demand refreshes. It is built for agencies and large in-house teams.

Type: rank tracker API
Best for: large keyword sets that need fresh data on demand
Location and device targeting: country or local level. The pages we checked don't state device options.
Refresh and scheduling: daily by default. You can also refresh one keyword or a whole domain on demand, every 2 hours.
AI Overview parsing: not stated on the pages we checked.
Starting price and free tier: an unlimited read API comes with the Expert plan at $764 a month. The write API, for adding keywords and domains, needs an Enterprise plan.

AccuRanker's features page describes full API access at no additional cost. That makes the read API good value if you already need Expert-level tracking. You can pull as much history as you like without counting units.

Limitation: the API sits behind a high-priced plan, and creating projects by API needs Enterprise. Confirm mobile tracking and AI Overview support with sales before you buy.

7. Ahrefs Rank Tracker

Ahrefs Rank Tracker is part of the Ahrefs toolset. Its API returns stored rankings for the projects you set up in Ahrefs.

Type: rank tracker API
Best for: teams already on Ahrefs who can work with weekly data
Location and device targeting: country, state or province, or city. Each request sets device to desktop or mobile.
Refresh and scheduling: weekly updates.
AI Overview parsing: responses list SERP features, including ai_overview. A best_position_kind field shows when your best position is inside an AI Overview.
Starting price and free tier: Lite costs $129 a month and includes 750 tracked keywords. It also includes API and MCP access with 1,000 credits per user.

The rank-tracker-overview endpoint needs a project ID, a device and a date. It returns position, URL, location and SERP features for each keyword. Ahrefs also offers an MCP server, so you can query rankings from an AI assistant.

Limitation: weekly data misses short-term swings, such as the days after an algorithm update. The 1,000 API credits on Lite also limit how often you can pull data.

What rank tracking costs per keyword

SERP APIs charge per search, so the cost of tracking comes down to one formula:

Monthly searches = keywords × devices × locations × checks per month

Take a common setup: 500 keywords, checked on desktop and mobile, from one location, every day. That is 500 × 2 × 1 × 30, or 30,000 searches a month. Here is what that costs with each provider.

Provider

What 30,000 checks needs

Monthly cost

Notes

SerpApi

30,000 searches

$275 (Big Data plan)

Real time; failed and cached searches are free

|
|

DataForSEO

30,000 SERPs

$18 standard, $36 priority, $60 live

Standard queue takes about 5 minutes

|
|

Serpent API

15,000 calls (desktop only)

About $9

No mobile option, so half the setup isn't possible

|
|

cloro

90,000 to 210,000 credits

$100 (Hobby plan)

Range depends on AI Overview and sync requests

|
|

Semrush

500 keywords in Position Tracking

$549 + API units

Advanced plan needed for API

|
|

AccuRanker

500 keywords

$764 (Expert plan)

Unlimited read API

|
|

Ahrefs

500 keywords

$129 (Lite plan)

Weekly updates only, so not a like-for-like match

Three things change the math. Each extra location adds another full set of searches. Tracking beyond the top 10 multiplies SERP API costs, since the end of num=100 means each extra page is a separate request. And rank trackers bill by tracked keyword, so their price stays flat however often they check.

For small keyword lists, a rank tracker can be the simpler buy. Once you track many locations or devices, a SERP API is usually cheaper per check.

FAQ

What is the best rank tracker tool?

It depends on how you'll use the data. For dashboards and client reports, a rank tracker such as Semrush, AccuRanker or Ahrefs is the simplest choice. For tracking inside your own app, or across many locations and devices, use a SERP API like SerpApi or DataForSEO. You get more control at a lower cost per check.

Which AI-powered rank tracker is the best?

Most tools now report AI Overviews in some form. Ahrefs and Semrush flag when one appears for a tracked keyword. SERP APIs like SerpApi, DataForSEO and cloro return the AI Overview text and its cited sources, so you can see whether your site is cited. To try this yourself, follow our guide to track AI Overview rankings in Python.

How do I check my ranking?

For a one-off check, search in a private browser window with location set to your target city. Then compare with the average position in Google Search Console. For ongoing tracking, use a rank tracker or a SERP API so every check uses the same location, device and language.

Start tracking your rankings

SerpApi gives you real-time Google results with city-level and device targeting, AI Overview parsing and pixel positions. Failed searches are never billed. Sign up for a free account to get 250 free searches a month, then try your first ranking check in minutes.

If you need help getting started, contact us. We're happy to help.

What Developers Built With SerpApi at API World 2026

Sonika Arora — Mon, 12 Oct 2026 21:43:46 +0000

At API World 2026, we wanted developers to do more than just hear about SerpApi. So we put our APIs in their hands and let them build.

At our booth, developers could experiment with SerpApi live, connect search data to their existing tools and workflows, and see what they could create with real-time web data. Many developers participated in our hackathon and used SerpApi to build real-time search capabilities, automated workflows, and AI-driven applications.

The result? A collection of 124 amazing projects that showed just how many ways developers can put search data to work.

Check out five of the many interesting projects built with SerpApi at API World.

1. AccessForm

Call. Talk. Your form is filled.

The people who most need help with official paperwork are the ones least able to do it online. A blind person with a hospital bill, a 70-year-old who can't walk to the bus, a student with ADHD facing a disability office form. They all have a phone. Most of them don't have a screen they can use, and some don't have internet at all.

How AccessForm helps

You call a number and say what's going on in your own words. AccessForm works out which official program fits, finds the real application form from the official source, asks you its questions one at a time in plain language, fills the actual PDF, and texts you the result with a list of what's still missing.

Everything happens in one phone call. It never submits, never signs, and never decides eligibility. If it can't verify an official form for the exact organization you named, it says so and stops.

https://www.youtube.com/embed/F-DBDH3wa48?feature=oembed

How SerpApi was used

Vapi runs the voice agent on a Twilio number. Every tool call hits a Next.js backend that uses SerpApi to find and verify the official form, OpenAI to read the form's fields and turn them into spoken questions, pdf-lib to fill the real PDF, Nutrient for accessibility tagging when credits allow, and Xano as the system of record for cases, answers, and completeness. A conversation page rebuilds the call live from Xano's event stream, so a laptop can watch a phone call as it happens.

The developers used SerpApi to search for and discover official application forms (PDFs) based on a user's specified need, organization, and location (e.g., executing live web queries to find official .gov, .edu, or official organizational site links).

Built with: SerpApi's Google Search API

👉 See project details

👉 Github

2. PantryProof

PantryProof turns fragmented recall evidence into cited, human-reviewed actions and a replayable closure packet for every partner pantry.

Food banks and partner pantries form the “true last mile” of a food recall. Products may arrive through central distribution, retail rescue, local purchasing, food drives, and other paths. Partner agencies may operate intermittently or rely on volunteers, so sending an alert does not prove that the affected product was found, contained, or communicated downstream.

PantryProof was built around a simple question: How can a food bank prove that every partner completed the correct recall response without letting an uncertain AI prediction make the final safety decision?

How PantryProof helps

PantryProof streamlines food safety responses across partner pantries by discovering official recall notices via SerpApi, extracting cited data, requiring human approval for extracted fields, and deterministically matching identifiers (UPC/GTIN and lot numbers) against inventory to trigger necessary quarantine, disposal, or notification actions before generating a verified, audit-ready PDF closure packet.

https://player.vimeo.com/video/1223743447?app_id=122963

The browser experience is built with TypeScript, JavaScript, semantic HTML, and responsive CSS. A deterministic TypeScript domain engine owns identifier normalization, matching policy, agency aggregation, human review state, response completeness, and closure readiness.

How SerpApi was used

Developers used SerpApi to get structured, current Google News recall results ordered newest first. They limit results to FDA and USDA domains. PantryProof validates every returned URL, determines evidence tier from the actual hostname rather than an untrusted label, and allows a coordinator to attach a reviewed result as canonical evidence.

This makes web search part of a traceable evidence workflow instead of treating the first search result as truth.

Built with: SerpApi's Google News API

👉 See project details

👉 Github

👉 See it live

3. ClinicalBrief AI (by LearnHive Labs)

Upload patient lab data - GPT-4o searches PubMed, retrieves clinical guidelines, and delivers a signed clinical brief to the clinician in under 60 seconds.

Clinicians spend 2+ hours per complex patient case searching literature, cross-referencing lab values against treatment protocols, and writing structured briefs. Most existing AI tools either hallucinate clinical recommendations or cost $50,000+ per year. There is no open, inspectable, evidence-grounded pipeline that any clinic can run.

How ClinicalBrief AI helps

ClinicalBrief AI was built based on a research publication on adaptive health orchestration to prove that a fully agentic, source-cited, confidence-gated clinical intelligence pipeline can be built on open infrastructure - in n8n, without a single line of server code.

https://www.youtube.com/embed/OMoF1RoaBQA?feature=oembed

A patient or pharmacy staff member fills a Google Sheet with lab values: HbA1c, LDL, BMI, blood pressure, medications, diagnoses. No app. No login. No technical knowledge required.

How SerpApi was used

When a new row is added, the n8n pipeline fires automatically - SerpApi queries Google Scholar in real-time using the patient's diagnoses. Three current PubMed publications are retrieved and injected into the agent prompt before any generation happens. GPT-4o generates clinical briefs using patient data, PubMed evidence, and clinical guidelines. Briefs with a confidence score of 80% or higher are electronically signed and delivered to clinicians, while those below are reviewed by humans. All data is sent to a physician.

Built with: SerpApi's Google Scholar API

👉 See project details

👉 Github

4. ApplyPilot

ApplyPilot finds real opportunities and proves why they match you - using grounded evidence, deterministic scoring, bounded AI, and explicit uncertainty instead of black-box recommendations.

Most AI job-matching systems give candidates a score. But a score can hide a dangerous assumption: when evidence is missing, an AI system may treat that missing information as if it were negative evidence. That idea became the foundation for an evidence-controlled opportunity intelligence system where AI can interpret evidence, but cannot silently manufacture it.

How ApplyPilot helps

ApplyPilot uses a multi-step process to match candidates with real job opportunities. It extracts evidence from résumés, verifies it, and then searches the web for relevant jobs. The final result is a ranked list of opportunities with detailed explanations of why they are a good fit.

The end-to-end workflow is:

Résumé → grounded evidence → human review → candidate profile → live web discovery → deterministic evaluation → bounded AI reasoning → self-audit → ranked opportunities → Why this job?

https://www.youtube.com/embed/X8vQZmbpb20?feature=oembed

How SerpApi was used

ApplyPilot uses SerpApi's Google Jobs API to discover real public opportunities during a Mission. The system preserves the provider query, raw listing evidence, canonical opportunity, and source lineage. Repostings can be deterministically deduplicated while preserving the underlying sources.

SerpApi helps ApplyPilot operate on changing real-world opportunities rather than a static demonstration dataset. Live web data changes the AI experience directly: the reasoning pipeline evaluates opportunities discovered during the run instead of answering from stale or fabricated listings.

Built with: SerpApi's Google Jobs API

👉 See project details

👉 Github

👉 See it live

5. DepositCheck

DepositCheck protects renters from listing scams by using SerpApi's Google Lens API to perform visual reverse image searches on listing photos, matching them against the live web to confirm if an apartment matches its claimed address or belongs to another property.

Rental scams work by theft, not invention. A scammer copies the photos and the description from a real listing, swaps the contact details and the address, and reposts it somewhere with weaker verification. Someone wires a deposit for a property the "landlord" has never owned.

The photos are the one thing the scammer cannot change. Swap them out and the listing stops looking like the flat that attracts victims in the first place. But a renter cannot check that. You would have to recognize one apartment interior among billions of published images, from memory, on a phone, while a "landlord" tells you three other people are viewing tonight.

How DepositCheck helps

You upload one photo from the listing and type the address you were given. SerpApi's google_lens engine with type=exact_matches runs Google's visual matching model over the live web index and returns every page carrying that same image. The claimed address is matched against what those pages say. The photo is deleted as soon as the lookup returns.

One of three verdicts is generated - CORROBORATED (a genuine syndicated listing), CONTRADICTED (the photos belong to one specific other address), or UNVERIFIED (not enough evidence either way).

https://player.vimeo.com/video/1223247077?app_id=122963

How SerpApi was used

DepositCheck uses SerpApi's Google Lens API with type=exact_matches to return every page carrying that same image. The claimed address is matched against what those pages say. The team aims to extend the solution in the future by using Google Maps API to verify the address is a residential property rather than a parking lot or a mailbox storefront and SerpApi's Google Search API on the landlord's phone number and email, which scam operations reuse across dozens of listings.

Built with: SerpApi's Google Lens API, Google Maps API (Future Extension), and Google Search API (Future Extension)

👉 See project details

👉 Github

👉 See it live

What Developers Are Building With Search Data

The projects built at API World were very different from one another, but they had something in common:

Developers were using search as a building block for something bigger

Some projects focused on AI and agents, using search results to give applications access to current information. Others used search data for automation, research, data collection, and discovery.

And because SerpApi returns structured search results in JSON and Markdown, developers can take that data and plug it directly into the applications and workflows they're already building.

Check out all the projects developers built with SerpApi at API World here:

DevNetwork [API + Cloud + AI] Hackathon 2026

The projects built at API World are just a small sample of what's possible.

Build with SerpApi

For us, the most exciting part wasn't simply seeing developers call an API and build a tool using it. It was seeing what they built once they had access to real search data.

Whether you're building an AI agent, research tool, automation, data pipeline, or something completely different, live search results can be an important source of fresh information for your application.

And you don't have to build the search infrastructure yourself.

With SerpApi, developers can access Google and other search engines through an easy and fast API and work with the results in a format that's ready to use.

Have an idea? Start building .

Building something useful with SerpApi? Share it with us.

Interested in exploring our use cases? Check out our use cases page .

Read Some Of Our Latest Posts

Introducing Review Audit: An open-source tool to check how real a Google Maps rating is

Cagdas Salur — Sun, 11 Oct 2026 21:37:48 +0000

Review Audit reads a Google Maps place's newest reviews, pulls the full Google history of the reviewers worth a closer look, and tells you how much of the rating the place's own reviewers can explain. What they can't explain gets set aside, and you see the rating without it. It is open source, runs on your machine with your own SerpApi key, and costs about 42 credits a place.

GitHub - serpapi/reviewaudit: Is this 4.8 real? Reads a Google Maps place’s newest reviews through SerpApi and shows the patterns worth a closer look: first-time reviewers, five-star bursts, reviews posted minutes apart. Runs locally on your own API key.

What an audit looks like

Each place you read gets one page: the verdict, the rating without the reviews it can't explain, and the evidence.

An audit of Gull Rock Dinner Cruise, an invented place from the demo

Every place in these screenshots is invented. Review Audit ships a demo that puts made-up businesses with made-up reviews on real Portland, Maine streets, so the images show the real tool without pointing at a real business. More on that at the end.

Why you can't spot a fake review by reading it

Every business with a rating has an incentive to improve it, and there is a market that will help. On the place page, a review that was paid for or asked for looks exactly like a real one: five stars, a name, a date.

They don't look the same in aggregate. Padded reviews tend to come from accounts that have never reviewed anything else. They arrive together, in office hours, minutes apart. They say the same thing in the same words. None of that is visible from one review, and all of it is visible from 200 of them plus the history of the people who wrote them. That is what Review Audit reads.

What it looks for

Each pattern is measured against the place itself, not against a global rule. A busy lunch spot that gets 30 reviews on a Saturday is not suspicious; a dentist that gets 30 in a week when its normal week is four is.

First-time reviewers

Reviewers with no record like it more than anyone else. Review Audit compares how often first-time accounts give five stars against accounts with 4 to 50 reviews, at the same place. The gap is the finding, not the count. A famous place gets plenty of first reviews; a place where first reviews are five stars far more often than everyone else's has something to explain.

A week with too many five-star reviews

A week carried far more five-star reviews than the place gets. Judged against the place's own median week, and flagged only when it is both at least 2.5 times normal and very unlikely by chance.

Five-star reviews minutes apart

Five-star reviews arrive in batches, minutes apart. More pairs within ten minutes than the hours people actually post at would predict, so a lunch rush is not mistaken for a batch upload.

The same waiter, over and over

Reviews that name the same member of staff, posted in pairs after dinner by people with the same surname, are a different kind of padding: reviews asked for at the table.

Repeated wording and shared reviewers

Reviews that say the same thing in the same words, and groups of accounts that keep reviewing the same places. Two accounts that share other places are usually a couple. Review Audit needs three.

Almost no four-star reviews

Nobody ever gives it four stars. Real places collect a four-star tail: the customer who enjoyed it but waited too long. A wall of fives with nothing beside it means someone is choosing who gets asked.

What it does with what it finds

None of these condemns a single review on its own. What gets set aside is only the excess over what the place's own reviewers explain, and every review set aside is listed with a link back to Google, so you can disagree.

The audits so far, most padded first

What the reviews say

Below the findings, each audit also reads the reviews for what they say: what people praise and what they complain about, food/service/atmosphere sub-ratings with and without the set-aside reviews, the hours people post at, where else the reviewers go, and how often the owner replies.

What the reviews say: praise, complaints, sub-ratings and the hours people post at

Each audit is a single self-contained HTML file, so it can be kept, compared and shared.

Try it in 5 minutes

To install Review Audit, you can use uv tool install command as follows:

uv tool install git+https://github.com/serpapi/reviewaudit
reviewaudit serve

That opens the app. Paste your key from serpapi.com/manage-api-key, search for a place, and press Read the reviews. The free plan's 250 searches a month read about five places.

Searching Google Maps from the app

The read runs in the background and shows each stage, so you can watch where the credits go:

A read in progress, one stage at a time

There is a CLI too:

export SERPAPI_KEY=...
reviewaudit "<place name> <city>"

How it uses SerpApi

Three SerpApi engines, about 42 calls:

step	engine	what it gives
1	Google Maps API	the place, its photo, hours and Google's lifetime star counts
2	Google Maps Reviews API ×10	the 200 newest reviews, each with the reviewer's review count, Local Guide status, photos, sub-ratings and an exact timestamp
3	Google Maps Contributor Reviews API ×30	everything the reviewers worth checking have ever reviewed

The cheap checks run first, so the 30 paid record lookups go to the reviewers they point at. Every response is cached, so re-reading a place or rebuilding its page costs nothing.

If you build on these APIs yourself

A few things worth knowing:

The contributor engine returns relative dates only ("a week ago"); the reviews engine has iso_date. So timing patterns come from the place's reviews, and a reviewer's record is used for what they reviewed, not when.
user.reviews on a review counts written reviews. The full record also has rating-only entries, so an account that says "1 review" can have 19 ratings behind it. Review Audit trusts the record.
The first page of reviews is 8, the rest 20.

What it can't tell you

It reports patterns worth a closer look, not verdicts on individual reviews, which is why every verdict reads "looks". It also has blind spots, and each audit prints them:

A place where everyone gives five stars looks the same whether they mean it or not. A flat 95% is not evidence; the gradient across account depth is.
A first review at a famous place is normal. A thin account alone never counts.
Hotels carry Tripadvisor and Trip.com reviews with no Google account behind them. Those are left out, and the page says how many.
200 reviews at a very busy place can be two weeks. The audit says so when the window is short, and --reviews 600 reads further back.

About those screenshots

Running a tool like this against real businesses and publishing the results is a different thing from building it. So the repo includes tools/demo.py: a fake Google Maps that answers the same three engines in the same shapes, for businesses that don't exist (every name was checked against Google Maps) on real Portland streets, with invented reviews and reviewers. The padding is written in on purpose, a burst here, a waiter named over and over there, and the real Review Audit finds it. Every image in the README file and in this post comes from it.

Get the code

The code is on GitHub. Issues and pull requests are welcome, especially new patterns with a rationale behind them, and staff-name and stop-word lists for languages beyond English and Turkish.

If you don't have a key yet, the free plan is 250 searches a month, enough for about five places.

Introducing SerpApi's OpenCode Plugin

Srujana Maddula — Tue, 06 Oct 2026 10:03:36 +0000

OpenCode is an open-source, model-agnostic AI coding agent that runs locally on your machine. It can edit files, run terminal commands, and work across multiple repos at once. Because it's model-agnostic, a lot of developers pair it with local LLMs like Ollama or LM Studio, so their code and data never leave their machine.

But OpenCode can't search the web for live data unless you give it access. You can do that with the SerpApi OpenCode plugin. It gives OpenCode access to Google, YouTube, Amazon, Walmart, and dozens of other search engines through a single tool call.

What is an OpenCode Plugin?

Plugins and Skills sound similar, but they do different jobs. Skills automate boring workflows, tasks where you'd otherwise repeat the same instructions every time. Plugins give OpenCode a new capability it didn't have before, a new tool, feature, or integration with an outside service.

SerpApi has released one such plugin, giving OpenCode real-time web search. It provides access to multiple SerpApi engines like Google, Bing, Yahoo, YouTube, and many others. It also ships with skills that guide OpenCode in selecting appropriate engine and parameters for each query.

So when a task needs live web data, OpenCode reaches for the SerpApi plugin, runs the search, and returns the results. Say OpenCode needs to find a current public GitHub repository for a coding task. It searches for it with the SerpApi plugin instead of guessing from training memory.

Why Use SerpApi Opencode Web Search Plugin?

Without the plugin, giving OpenCode web access means building a separate API integration yourself: handling auth, setting parameters, and parsing responses.

The SerpApi OpenCode plugin replaces all of that with one integration. Install it once, and OpenCode gets dedicated access to specific engines and search verticals instead of just generic web search.

Here are some of the popular engines the plugin supports:

Vertical	Supported engines	Use cases
Web Search	Google, Bing, DuckDuckGo, Yahoo, Yandex, Baidu, Naver	Full-featured web search with knowledge graph, ads, and SERP extras
AI Search	Google AI Mode, Google AI Overview, Bing Copilot, Brave AI Mode	AI search results, brand monitoring in AI results, and AI-generated summaries
Shopping	Amazon, Walmart, eBay, Google Shopping, Home Depot	Product search, listings, and pricing
Local	Google Maps	Local business listings and place reviews
News & Trends	Google News, Google Trends	Current news articles and search trend data
Media	Google Images, Google Videos, Google Lens, YouTube	Image search, video search, reverse image search, and video rankings
Travel	Google Flights, Google Hotels, Google Travel Explore	Flight deals, hotel reviews, and travel destination search
Jobs, Finance & Apps	Google Jobs, Google Finance, Google Play, Apple App Store	Job search, stock tracking, finance data analysis, and app store data
SEO	Google Autocomplete	Keyword research

Each engine returns data shaped for the question you're asking. For instance, searching Amazon returns product listings and pricing in clean JSON. You also don't need to remember engine names or query formats. OpenCode picks the right engine and parameters based on the query.

Installing the OpenCode SerpApi Plugin

The OpenCode SerpApi plugin is available as an npm package, so setup is really just a config change plus an API key.

1. Add the plugin to your config

In your project root, open opencode.json (or opencode.jsonc). If neither file exists, create opencode.json. Then add opencode-serpapi to the plugin array:

{
  "$schema": "https://opencode.ai/config.json",
  "plugin": ["opencode-serpapi"]
}

Want it available across all your projects instead of just one? Add the same block to your global config at ~/.config/opencode/opencode.json instead.

OpenCode installs it automatically from npm the next time you run opencode in that project; doesn’t require a separate npm i command.

2. Set your SerpApi API key

Grab a key from your SerpApi dashboard and export it as an environment variable:

export SERPAPI_API_KEY="add_your_api_key"

3. Start OpenCode

On startup, OpenCode caches the plugin and registers the serpapi_search tool for your session.

Ask OpenCode something that needs a live web search, e.g.:

"Search Google for the latest news on the Fed rate decision"

It returns live search results, as shown in the image below:

OpenCode web search plugin

How OpenCode uses SerpApi under the hood?

The plugin has three parts, each with a clear job: a connection to SerpApi's MCP server, a native search tool as a backup, and a skill that tells OpenCode which search engine to use.

1. The MCP connection

When OpenCode starts, the plugin automatically registers SerpApi's hosted MCP server as a remote MCP server. You don't need to add it to your config by hand.

The plugin reads your SERPAPI_API_KEY environment variable and sends it as an Authorization: Bearer header with each request.

The server then checks the engine and its parameters, and runs the search. When OpenCode doesn't need the full JSON, it sets the engine's output parameter to Markdown, which uses about half the tokens for the same task.

2. The serpapi_search fallback

If the MCP server isn't available, OpenCode can still search. The plugin exposes a native serpapi_search tool that calls SerpApi's search.json endpoint directly, using the runtime's built-in fetch. The tool takes a single q argument and maps it to the field each engine expects.

3. The serpapi-web-search skill

The connection gives OpenCode access to various engines. The skill tells it which one fits the task, using bundled routing guidance and engine reference files.

Ask OpenCode to find standing desks under $300 on Walmart, and the skill points it to the Walmart engine and its parameters. Without it, OpenCode might run a generic Google search and get blog posts and ads instead of product listings.

Conclusion

This SerpApi plugin makes web search a native capability for OpenCode. It's open source, and setup just takes a config change and an API key.

Check out theGitHub repo if you want to see how it's built, report an issue, or request a feature.

We are continuing to expand SerpApi integrations to make sure AI can access live search data, whether that is through MCP, Claude Code, and many other AI integrations. If you dont find a SerpApi integration for the tool you work in, contact us at contact@serpapi.com

Comparing Markdown Search Results: SerpApi vs. Exa, Tavily, and Firecrawl

Tomás Murúa — Mon, 05 Oct 2026 01:52:21 +0000

Tavily, Exa, Firecrawl, and SerpApi all return markdown for LLMs, but they aren't converting the same thing. Tavily, Exa, and Firecrawl convert the pages a search found. SerpApi converts the search itself, so you get the results page from the real search engine as clean markdown. Markdown Output works the same way across all of our 100+ search APIs.

This blog runs the same three queries through all four providers, using SerpApi's Google Search API as the baseline. It compares where the markdown lives in each response, what it costs in tokens, and what a model can answer from it. By the end, you'll know which output fits your agent: the results page, a list of links, or the pages behind them.

We've already compared positioning and pricing in SerpApi vs. Firecrawl, SerpApi vs. Exa, and SerpApi Alternatives: Best Web Search APIs. This one is only about the output.

How We Compared

Every provider got the same query on its playground defaults, on a free plan, with generated answers turned off. The only settings we changed are the content modes named in each table and the location. We set the location as close to Austin, Texas, as each provider allows, which meant the United States for Tavily and Exa, and Austin for Firecrawl. The SerpApi baseline is Google Search with location=Austin, Texas, United States, gl=us, and hl=en, the same defaults as the playground. We also set no_cache=true so every SerpApi search was a live fetch.

We ran three queries, each one testing something different:

Query	What it tests
`coffee`	SERP features. A one-word query where Google shows most of them at once
`how to make delicious coffee`	Semantic question. The kind of question an agent forwards from a user
`grok 4.7`	Freshness. Announced 45 minutes before capture, so it shows who searches live and who reads from an index

Token counts use tiktoken with the o200k_base encoding on the full saved response, JSON or markdown, nothing stripped. Each provider section shows a trimmed copy of its response.

Test 1: SERP Features

Here is every mode side by side for coffee.

Provider	Mode	Tokens	What the tokens buy
SerpApi	Google Search, JSON	66,237	Full SERP as structured objects
SerpApi	Google Search, markdown	13,025	Full SERP as a document
Tavily	`basic`, snippets only	5,283	10 titles, URLs, snippets
Tavily	`basic`, `include_raw_content=markdown`	90,234	8 titles plus 7 page bodies
Tavily	`advanced`, `include_raw_content=markdown`	104,792	10 titles plus 10 page bodies
Exa	bare	701	10 titles and URLs
Exa	`highlights`	14,172	10 titles plus extracted passages
Exa	`text`	133,343	10 titles plus 10 page bodies
Firecrawl	search only	8,747	10 titles, URLs, descriptions
Firecrawl	search + scrape, markdown	130,041	10 titles plus 9 page bodies

Two things to read from this table. The cheapest rows carry almost nothing, 10 links and at most a snippet each. Among the responses a model can actually answer from, SerpApi markdown is the smallest at 13,025 tokens. The sections below show one of these responses from each provider.

SerpApi

SerpApi scrapes the Google results page in real time and returns it as JSON or as markdown. The Markdown Output page covers how to use it, and our video on supercharging AI agents with real-time data shows it inside an agent loop. Here we only look at the coffee response.

The markdown is not a dump of the JSON. It is the results page curated for an LLM, with one section per SERP feature in the order Google showed them. The local pack and organic results become tables with links, and the knowledge graph becomes a list of its facts. Nothing inside the linked pages is fetched; what you get is what a person in Austin sees when they type coffee.

This is the opening of the Google Search markdown document from the table:

---
search_metadata:
  id: 6aa9bc62b9dbc418dd7b473f
  status: Success
  json_endpoint: "https://serpapi.com/searches/.../6aa9bc62b9dbc418dd7b473f.json"
  markdown_endpoint: "https://serpapi.com/searches/.../6aa9bc62b9dbc418dd7b473f.md"
  created_at: "2026-09-15T21:45:06.935Z"
search_parameters:
  engine: google
  q: Coffee
  location_requested: Austin, Texas, United States
  hl: en
  gl: us
---

## Search Information

- Query Displayed: Coffee
- Total Results: 214
- Organic Results State: Results for exact spelling

## Local Map

[![Image](https://serpapi.com/searches/6aa9bc62b9dbc418dd7b473f/images/M0vMWN9e2SQWmXNdUKaKlQ.png)](https://www.google.com/search?q=Coffee&...&udm=1)

## Local Results

### Places

| Position | Title | Type | Rating | Reviews | Price | Description | Thumbnail | Thumbnail Large | Links Phone | Links Website | Place Id | Gps Coordinates Latitude | Gps Coordinates Longitude | Address | Hours | Phone |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | Houndstooth Coffee | Coffee shop | 4.5 | 1210 | $1–10 | "Excellent coffee, offer singe source and blends, the options always rotate" | ![Thumbnail](https://serpapi.com/searches/6aa9bc62b9dbc418dd7b473f/images/-WgD2tu7k8gkCk2xfEgTRbF9GSQB9Snw18rTTBABMPE7QjI8PtOIbqd0wV7X81n3.jpeg) | https://lh3.googleusercontent.com/gps-cs-s/AHRPTWnWTvhbq7JcIcVUxEmUhGnD40dIdSQbygpfH9KvFfmvSN9Hix8E60n3riRiG648DFrJz7Z8raNZfxcVWY5UbcOHkCyE56MPZoVV2rslKvPAVcTTZkyfbbVHBGYpciBgoKjzCcrVFA=w1000-h1000-c-n | tel:(512) 394-6051 | http://www.houndstoothcoffee.com/?utm_source=google&utm_medium=organic&utm_campaign=googlebusiness | 11265938073076301333 | 30.266216 | -97.743066 | 401 Congress Ave. #100c, Austin, TX 78701 | Open · Closes 7 PM | (512) 394-6051 |

<!-- 2 more places, then 8 more sections, 13,025 tokens in total -->

This is the results page: SerpApi returns what Google ranked, grouped, and labeled. The local pack, the knowledge graph, and the AI Overview exist here and nowhere else in this comparison.
A fifth of the JSON: Markdown drops the JSON brackets and keys and keeps the text a model reads, which is the reason the feature exists.
Trim what you don't need: The JSON Restrictor keeps only the sections you ask for, and it also works with Markdown output. For example, json_restrictor=organic_results keeps only the organic results.
Every search has an ID and a URL: search_metadata.id and markdown_endpoint are in the frontmatter. Open the URL later and you get the same document back.

Tavily

Tavily is a search API built for agents. A basic search for coffee returns 10 results with a title, a URL, a snippet in content, and a relevance score. There is no markdown yet. The markdown appears when you set include_raw_content to markdown. It lands inside each result as raw_content, the converted body of that result’s page. The response stays JSON and the markdown is a string field inside it.

This is the first result of the basic search with include_raw_content=markdown:

{
  "query": "coffee",
  "results": [
    {
      "url": "https://en.wikipedia.org/wiki/Coffee",
      "title": "Coffee",
      "content": "Coffee is a beverage brewed from roasted ground coffee beans. Darkly colored and bitter, coffee has a stimulating effect on humans due to its caffeine content, [...]",
      "score": 0.83742094,
      "raw_content": "[Jump to content](#bodyContent)\n\n[![](/static/images/icons/enwiki-25.svg)  ![Wikipedia](/static/images/mobile/copyright/wikipedia-wordmark-en-25.svg) ![The Free Encyclopedia](/static/images/mobile/copyright/wikipedia-tagline-en-25.svg)](/wiki/Main_Page)\n\n[Search](/wiki/Special:Search \"Search Wikipedia [f]\")\n\n## Contents\n\n* [1 Etymology](#Etymology)\n* [2 History](#History)\n  + [2.1 Legendary accounts and myths](#Legendary_accounts_and_myths)",
      // raw_content continues for 242,608 characters
      "favicon": "https://en.wikipedia.org/static/apple-touch/wikipedia.png",
      "id": "fe77ba-00"
    }
    // 7 more results, raw_content present on 7 of 8
  ],
  "response_time": 1.43,
  "request_id": "6e4056c2-7185-4d66-ba80-44fe823d7392"
}

raw_content is the page, not the SERP: For coffee the first result is Wikipedia, and raw_content starts with Wikipedia’s navigation links and table of contents. This response has no local pack, knowledge graph, or People Also Ask.
The pages are the cost: The snippets-only response is 5,283 tokens. The extra tokens in the other Tavily rows are the page bodies it adds.

Exa

Exa searches its own neural index of the web. The bare search returns 10 results with id, title, url, publishedDate on some, and no snippet at all. Page content comes through two opt-in fields. The highlights field is an array of passages Exa extracted from each page. The text field is the full page body, which Exa returns as markdown-style text by default.

This is the first result of the text search:

{
  "requestId": "6e0760b35bd59e7fd1699bc92570c7ff",
  "results": [
    {
      "id": "https://en.wikipedia.org/wiki/Coffee",
      "title": "Coffee",
      "url": "https://en.wikipedia.org/wiki/Coffee",
      "text": "Coffee\n\nCoffee\n\n| Latte and black filtered coffee | |\n| --- | --- |\n| Type | Usually hot; can be iced |\n| Origin | Yemen [1] [2] [3] |\n| Introduced | 15th century |\n| Color | Black, dark brown, light brown, beige |\n| Flavor | Distinctive, somewhat bitter |\n| Ingredients | Roasted coffee beans |\n| Standard drinkware | Mug |\n\nCoffee is a beverage brewed from roasted ground coffee beans.",
      // text continues for 102,126 characters
      "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/1/19/Coffee_beans_unroasted.jpg/250px-Coffee_beans_unroasted.jpg"
    }
    // 9 more results
  ],
  "searchTime": 880.4,
  "costDollars": {
    "total": 0.007,
    "search": { "neural": 0.007 }
  }
}

text is the page, not the SERP: The Wikipedia result keeps its infobox as a markdown table, but nothing from Google's results page is in it.
text is the largest response in this comparison: Exa inlines the full body of all 10 pages.

Firecrawl

Firecrawl is a page scraper with a /search endpoint on top. With scraping off, the search returns 10 web results with url, title, and description. That is 8,747 tokens and no markdown. With “scrape content from search results” on and the format set to markdown, each result gains a markdown field holding the main content of the page. It also gets a metadata object with the page’s own title, description, and status code.

This is the first result of search + scrape with markdown format:

{
  "web": [
    {
      "url": "https://en.wikipedia.org/wiki/Coffee",
      "title": "Coffee - Wikipedia",
      "description": "For other uses, see [Coffee (disambiguation)](https://en.wikipedia.org/wiki/Coffee_(disambiguation)).",
      "position": 1,
      "markdown": "[Jump to content](https://en.wikipedia.org/wiki/Coffee#bodyContent)\n\n[![Page semi-protected](https://thumb.wikimedia.org/wikipedia/en/thumb/1/1b/Semi-protection-shackle.svg/20px-Semi-protection-shackle.svg.png)](https://en.wikipedia.org/wiki/Wikipedia:Protection_policy#semi \"This article is semi-protected.\")\n\nFrom Wikipedia, the free encyclopedia\n\nBrewed beverage\n\nThis article is about the beverage. For other uses, see [Coffee (disambiguation)](https://en.wikipedia.org/wiki/Coffee_(disambiguation) \"Coffee (disambiguation)\").",
      // markdown continues for 249,433 characters
      "metadata": {
        "title": "Coffee - Wikipedia",
        "sourceURL": "https://en.wikipedia.org/wiki/Coffee",
        "statusCode": 200,
        "cachedAt": "2026-09-15T15:48:22.840Z",
        "cacheState": "hit",
        "creditsUsed": 1
        // 19 more keys, including og:* tags copied from the page head
      }
    }
    // 9 more results, markdown present on 9 of 10
  ],
  "creditsUsed": 11
}

Pages can come from a cache: The Wikipedia page above has cacheState: hit, and its cachedAt shows Firecrawl stored it about seven hours before our request. For coffee, 6 of the 9 pages with markdown came from that cache.
Search plus scrape is the expensive path: It costs about 10 times the tokens of the SerpApi markdown for the same query, because the response includes 9 full pages.

Test 2: Semantic Question

Now let’s test a question a user would ask an agent, how to make delicious coffee. Google shows a different page for it. The local pack and knowledge graph are gone, and in their place are recipes and videos. The related questions and the AI Overview are still there. The other three providers still return the top pages.

Tavily, Exa, and Firecrawl return the same kind of response we saw for coffee, page bodies inside JSON, so here we only show SerpApi. The part to look at is the AI Overview, Google’s own answer to the question. It comes in the markdown with its sources:

## Ai Overview

### Text Blocks

1. To make tasty coffee at home, start with freshly ground whole beans, a precise coffee-to-water ratio, and filtered water heated to the optimal temperature.
2. Essential Rules for Great Coffee

<!-- 7 more text blocks -->

### References

| Source | Index | Title | Snippet | Thumbnail |
| --- | --- | --- | --- | --- |
| thekitchn.com | 2 | [https://www.thekitchn.com/best-coffee-brewing-method-22977756](https://www.thekitchn.com/best-coffee-brewing-method-22977756) |  |  |
| coffeechronicler.com | 3 | [https://coffeechronicler.com/best-way-to-make-coffee-at-home/](https://coffeechronicler.com/best-way-to-make-coffee-at-home/) |  |  |

<!-- 8 references in total, then Organic Results, Related Searches, Pagination -->

An agent can quote this answer or open one of its sources. The other providers can’t return it, because the AI Overview only exists on Google's results page, not on any of the pages they fetch.

For a how-to question, SerpApi returns Google's answer and its sources in 11,266 tokens. The next smallest full-page response was Exa's, at 32,772 tokens.

Test 3: Freshness

The grok 4.7 query is the freshness test. xAI announced it on X, and we ran the query 45 minutes later. By then Google had indexed the announcement, but most of the web had not. A provider searching live should return the release. A provider reading from an index returns whatever it crawled last.

SerpApi's response is about a third the size of the others, but the bigger difference is the content. SerpApi returned Google's results page with the xAI announcement first and the launch coverage in Top Stories. All 10 of Tavily's results were written before the launch, so a model reading them would answer that Grok 4.7 is not out yet. Exa ranked the announcement and Vercel's changelog first, then filled the rest with pre-launch speculation. Firecrawl found three posts about the release, and only the two Cursor forum threads came back with content.

For breaking news, SerpApi returned the most coverage of the release. Eight of the 11 results across its organic results and Top Stories were about the launch.

Results Across All Three Queries

On every query, SerpApi's markdown was the smallest response a model could answer from. It used 10,209 to 13,025 tokens, while the full-page modes in the chart used 30,802 to 133,343. The gap is widest on coffee, where all three full-page responses include the Wikipedia article on coffee.

Location and Traceability

The tests above compared what each provider returned. Two differences don't show up in a single response, but they matter once an agent runs in production.

Customizing the Search

Each SerpApi engine exposes the main parameters its search engine offers, and the markdown follows them. For Google Search that means you can pin a query to what a specific user would see.

Parameter	What it controls
`location`	City, state, or neighborhood the search comes from
`gl`	Country of the results
`hl`	Interface language
`google_domain`	Which Google domain to query, such as `google.co.uk`
`device`	Desktop, tablet, or mobile layout

The other three providers accept at best a country or a city. Each engine lists its parameters in its docs, for example the Google Search API docs.

Trace Every Search

Every SerpApi response, JSON or markdown, opens with search_metadata. It has the search id, a json_endpoint, a markdown_endpoint, and the created_at timestamp. The Search Archive API keeps each search for 31 days, so the same id returns the same results page later, in either format. Every SerpApi excerpt in this post came from an archived search.

For an agent, a log line with a search id is a record of exactly what the model read. If a user asks why the model said something, you can open the search and check. Tavily returns a request_id and Exa a requestId, and neither can fetch that search again. Firecrawl returns a search id, plus a scrapeId and cacheState for each page, but its API documents no way to fetch the search again. With those three, the only record is the copy you kept.

When to Use Which

The question is whether your agent needs the search or the pages behind it.

Tavily, Exa, and Firecrawl aren't doing anything wrong. Returning pages is the product, and for a research agent that needs to read every source, it is the right product. The token counts above are the size of those pages, not a flaw.

The table below shows each case in more detail, with the reason behind each pick:

You need	Pick	Because
What Google shows for a query, including local pack, knowledge graph, and AI Overview	SerpApi markdown	The only response here with those sections, and the smallest one a model can answer from
Results for something that happened this hour	SerpApi markdown	A live Google search, with Google's relative dates on most rows
Results as a user in a specific city, country, or device would see them	SerpApi markdown	`location`, `gl`, `hl`, and `device` pin the search down to the neighborhood
A record of what the model read	SerpApi markdown	Every search keeps an `id` and a URL for 31 days
Fields to parse in code, such as ratings, prices, or positions	SerpApi JSON	Typed fields, no markdown to parse
A short list of URLs to decide what to fetch next	Tavily `basic`, Exa bare, Firecrawl search only	Under 9,000 tokens each. Nothing to answer from, but nothing wasted either
The full body of the pages behind a query	Tavily `include_raw_content`, Exa `text`, Firecrawl search + scrape	They return the pages. Expect 30,000 to 135,000 tokens, and clean the navigation yourself.

Conclusion

Any agent that needs to know what the web says right now, where a user would see it, and exactly what the model read needs the search before it needs the pages. SerpApi's Markdown Output returns that search as one document, from Google Search or any of our other 100+ search APIs.

How to Do Amazon Keyword Research with Python

Noraina Nordin — Sat, 03 Oct 2026 09:41:12 +0000

Amazon keyword research starts with one question: what do shoppers actually type into the search bar? The most direct answer is Amazon's own autocomplete. When you type "coffee b", Amazon suggests "coffee beans", "coffee bean grinder", and "coffee bar accessories". Those suggestions are coming from real search behavior.

Amazon Homepage

Most free Amazon keyword research tools are built on the same data, but they limit how many lookups you can run, and it's hard for you to integrate them into your own workflow. In this tutorial, you'll use the Amazon Autocomplete API from SerpApi to get these suggestions as a clean JSON or Markdown output, expand one seed keyword into dozens of long-tail terms, and save them to a CSV file.

What data does the Amazon Autocomplete API return?

Each request returns a suggestions array. Each suggestion includes:

value: the suggested search term, for example coffee beans.
type: the suggestion type. KEYWORD is a regular search suggestion. WIDGET is a grouped block, such as a price-range shortcut.
items: for WIDGET suggestions, a list of options, each with its own value and amazon_link.
thumbnail: a small product image for the suggestion, when Amazon shows one.
amazon_link: the Amazon search results URL for that suggestion.
serpapi_link: a ready-made SerpApi request that gets autocomplete suggestions for this suggestion. Useful for going one level deeper.
serpapi_amazon_link: a ready-made Amazon Search API request for this suggestion, so you can see which products rank for it.

Here's the result from the trimmed response for coffee in JSON

{
  "suggestions": [
    {
      "value": "Coffee table by price",
      "type": "WIDGET",
      "items": [
        {
          "value": "Under $100",
          "amazon_link": "https://www.amazon.com/s?k=coffee+table&amp=&rh=n%3A1063318%2Cp_n_price%3A8570764011&amp=&rnid=8570763011&amp=&nav_sdd=aps&amp=&crid=26CJG6W215CUX&amp=&sprefix=coffee&amp=&ref=nb_sb_ss_w_sbl-tr-t1_k0_1_6_0"
        },
        ...
        ...
        ...
      ]
    },
    {
      "value": "coffee table",
      "type": "KEYWORD",
      "thumbnail": "https://m.media-amazon.com/images/I/41Q0IEeqGBL.__AC_SL75__.jpg",
      "serpapi_link": "https://serpapi.com/search.json?amazon_domain=amazon.com&engine=amazon_autocomplete&k=coffee+table",
      "serpapi_amazon_link": "https://serpapi.com/search.json?amazon_domain=amazon.com&device=desktop&engine=amazon&k=coffee+table",
      "amazon_link": "https://www.amazon.com/s?k=coffee%20table"
    },
    ...
    ...
    ...
 }

SerpApi returns Amazon's search suggestions in a structured output

Getting started with SerpApi

You need a free SerpApi account before you can use the API. You can upgrade to a paid plan later if you need more searches, faster speeds, or additional features.

First, create an account and verify your email. Then grab your API key from your account dashboard. The free plan includes 250 searches per month, and you can try any query live in the interactive playground before writing code.

SerpApi API key dashboard

You should store your API key in a safe location if you're sharing or publishing your code. If it's ever leaked, you can generate a new one from the dashboard. The examples below read the key from an environment variable.

Install the SerpApi library (optional)

SerpApi has official libraries for Python, JavaScript, Ruby, Java, and more. They're a thin wrapper around the API and aren't required. The API works just as well with plain GET requests, cURL, or fetch() in Node.js.

For the Python examples, install the official client:

pip install serpapi

Review the Amazon Autocomplete API documentation

The engine is amazon_autocomplete, and the text you'd type into Amazon's search bar goes in the k parameter. Other useful parameters:

amazon_department: limit suggestions to one department. Defaults to aps (All Departments). See the full list of Amazon departments.
amazon_domain: the Amazon marketplace to use, such as amazon.co.uk or amazon.de. Defaults to amazon.com. See the supported Amazon domains.
output: set to md for a token-efficient Markdown response.

For the full field reference and live examples, see the Amazon Autocomplete API documentation.

SerpApi's Amazon Autocomplete API documentation

How to scrape Amazon autocomplete suggestions

The results are identical across every library, so use whichever method fits your stack.

GET request

This gets suggestions for "coffee":

https://serpapi.com/search.json?engine=amazon_autocomplete&k=coffee&api_key=SERPAPI_API_KEY

By default you get JSON, but you can request Markdown by adding output=md (or using the /search.md endpoint, or the Accept: text/markdown header). Markdown returns the data in a more token-efficient format built with tables and links, which is handy when feeding results to an LLM or AI agent:

https://serpapi.com/search.json?engine=amazon_autocomplete&k=coffee&output=md&api_key=SERPAPI_API_KEY

Python

This gets suggestions for "coffee" and prints each keyword with its type, using the official SerpApi Python library:

import os
import serpapi

client = serpapi.Client(api_key=os.environ["SERPAPI_API_KEY"])

results = client.search({
    "engine": "amazon_autocomplete",
    "k": "coffee",
})

for suggestion in results.get("suggestions", []):
    print(f"{suggestion['value']} | {suggestion['type']}")

Output:

Coffee table by price | WIDGET
coffee table | KEYWORD
coffee pods | KEYWORD
coffee | KEYWORD
coffee maker | KEYWORD
coffee creamer | KEYWORD
coffee beans | KEYWORD
coffee grinder | KEYWORD
coffee filters | KEYWORD
coffee bean grinder | KEYWORD
coffee machine | KEYWORD

JavaScript and Node.js

This runs the same request with the SerpApi JavaScript library:

import { getJson } from "serpapi";

const results = await getJson({
  engine: "amazon_autocomplete",
  api_key: process.env.SERPAPI_API_KEY,
  k: "coffee",
});

for (const suggestion of results.suggestions ?? []) {
  console.log(`${suggestion.value} | ${suggestion.type}`);
}

cURL

This gets suggestions straight from the command line:

curl --get https://serpapi.com/search \
 -d api_key="SERPAPI_API_KEY" \
 -d engine="amazon_autocomplete" \
 -d k="coffee"

Filter suggestions by Amazon department and marketplace

The suggestions for a broad word mix with unrelated products. For example, "coffee" in All Departments, Amazon suggests "coffee table" and "coffee grinder" alongside "coffee pods".

You can set amazon_department to narrow the results. With amazon_department=grocery, the same query returns only food and drink terms: "coffee pods", "coffee creamer", "coffee syrup", "coffee k cups", "coffee mate", and "coffee substitutes".

https://serpapi.com/search.json?engine=amazon_autocomplete&k=coffee&amazon_department=grocery&api_key=YOUR_API_KEY

To research another marketplace, use amazon_domain with any of the supported Amazon domains. Suggestions reflect what shoppers search for on that site:

https://serpapi.com/search.json?engine=amazon_autocomplete&k=coffee&amazon_domain=amazon.co.uk&api_key=YOUR_API_KEY

Build a free Amazon keyword research tool with Python

Each request returns about 10 suggestions. To build a real keyword list, use the same trick as most Amazon keyword suggestion tools by adding each letter of the alphabet after your seed keyword. For example, "coffee a", "coffee b", "coffee c", and so on; each returns a different set of long-tail suggestions.

Here's what the script below does:

Runs the seed keyword plus one query per letter (27 queries in total)
Keeps only KEYWORD suggestions and skips WIDGET blocks
Removes duplicates
Saves the list to amazon_keywords.csv, with the query that found each keyword

import csv
import os
import string

import serpapi

client = serpapi.Client(api_key=os.environ["SERPAPI_API_KEY"])

SEED = "coffee"
DEPARTMENT = "grocery"   # use "aps" for All Departments
MAX_QUERIES = 27         # 1 seed + 26 letters; each query uses one search


def get_suggestions(query):
    results = client.search({
        "engine": "amazon_autocomplete",
        "k": query,
        "amazon_department": DEPARTMENT,
    })
    return [
        s["value"]
        for s in results.get("suggestions", [])
        if s.get("type") == "KEYWORD"
    ]


queries = [SEED] + [f"{SEED} {letter}" for letter in string.ascii_lowercase]
queries = queries[:MAX_QUERIES]

keywords = {}
for query in queries:
    for keyword in get_suggestions(query):
        keywords.setdefault(keyword.lower(), query)

with open("amazon_keywords.csv", "w", encoding="UTF-8", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["keyword", "source_query"])
    for keyword, source in sorted(keywords.items()):
        writer.writerow([keyword, source])

print(f"Ran {len(queries)} queries and saved {len(keywords)} unique keywords to amazon_keywords.csv")

💡 MAX_QUERIES caps the number of searches per run. Lower it while testing so you don't use up your monthly searches.

Amazon keywords research output in CSV format

There are a few ways you can extend this tools:

Add modifiers. Besides letters, try words shoppers use to narrow a search, such as "for", "best", "organic", or "with".
Go one level deeper. Run the script again with a top suggestion as the new seed, for example "coffee beans".
Compare marketplaces. Add amazon_domain and run the same seed on amazon.com and amazon.co.uk.

Check which products rank for each keyword

A keyword list tells you what shoppers search for. For Amazon product keyword research, the next step is to see what they find. Every KEYWORD suggestion includes a serpapi_amazon_link, which is a ready-made Amazon Search API request for that term. You can call it to get the ranking products, their prices, ratings, and review counts.

For a full walkthrough of the Amazon Search API, pagination, and product details, see our guide on how to scrape Amazon product data below:

Scrape Amazon Product Data (complete 2026 tutorial)

FAQ

Does Amazon have a keyword tool?

Amazon doesn't offer a public keyword research tool for shoppers. Sellers can see some search term data in Seller Central, but the most accessible source of real search terms is the autocomplete dropdown in Amazon's search bar, which is what this API returns.

What are keywords in Amazon?

Amazon keywords are the words and phrases shoppers type into Amazon's search bar. Sellers use them in product titles, bullet points, backend search terms, and ad campaigns so their products show up for those searches.

Is there a free Amazon keyword research tool?

Several free tools exist, but most limit how many lookups you can run. The script in this post is a free alternative for up to 250 searches per month on SerpApi's free plan. Each query uses one search.

Does the Amazon Autocomplete API show search volume?

No. It returns the suggestions Amazon shows, in the order Amazon shows them, but not search volume. Use the suggestions to find which terms exist, then check volume with a separate keyword tool if you need it.

Conclusion

The Amazon Autocomplete API turns Amazon's search suggestions into structured JSON and Markdown output, so you can run Amazon keyword research in bulk instead of typing into the search bar one query at a time. You can then pair it with the Amazon Search API to see which products rank for each keyword, or the Amazon Product API to track prices on the products you find.

If you need help getting started, contact us. We're happy to help.