DEV Community: Viktor

How to Scrape Dribbble with Go: Designer Profiles, Shots, Collections, and a Portfolio Tracker (2026)

Viktor — Mon, 02 Nov 2026 04:05:29 +0000

Dribbble is where a lot of designers keep their portfolio, so it's a natural data source if you're scouting talent, studying which kinds of shots get attention, or keeping an eye on your own numbers. The first thing most people search for is "Dribbble API". There is one. It just doesn't do what most people expect.

So this post covers both: what the official Dribbble API gives you, and how to build a small Dribbble scraper in Go for the public pages it doesn't cover. The scraper uses the standard library plus golang.org/x/net/html, and it will:

Respect robots.txt with a tiny matcher, and stop cleanly at bot challenges
Read a designer profile: name, headline, bio, location, followers, skills
Page through a designer's shots with likes, views, and publish dates
List a designer's collections, and read the shots inside one
Snapshot a few portfolios into CSV and report what changed

Everything below was run on October 30, 2026, and the output is real.

Before you start: Dribbble's Terms of Service list "copying, distributing, or disclosing any part of the Website ... by any automated or non-automated 'scraping'" as a prohibited activity. robots.txt allows the pages used here, but robots rules and terms are different things. Read the terms and weigh the legal risk for your use case before running anything, and prefer the official API wherever it covers what you need. Treat this as a learning project or small-scale research: keep request rates low, don't republish people's work, don't resell the data, and don't use it to mass-message designers. If you only need your own data, use the official API.

Is there a Dribbble API?

Yes: the Dribbble API v2 (https://api.dribbble.com/v2/, docs at developer.dribbble.com). You register an application, do the OAuth flow, and send Authorization: Bearer <token>. The limit is 60 requests per minute and 1,440 per day per authenticated user, with X-RateLimit-* headers and Link header pagination.

The catch is in the endpoint list. v2 has GET /user, GET /user/shots, GET /shots/:id, create/update/delete shot, projects, and jobs. All of them are about the authenticated user. The docs for "Get a shot" say it plainly: "This endpoint returns only shots owned by the currently authenticated user." The old v1 endpoints for any user's shots, followers, and popular lists are gone. Without a token, GET https://api.dribbble.com/v2/user returns 401, and /v2/users/simplebits/shots is a 404.

You want	Use
Your own shots, profile, projects (e.g. to show your Dribbble work on your site)	Dribbble API v2
Other designers' public profiles, shots, likes/views, collections	The public website, which is what this scraper reads

How dribbble.com looks from a scraper's point of view

I checked the site with curl and with Go's HTTP client first. Here is what came back:

path                                            status  notes
/robots.txt                                     200
/simplebits                                     200     ~300 KB, shots embedded as JSON
/simplebits?page=2                              200     next 24 shots
/simplebits/about                               200     bio, location, skills, counts
/simplebits/collections                         200     6 collections per page
/simplebits/collections/4582-Data-Viz           200     24 shots per page
/shots/27755136-Measure-Twice-Print             202     AWS WAF challenge, empty page
/shots/popular, /search/logo                    202     AWS WAF challenge
/shots/popular.rss                              202     AWS WAF challenge
/simplebits.rss, /shots.rss                     404     the old RSS feeds are gone

The status codes were the same with a Chrome User-Agent, curl's default, and Go's Go-http-client/1.1. So three findings shaped the code.

1. robots.txt is clear about what's off-limits. Profiles, collections, and shot pages are allowed. These are disallowed for every crawler:

Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/tags$
Disallow: /*/tags/*
Disallow: /shots/*/comments
Disallow: /shots/*/fans
Disallow: /shots/*/rebounds
Disallow: /shots/*/buckets
Disallow: /shots/week/*   (and /day/*, /month/*, /year/*)

So no follower lists, no who-liked-this lists, no comments. The scraper below checks every path against these rules before it sends a request.

2. Profile and collection pages are server-rendered, and the shots come as JSON. The profile page includes a <script> with var newestShots = [...], one object per shot in the grid:

{"id":27755136,"title":"Measure Twice Print","path":"/shots/27755136-Measure-Twice-Print",
 "published_at":"September 23, 2026","is_rebound":null,"rebounds_count":0,
 "attachments_count":0,"view_count":"23.3k","likes_count":"108","liked":null,"ga":[]}

That's easier and sturdier than scraping the grid's CSS. Logged out, the page shows a "Sign up to continue" button where infinite scroll would be, but the classic ?page=2 link is still in the HTML and works.

3. Individual shot pages, popular, and search sit behind an AWS WAF challenge. They return HTTP 202 with x-amzn-waf-action: challenge and a small JavaScript page that a real browser solves. I didn't try to get around it, and you shouldn't either. Everything in this post comes from profile and collection pages, which already show titles, likes, views, and dates for every shot. The scraper treats a 202 as "stop", not "retry".

One more detail: the page's own config included "isRobot": true for my requests. Dribbble knows exactly what you are, so be polite.

Step 1: Project setup and a polite fetch helper

mkdir dribbblescraper && cd dribbblescraper
go mod init dribbblescraper
go get golang.org/x/net/html

getHTML checks robots.txt, keeps at least two seconds between requests, backs off on 429/5xx, and returns ErrChallenged on a WAF challenge. The User-Agent is honest. Put your own contact URL in it.

package main

import (
    "bufio"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "regexp"
    "strings"
    "time"
)

const (
    base      = "https://dribbble.com"
    userAgent = "dribbblescraper/0.1 (+https://example.com/contact)"
)

var (
    client = &http.Client{Timeout: 25 * time.Second}

    // ErrChallenged means AWS WAF answered with a JavaScript challenge
    // (HTTP 202, empty-ish body, x-amzn-waf-action: challenge). We stop
    // instead of trying to solve it.
    ErrChallenged = errors.New("blocked by AWS WAF challenge")
    ErrDisallowed = errors.New("path disallowed by robots.txt")

    lastRequest time.Time
    minGap      = 2 * time.Second
)

// getHTML fetches a dribbble.com path after checking robots.txt, waits at
// least minGap between requests, and backs off on 429/5xx.
func getHTML(path string) (string, error) {
    if !robots.Allowed(path) {
        return "", fmt.Errorf("%w: %s", ErrDisallowed, path)
    }
    for attempt := 0; attempt < 4; attempt++ {
        if wait := minGap - time.Since(lastRequest); wait > 0 {
            time.Sleep(wait)
        }
        lastRequest = time.Now()

        req, _ := http.NewRequest("GET", base+path, nil)
        req.Header.Set("User-Agent", userAgent)
        req.Header.Set("Accept", "text/html")
        req.Header.Set("Accept-Language", "en-US,en;q=0.9")
        res, err := client.Do(req)
        if err != nil {
            time.Sleep(time.Duration(attempt+1) * 3 * time.Second)
            continue
        }
        body, err := io.ReadAll(io.LimitReader(res.Body, 8<<20))
        res.Body.Close()
        if err != nil {
            return "", err
        }
        switch {
        case res.Header.Get("x-amzn-waf-action") != "" || res.StatusCode == 202:
            return "", fmt.Errorf("%w: %s", ErrChallenged, path)
        case res.StatusCode == 429 || res.StatusCode >= 500:
            fmt.Fprintf(os.Stderr, "HTTP %d on %s, backing off\n", res.StatusCode, path)
            time.Sleep(time.Duration(attempt+1) * 15 * time.Second)
            continue
        case res.StatusCode == 404:
            return "", fmt.Errorf("not found: %s", path)
        case res.StatusCode != 200:
            return "", fmt.Errorf("GET %s: HTTP %d", path, res.StatusCode)
        }
        return string(body), nil
    }
    return "", errors.New("gave up after retries: " + path)
}

The robots matcher is about 50 lines. It reads the User-agent: * group and turns each rule into a regexp, so * matches anything and a trailing $ anchors the end:

// --- a tiny robots.txt matcher (User-agent: * group, * and $ wildcards) ---

type robotsRules struct{ allow, disallow []*regexp.Regexp }

var robots = &robotsRules{}

func loadRobots() error {
    req, _ := http.NewRequest("GET", base+"/robots.txt", nil)
    req.Header.Set("User-Agent", userAgent)
    res, err := client.Do(req)
    if err != nil {
        return err
    }
    defer res.Body.Close()
    if res.StatusCode != 200 {
        return fmt.Errorf("robots.txt: HTTP %d", res.StatusCode)
    }
    inStar := false
    sc := bufio.NewScanner(res.Body)
    for sc.Scan() {
        line := strings.TrimSpace(strings.SplitN(sc.Text(), "#", 2)[0])
        k, v, ok := strings.Cut(line, ":")
        if !ok {
            continue
        }
        k, v = strings.ToLower(strings.TrimSpace(k)), strings.TrimSpace(v)
        switch {
        case k == "user-agent":
            inStar = v == "*"
        case inStar && k == "disallow" && v != "":
            robots.disallow = append(robots.disallow, robotsPattern(v))
        case inStar && k == "allow" && v != "":
            robots.allow = append(robots.allow, robotsPattern(v))
        }
    }
    return sc.Err()
}

// robotsPattern turns "/*/followers$" into ^/[^?]*?/followers$ style regexps.
func robotsPattern(p string) *regexp.Regexp {
    anchored := strings.HasSuffix(p, "$")
    p = strings.TrimSuffix(p, "$")
    parts := strings.Split(p, "*")
    for i := range parts {
        parts[i] = regexp.QuoteMeta(parts[i])
    }
    re := "^" + strings.Join(parts, ".*")
    if anchored {
        re += "$"
    }
    return regexp.MustCompile(re)
}

func (r *robotsRules) Allowed(path string) bool {
    // Match on the path without the query string for $-anchored rules.
    p, _, _ := strings.Cut(path, "?")
    for _, a := range r.allow {
        if a.MatchString(p) {
            return true
        }
    }
    for _, d := range r.disallow {
        if d.MatchString(p) {
            return false
        }
    }
    return true
}

A test pins down the rules we care about. It runs against the live robots.txt:

package main

import (
    "errors"
    "testing"
)

func TestRobots(t *testing.T) {
    if err := loadRobots(); err != nil {
        t.Skip("no network:", err)
    }
    cases := map[string]bool{
        "/simplebits":                     true,
        "/simplebits?page=2":              true,
        "/simplebits/about":               true,
        "/simplebits/collections":         true,
        "/simplebits/followers":           false,
        "/simplebits/following":           false,
        "/simplebits/tags":                false,
        "/shots/123-Foo/comments":         false,
        "/shots/123-Foo/fans":             false,
        "/shots/week/2026-10-01":          false,
        "/shots/27755136-Measure-Twice-P": true,
    }
    for path, want := range cases {
        if got := robots.Allowed(path); got != want {
            t.Errorf("Allowed(%q) = %v, want %v", path, got, want)
        }
    }
}

func TestShotPageIsChallenged(t *testing.T) {
    if err := loadRobots(); err != nil {
        t.Skip("no network:", err)
    }
    _, err := getHTML("/shots/27755136-Measure-Twice-Print")
    t.Logf("shot page: %v", err)
    if err != nil && !errors.Is(err, ErrChallenged) {
        t.Errorf("unexpected error: %v", err)
    }
}

$ go test -v ./...
=== RUN   TestRobots
--- PASS: TestRobots (0.05s)
=== RUN   TestShotPageIsChallenged
    robots_test.go:37: shot page: blocked by AWS WAF challenge: /shots/27755136-Measure-Twice-Print
--- PASS: TestShotPageIsChallenged (0.07s)
PASS

Step 2: Small HTML helpers

golang.org/x/net/html gives you a parse tree but no selectors, so a few helpers go a long way. parseCount matters more than it looks: Dribbble shows views as 23.3k, and counts on the about page as 117,603.

package main

import (
    "strconv"
    "strings"

    "golang.org/x/net/html"
)

func parseDoc(s string) (*html.Node, error) { return html.Parse(strings.NewReader(s)) }

func attr(n *html.Node, key string) string {
    for _, a := range n.Attr {
        if a.Key == key {
            return a.Val
        }
    }
    return ""
}

func hasAttr(n *html.Node, key string) bool {
    for _, a := range n.Attr {
        if a.Key == key {
            return true
        }
    }
    return false
}

func hasClass(n *html.Node, class string) bool {
    if n.Type != html.ElementNode {
        return false
    }
    for _, c := range strings.Fields(attr(n, "class")) {
        if c == class {
            return true
        }
    }
    return false
}

// findAll walks the tree and returns every node matching pred.
func findAll(n *html.Node, pred func(*html.Node) bool) []*html.Node {
    var out []*html.Node
    var walk func(*html.Node)
    walk = func(n *html.Node) {
        if pred(n) {
            out = append(out, n)
        }
        for c := n.FirstChild; c != nil; c = c.NextSibling {
            walk(c)
        }
    }
    walk(n)
    return out
}

func first(n *html.Node, pred func(*html.Node) bool) *html.Node {
    if all := findAll(n, pred); len(all) > 0 {
        return all[0]
    }
    return nil
}

func byClass(c string) func(*html.Node) bool {
    return func(n *html.Node) bool { return hasClass(n, c) }
}

func byTag(tag string) func(*html.Node) bool {
    return func(n *html.Node) bool { return n.Type == html.ElementNode && n.Data == tag }
}

// text returns the whitespace-collapsed text content of a node (nil-safe).
func text(n *html.Node) string {
    if n == nil {
        return ""
    }
    var b strings.Builder
    var walk func(*html.Node)
    walk = func(n *html.Node) {
        if n.Type == html.TextNode {
            b.WriteString(n.Data + " ")
        }
        if n.Type == html.ElementNode && (n.Data == "script" || n.Data == "style" || n.Data == "svg") {
            return
        }
        for c := n.FirstChild; c != nil; c = c.NextSibling {
            walk(c)
        }
    }
    walk(n)
    return strings.Join(strings.Fields(b.String()), " ")
}

// parseCount turns "117,603", "23.3k", "1.2m" or "381" into an int.
func parseCount(s string) int {
    s = strings.ToLower(strings.TrimSpace(strings.ReplaceAll(s, ",", "")))
    mult := 1.0
    switch {
    case strings.HasSuffix(s, "k"):
        mult, s = 1_000, strings.TrimSuffix(s, "k")
    case strings.HasSuffix(s, "m"):
        mult, s = 1_000_000, strings.TrimSuffix(s, "m")
    }
    f, err := strconv.ParseFloat(s, 64)
    if err != nil {
        return 0
    }
    return int(f*mult + 0.5)
}

// leadingCount reads the number at the start of "223 Shots" or "117,603 followers".
func leadingCount(s string) int {
    f := strings.Fields(s)
    if len(f) == 0 {
        return 0
    }
    return parseCount(f[0])
}

Step 3: A designer profile

The /about page has everything: name, headline, bio, location, "Member since", Pro badge, skills, and follower counts. The trap is that there are (at least) two profile templates. On simplebits (Dan Cederholm, a Dribbble co-founder) the masthead shows "117,603 followers / 1,820 following / 46,450 likes". On Betraydan the masthead shows location and specializations instead, and there's no likes count at all. Both templates have a .profile-stats-section with count + meta spans, so the code reads followers from there and only takes likes when they're shown.

package main

import (
    "net/url"
    "regexp"
    "strings"

    "golang.org/x/net/html"
)

type Profile struct {
    Username    string   `json:"username"`
    Name        string   `json:"name"`
    Headline    string   `json:"headline"`
    Bio         string   `json:"bio"`
    Location    string   `json:"location"`
    MemberSince string   `json:"member_since"`
    Pro         bool     `json:"pro"`
    Followers   int      `json:"followers"`
    Following   int      `json:"following"`
    Likes       int      `json:"likes"` // 0 when the template doesn't show it
    Skills      []string `json:"skills"`
}

var usernameRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$`)

// normalizeUser accepts "simplebits", "@simplebits" or "https://dribbble.com/simplebits/about".
func normalizeUser(s string) string {
    s = strings.TrimPrefix(strings.TrimSpace(s), "@")
    if u, err := url.Parse(s); err == nil && u.Host != "" {
        s = strings.Split(strings.Trim(u.Path, "/"), "/")[0]
    }
    if !usernameRe.MatchString(s) {
        return ""
    }
    return s
}

// fetchProfile reads /<user>/about, which has everything the main profile page
// has in its masthead plus bio, location, "Member since" and skills.
func fetchProfile(user string) (Profile, error) {
    p := Profile{Username: user}
    page, err := getHTML("/" + user + "/about")
    if err != nil {
        return p, err
    }
    doc, err := parseDoc(page)
    if err != nil {
        return p, err
    }
    p.Name = text(first(doc, byClass("masthead-profile-name")))
    p.Headline = text(first(doc, byClass("masthead-intro")))

    if p.Headline == "" { // the second profile template shows specializations instead
        p.Headline = text(first(doc, byClass("masthead-profile-specializations")))
    }

    // Dribbble has (at least) two profile templates. Both have a
    // ".profile-stats-section" with <span class="count"> + <span class="meta">
    // pairs; only one also has "N likes" in ".masthead-stats".
    if sec := first(doc, byClass("profile-stats-section")); sec != nil {
        for _, a := range findAll(sec, byTag("a")) {
            n := leadingCount(text(first(a, byClass("count"))))
            switch text(first(a, byClass("meta"))) {
            case "followers":
                p.Followers = n
            case "following":
                p.Following = n
            }
        }
    }
    if stats := first(doc, byClass("masthead-stats")); stats != nil {
        for _, st := range findAll(stats, byClass("stat")) {
            if t := text(st); strings.HasSuffix(t, "likes") {
                p.Likes = leadingCount(t)
            }
        }
    }

    var bio []string
    for _, b := range findAll(doc, byClass("bio-text")) {
        bio = append(bio, text(b))
    }
    p.Bio = strings.Join(bio, "\n")

    // The info section is rendered twice (small screens and large), so take the first.
    p.Location = text(first(doc, byClass("location")))
    p.MemberSince = strings.TrimPrefix(text(first(doc, byClass("created"))), "Member since ")
    p.Pro = first(doc, func(n *html.Node) bool { return hasClass(n, "info-item") && hasClass(n, "pro") }) != nil

    if sec := first(doc, byClass("profile-skills-section")); sec != nil {
        for _, a := range findAll(sec, byTag("a")) {
            if s := attr(a, "data-skill"); s != "" {
                p.Skills = append(p.Skills, s)
            }
        }
    }
    return p, nil
}

$ ./dribbblescraper profile simplebits
{
  "username": "simplebits",
  "name": "Dan Cederholm",
  "headline": "Making type \u0026 goods at SimpleBits®. Dribbble Co-Founder.",
  "bio": "Making type and goods at SimpleBits® since 2002.\nJoin my Secret Club™ for all my fonts, early releases, private community, discounts, and other good stuff.\nsimplebits dot com",
  "location": "Salem, MA",
  "member_since": "Jul 2009",
  "pro": true,
  "followers": 117603,
  "following": 1820,
  "likes": 46450,
  "skills": ["branding", "fonts", "icon", "illustration", "logo", "logo design", "type design", "ui"]
}

$ ./dribbblescraper profile https://dribbble.com/Betraydan/about
{
  "username": "Betraydan",
  "name": "Daniel Klopper",
  "headline": "Product Design, UI / Visual Design, Mobile Design",
  "location": "Cape Town, South Africa",
  "member_since": "Jul 2011",
  "pro": false,
  "followers": 19180,
  "following": 303,
  "likes": 0,
  ...
}

My first version only read the masthead and returned followers: 0 for Betraydan. Test against more than one profile before you trust a parser.

Two things you won't get logged out: social links and email. They sit behind "Get in touch", which opens a signup dialog. That's fine. That's the designer's choice.

Step 4: Shots, page by page

extractJSONVar finds var newestShots = and hands the rest of the page to json.Decoder. The decoder stops at the end of the JSON value, so there's no need for a fragile regexp to find the closing ].

Thumbnails are lazy-loaded: the <img src> is a 1×1 GIF, and the real URL is in data-src (and in a <noscript> copy). If your scraper saves src, you get 24 transparent pixels.

package main

import (
    "encoding/json"
    "fmt"
    "strings"
    "time"
)

// Shot is what the profile page ships in `var newestShots = [...]`,
// plus the thumbnail we read from the HTML grid.
type Shot struct {
    ID          int64  `json:"id"`
    Title       string `json:"title"`
    Path        string `json:"path"`
    PublishedAt string `json:"published_at"` // "September 23, 2026"
    Rebounds    int    `json:"rebounds_count"`
    Attachments int    `json:"attachments_count"`
    ViewsRaw    string `json:"view_count"`  // "23.3k" (rounded!)
    LikesRaw    string `json:"likes_count"` // "108"
    Views       int    `json:"-"`
    Likes       int    `json:"-"`
    Thumbnail   string `json:"-"`
}

func (s Shot) URL() string { return base + s.Path }

func (s Shot) Published() time.Time {
    t, _ := time.Parse("January 2, 2006", s.PublishedAt)
    return t
}

// extractJSONVar finds `var <name> = ` in a page and decodes the JSON value
// that follows. json.Decoder stops at the end of the value, so we don't need
// to guess where the array ends.
func extractJSONVar(page, name string, out any) error {
    marker := "var " + name + " = "
    i := strings.Index(page, marker)
    if i < 0 {
        return fmt.Errorf("%q not found (markup changed?)", marker)
    }
    return json.NewDecoder(strings.NewReader(page[i+len(marker):])).Decode(out)
}

// fetchShotsPage reads one page of /<user>?page=N. Logged out, each page has 24 shots.
func fetchShotsPage(user string, page int) (shots []Shot, hasNext bool, err error) {
    body, err := getHTML(fmt.Sprintf("/%s?page=%d", user, page))
    if err != nil {
        return nil, false, err
    }
    if err := extractJSONVar(body, "newestShots", &shots); err != nil {
        return nil, false, err
    }

    // Thumbnails are lazy-loaded: src is a 1x1 GIF, the real URL is in data-src.
    thumbs := map[string]string{}
    if doc, err := parseDoc(body); err == nil {
        for _, li := range findAll(doc, byClass("shot-thumbnail")) {
            id := attr(li, "data-thumbnail-id")
            if img := first(li, byTag("img")); img != nil && id != "" {
                src := attr(img, "data-src")
                if src == "" {
                    src = attr(img, "src")
                }
                thumbs[id] = strings.Split(src, "?")[0]
            }
        }
    }
    for i := range shots {
        shots[i].Views = parseCount(shots[i].ViewsRaw)
        shots[i].Likes = parseCount(shots[i].LikesRaw)
        shots[i].Thumbnail = thumbs[fmt.Sprint(shots[i].ID)]
    }
    hasNext = strings.Contains(body, `class="next_page"`) && strings.Contains(body, `rel="next"`)
    return shots, hasNext, nil
}

// fetchShots pages through a profile until maxPages, an empty page, or no "Next" link.
func fetchShots(user string, maxPages int) ([]Shot, error) {
    var all []Shot
    seen := map[int64]bool{}
    for page := 1; page <= maxPages; page++ {
        batch, hasNext, err := fetchShotsPage(user, page)
        if err != nil {
            return all, err
        }
        added := 0
        for _, s := range batch {
            if !seen[s.ID] { // pages can overlap if a new shot is posted mid-crawl
                seen[s.ID] = true
                all = append(all, s)
                added++
            }
        }
        if added == 0 || !hasNext {
            break
        }
    }
    return all, nil
}

$ ./dribbblescraper shots -pages 3 -out simplebits.csv simplebits
72 shots from 3 page(s) of simplebits
  2026-09-23    108 ♥   23300 views  Measure Twice Print
  2026-09-23    108 ♥   32200 views  Hubano 2.0
  2026-07-21     46 ♥    7400 views  Serif Pin
  2026-07-21     47 ♥    5600 views  Kyroh Typeface
  2026-06-10     48 ♥   16800 views  Beeswacks Typeface

$ head -2 simplebits.csv
id,published,title,likes,views,views_raw,rebounds,url,thumbnail
27755136,2026-09-23,Measure Twice Print,108,23300,23.3k,0,https://dribbble.com/shots/27755136-Measure-Twice-Print,https://cdn.dribbble.com/userupload/49110833/file/d68cabe4e8579fe05889da8d308f0ea0.jpg

Note the views_raw column. 23.3k becomes 23,300, but the real number is somewhere between 23,250 and 23,349. Keep the raw string, and don't report view growth smaller than the rounding step.

Step 5: Collections

Collections are curated sets of other people's shots. A designer's /collections page lists them 6 at a time, and ?page=2 gives the next 6. Each one is an li.shots-group-item with a name, a shot count, and a designer count:

package main

import (
    "fmt"
    "net/url"
    "regexp"
    "strings"

    "golang.org/x/net/html"
)

type Collection struct {
    ID        string `json:"id"`
    Name      string `json:"name"`
    URL       string `json:"url"`
    Shots     int    `json:"shots"`
    Designers int    `json:"designers"`
}

// fetchCollections reads /<user>/collections?page=N (6 per page, logged out).
func fetchCollections(user string, maxPages int) ([]Collection, error) {
    var all []Collection
    for page := 1; page <= maxPages; page++ {
        body, err := getHTML(fmt.Sprintf("/%s/collections?page=%d", user, page))
        if err != nil {
            return all, err
        }
        doc, err := parseDoc(body)
        if err != nil {
            return all, err
        }
        items := findAll(doc, byClass("shots-group-item"))
        for _, li := range items {
            c := Collection{ID: attr(li, "data-id"), Name: text(first(li, byClass("collection-name")))}
            if a := first(li, byClass("shots-group")); a != nil {
                c.URL = base + attr(a, "href")
            }
            c.Shots = leadingCount(text(first(li, byClass("shots-count"))))
            c.Designers = leadingCount(text(first(li, byClass("designers-count"))))
            all = append(all, c)
        }
        if len(items) == 0 {
            break
        }
    }
    return all, nil
}

A single collection page uses newer markup (shot2-item), 24 shots per page. Here the title and path are attributes on a <drb-shot-metadata> custom element, which is nicer than digging through text. The author is the profile link inside the details block.

type CollectionShot struct {
    ID         string
    Title      string
    URL        string
    Author     string // profile path, e.g. "Betraydan"
    AuthorName string
    Likes      int
    Views      int
    Thumbnail  string
}

type CollectionPage struct {
    Name      string
    Curator   string
    Shots     int
    Designers int
    Items     []CollectionShot
}

var collectionPath = regexp.MustCompile(`^/[A-Za-z0-9_-]+/collections/\d+(-[^/]*)?$`)

// fetchCollection reads one collection (24 shots per page) from a full URL.
func fetchCollection(raw string, maxPages int) (CollectionPage, error) {
    var out CollectionPage
    u, err := url.Parse(raw)
    if err != nil || !collectionPath.MatchString(u.Path) {
        return out, fmt.Errorf("want https://dribbble.com/<user>/collections/<id>-<slug>, got %q", raw)
    }
    seen := map[string]bool{}
    for page := 1; page <= maxPages; page++ {
        body, err := getHTML(fmt.Sprintf("%s?page=%d", u.Path, page))
        if err != nil {
            return out, err
        }
        doc, err := parseDoc(body)
        if err != nil {
            return out, err
        }
        if page == 1 {
            out.Name = text(first(doc, byClass("collection-page__title")))
            out.Curator = text(first(doc, byClass("curator__name")))
            for _, v := range findAll(doc, byClass("collection-page__meta-details-value")) {
                t := text(v)
                switch {
                case strings.HasSuffix(t, "Shots"):
                    out.Shots = leadingCount(t)
                case strings.HasSuffix(t, "Designers"):
                    out.Designers = leadingCount(t)
                }
            }
        }
        added := 0
        for _, it := range findAll(doc, byClass("shot2-item")) {
            id := attr(it, "data-id")
            if id == "" || seen[id] {
                continue
            }
            seen[id] = true
            added++
            s := CollectionShot{ID: id}
            if m := first(it, byTag("drb-shot-metadata")); m != nil {
                s.Title, s.URL = attr(m, "title"), base+attr(m, "path")
            }
            if img := first(it, byTag("img")); img != nil {
                s.Thumbnail = strings.Split(attr(img, "src"), "?")[0]
            }
            if d := first(it, byClass("shot2-item-details__user-container")); d != nil {
                if a := first(d, byTag("a")); a != nil {
                    s.Author = strings.Trim(attr(a, "href"), "/")
                }
                s.AuthorName = text(first(d, byClass("shot2-item-details__user-name")))
            }
            // Stats: the like count sits in [data-shot-like-count]; views are the
            // text of the last ".shot2-item-stats__item".
            stats := findAll(it, byClass("shot2-item-stats__item"))
            if lc := first(it, func(n *html.Node) bool { return hasAttr(n, "data-shot-like-count") }); lc != nil {
                s.Likes = parseCount(text(lc))
            }
            if len(stats) > 0 {
                s.Views = parseCount(text(stats[len(stats)-1]))
            }
            out.Items = append(out.Items, s)
        }
        if added == 0 {
            break
        }
    }
    return out, nil
}

$ ./dribbblescraper collections simplebits
  Data Viz                223 shots ·  160 designers  https://dribbble.com/simplebits/collections/4582-Data-Viz
  Glyphs                  176 shots ·  111 designers  https://dribbble.com/simplebits/collections/4581-Glyphs
  iOS Icons               284 shots ·  156 designers  https://dribbble.com/simplebits/collections/4144-iOS-Icons
  Forms                    40 shots ·   33 designers  https://dribbble.com/simplebits/collections/3834-Forms
  Pixels                   28 shots ·   17 designers  https://dribbble.com/simplebits/collections/3568-Pixels
  Sketches                 87 shots ·   60 designers  https://dribbble.com/simplebits/collections/3542-Sketches
  Cool Jobs                35 shots ·   32 designers  https://dribbble.com/simplebits/collections/3297-Cool-Jobs
  Great Illustration      496 shots ·  256 designers  https://dribbble.com/simplebits/collections/3295-Great-Illustration
  Great Type              202 shots ·  117 designers  https://dribbble.com/simplebits/collections/3294-Great-Type
  Great Marks             221 shots ·  143 designers  https://dribbble.com/simplebits/collections/2754-Great-Marks

$ ./dribbblescraper collection -pages 2 https://dribbble.com/simplebits/collections/4582-Data-Viz
Data Viz, curated by Dan Cederholm: 223 shots by 160 designers (48 fetched)
    381 ♥   17700 views  Analytics                    by Betraydan
    559 ♥   30100 views  Managing Dashboard           by CindyWang
    548 ♥   26900 views  Paper infographic*           by kellianderson
    136 ♥   33600 views  Maxim Magazine Space Oddities Infographic by DKNG
     15 ♥    1700 views  1% of the 1% Data Viz        by AmyCesal

Curated collections are a good discovery layer. "Data Viz" alone links to 160 designers, which is a much better list than anything you'd get from search (which is challenged anyway).

Step 6: A portfolio tracker

Point-in-time numbers aren't very interesting. Changes are. track appends one row per profile to profiles.csv and one row per shot to shots.csv, all with the same snapshot_at. report compares each designer's last two snapshots: follower change, the shots that gained the most likes, and shots that are new since the previous run.

package main

import (
    "encoding/csv"
    "fmt"
    "os"
    "path/filepath"
    "sort"
    "strconv"
    "strings"
    "time"
)

// appendCSV appends rows to path, writing the header first if the file is new.
func appendCSV(path string, header []string, rows [][]string) error {
    _, statErr := os.Stat(path)
    f, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
    if err != nil {
        return err
    }
    defer f.Close()
    w := csv.NewWriter(f)
    if os.IsNotExist(statErr) {
        w.Write(header)
    }
    w.WriteAll(rows)
    return w.Error()
}

// track takes one snapshot of each profile and its newest shots.
func track(dir string, users []string, pages int) error {
    if err := os.MkdirAll(dir, 0o755); err != nil {
        return err
    }
    now := time.Now().UTC().Format(time.RFC3339)
    for _, user := range users {
        p, err := fetchProfile(user)
        if err != nil {
            fmt.Fprintf(os.Stderr, "%s: %v\n", user, err)
            continue
        }
        shots, err := fetchShots(user, pages)
        if err != nil {
            fmt.Fprintf(os.Stderr, "%s shots: %v\n", user, err)
        }
        err = appendCSV(filepath.Join(dir, "profiles.csv"),
            []string{"snapshot_at", "username", "followers", "following", "likes", "shots_seen"},
            [][]string{{now, user, strconv.Itoa(p.Followers), strconv.Itoa(p.Following), strconv.Itoa(p.Likes), strconv.Itoa(len(shots))}})
        if err != nil {
            return err
        }
        var rows [][]string
        for _, s := range shots {
            rows = append(rows, []string{now, user, strconv.FormatInt(s.ID, 10), s.Title,
                s.Published().Format("2006-01-02"), strconv.Itoa(s.Likes), strconv.Itoa(s.Views), s.URL()})
        }
        if err := appendCSV(filepath.Join(dir, "shots.csv"),
            []string{"snapshot_at", "username", "shot_id", "title", "published", "likes", "views", "url"}, rows); err != nil {
            return err
        }
        fmt.Printf("%-16s followers %d · likes %d · %d shots snapshotted\n", user, p.Followers, p.Likes, len(shots))
    }
    return nil
}

report reads both CSVs back, groups rows by user and snapshot time, and diffs the last two snapshots:

func readCSV(path string) ([][]string, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, err
    }
    defer f.Close()
    rows, err := csv.NewReader(f).ReadAll()
    if len(rows) > 0 {
        rows = rows[1:] // drop header
    }
    return rows, err
}

// report compares the two most recent snapshots per user.
func report(dir string) error {
    profiles, err := readCSV(filepath.Join(dir, "profiles.csv"))
    if err != nil {
        return err
    }
    shots, err := readCSV(filepath.Join(dir, "shots.csv"))
    if err != nil {
        return err
    }
    snaps := map[string][]string{} // user -> snapshot times, oldest first
    follow := map[string]int{}     // user|time -> followers
    for _, r := range profiles {
        snaps[r[1]] = append(snaps[r[1]], r[0])
        follow[r[1]+"|"+r[0]], _ = strconv.Atoi(r[2])
    }
    likes := map[string]int{} // user|time|shot -> likes
    titles := map[string]string{}
    for _, r := range shots {
        likes[r[1]+"|"+r[0]+"|"+r[2]], _ = strconv.Atoi(r[5])
        titles[r[2]] = r[3]
    }
    users := make([]string, 0, len(snaps))
    for u := range snaps {
        users = append(users, u)
    }
    sort.Strings(users)
    for _, u := range users {
        ts := snaps[u]
        if len(ts) < 2 {
            fmt.Printf("%s: only one snapshot so far\n", u)
            continue
        }
        prev, cur := ts[len(ts)-2], ts[len(ts)-1]
        fmt.Printf("%s  %s → %s\n  followers %+d\n", u, prev[:16], cur[:16], follow[u+"|"+cur]-follow[u+"|"+prev])
        type gain struct {
            id string
            d  int
        }
        var gains []gain
        var fresh []string
        for k, v := range likes {
            parts := strings.SplitN(k, "|", 3)
            user, t, id := parts[0], parts[1], parts[2]
            if user != u || t != cur {
                continue
            }
            if old, ok := likes[u+"|"+prev+"|"+id]; ok {
                if v != old {
                    gains = append(gains, gain{id, v - old})
                }
            } else {
                fresh = append(fresh, titles[id]) // not in the previous snapshot
            }
        }
        sort.Slice(gains, func(i, j int) bool { return gains[i].d > gains[j].d })
        if len(gains) == 0 {
            fmt.Println("  no like changes on tracked shots")
        }
        for _, g := range gains[:min(3, len(gains))] {
            fmt.Printf("  %+5d likes  %s\n", g.d, titles[g.id])
        }
        for _, t := range fresh {
            fmt.Printf("  new shot: %s\n", t)
        }
    }
    return nil
}

$ ./dribbblescraper track -out portfolio -pages 1 simplebits Betraydan
simplebits       followers 117603 · likes 46450 · 24 shots snapshotted
Betraydan        followers 19180 · likes 0 · 24 shots snapshotted

$ ./dribbblescraper report -out portfolio      # one minute later
Betraydan  2026-10-30T04:08 → 2026-10-30T04:09
  followers +0
  no like changes on tracked shots
simplebits  2026-10-30T04:08 → 2026-10-30T04:09
  followers +0
  no like changes on tracked shots

Nothing changed in a minute, which is the honest result. Run it once a day from cron and the report becomes useful:

# 09:00 every day, a handful of profiles, 1 page of shots each
0 9 * * * cd /home/me/dribbblescraper && ./dribbblescraper track -out portfolio -pages 1 simplebits Betraydan >> track.log 2>&1

Two profiles with one page each is 4 requests, about 8 seconds with the 2-second gap. Keep it that small.

Step 7: The CLI

package main

import (
    "encoding/csv"
    "encoding/json"
    "flag"
    "fmt"
    "os"
    "strconv"
    "strings"
)

func main() {
    if len(os.Args) < 2 {
        fmt.Println("usage: dribbblescraper profile|shots|collections|collection|track|report [flags] args")
        os.Exit(1)
    }
    cmd := os.Args[1]
    fs := flag.NewFlagSet(cmd, flag.ExitOnError)
    pages := fs.Int("pages", 2, "max pages to fetch")
    out := fs.String("out", "", "CSV output file (shots) or directory (track/report)")
    fs.Parse(os.Args[2:])

    if err := loadRobots(); err != nil {
        fmt.Fprintln(os.Stderr, "can't read robots.txt, refusing to crawl:", err)
        os.Exit(1)
    }
    if err := run(cmd, fs.Args(), *pages, *out); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}

func run(cmd string, args []string, pages int, out string) error {
    if len(args) == 0 && cmd != "report" {
        return fmt.Errorf("%s: missing argument", cmd)
    }
    if (cmd == "profile" || cmd == "shots" || cmd == "collections") && normalizeUser(args[0]) == "" {
        return fmt.Errorf("%q is not a Dribbble username or profile URL", args[0])
    }
    switch cmd {
    case "profile":
        p, err := fetchProfile(normalizeUser(args[0]))
        if err != nil {
            return err
        }
        b, _ := json.MarshalIndent(p, "", "  ")
        fmt.Println(string(b))

    case "shots":
        user := normalizeUser(args[0])
        shots, err := fetchShots(user, pages)
        if err != nil && len(shots) == 0 {
            return err
        }
        fmt.Printf("%d shots from %d page(s) of %s\n", len(shots), pages, user)
        for _, s := range shots[:min(5, len(shots))] {
            fmt.Printf("  %s  %5d ♥ %7d views  %s\n", s.Published().Format("2006-01-02"), s.Likes, s.Views, s.Title)
        }
        if out != "" {
            return writeShotsCSV(out, shots)
        }

    case "collections":
        cs, err := fetchCollections(normalizeUser(args[0]), pages)
        if err != nil && len(cs) == 0 {
            return err
        }
        for _, c := range cs {
            fmt.Printf("  %-22s %4d shots · %4d designers  %s\n", c.Name, c.Shots, c.Designers, c.URL)
        }

    case "collection":
        c, err := fetchCollection(args[0], pages)
        if err != nil && len(c.Items) == 0 {
            return err
        }
        fmt.Printf("%s, curated by %s: %d shots by %d designers (%d fetched)\n", c.Name, c.Curator, c.Shots, c.Designers, len(c.Items))
        for _, s := range c.Items[:min(5, len(c.Items))] {
            fmt.Printf("  %5d ♥ %7d views  %-28s by %s\n", s.Likes, s.Views, s.Title, s.Author)
        }

    case "track":
        if out == "" {
            out = "portfolio"
        }
        var users []string
        for _, a := range args {
            if u := normalizeUser(a); u != "" {
                users = append(users, u)
            }
        }
        return track(out, users, pages)

    case "report":
        if out == "" {
            out = "portfolio"
        }
        return report(out)

    default:
        return fmt.Errorf("unknown command %q", cmd)
    }
    return nil
}

func writeShotsCSV(path string, shots []Shot) error {
    f, err := os.Create(path)
    if err != nil {
        return err
    }
    defer f.Close()
    w := csv.NewWriter(f)
    w.Write([]string{"id", "published", "title", "likes", "views", "views_raw", "rebounds", "url", "thumbnail"})
    for _, s := range shots {
        w.Write([]string{strconv.FormatInt(s.ID, 10), s.Published().Format("2006-01-02"), strings.TrimSpace(s.Title),
            strconv.Itoa(s.Likes), strconv.Itoa(s.Views), s.ViewsRaw, strconv.Itoa(s.Rebounds), s.URL(), s.Thumbnail})
    }
    w.Flush()
    return w.Error()
}

go build -o dribbblescraper .
./dribbblescraper profile simplebits
./dribbblescraper shots -pages 3 -out simplebits.csv simplebits
./dribbblescraper collection https://dribbble.com/simplebits/collections/4582-Data-Viz
./dribbblescraper track -out portfolio simplebits Betraydan && ./dribbblescraper report -out portfolio

Pitfalls, in one place

The Dribbble API is for your own account. v2 only returns the authenticated user's shots. No public endpoints for other designers.
Terms of Service. Dribbble's terms prohibit scraping. robots.txt allowing a page doesn't change that. Keep it small, slow, and non-commercial.
Shot pages, popular, and search are behind AWS WAF. You get 202 plus x-amzn-waf-action: challenge. Stop there. Profile and collection grids already have titles, likes, and views.
robots.txt forbids followers, following, tags, comments, fans, and rebounds. Check every path, not just the first one.
The old RSS feeds are gone. /<user>.rss and /shots.rss return 404, and /shots/popular.rss is challenged.
Two profile templates. Followers live in different places, and likes are missing on one. Test with more than one designer.
Lazy-loaded images. src is a 1×1 GIF; use data-src.
Counts are rounded. 23.3k views. Keep the raw string.
Dates are display strings. published_at is "September 23, 2026", with no time or time zone. Parse with "January 2, 2006" and treat it as a day.
Pages can shift. If a designer posts while you crawl, page 2 can repeat a shot from page 1. De-duplicate by shot ID.
The markup will change. The shots JSON and the custom elements (drb-shot-metadata, data-shot-like-count) are steadier than layout classes, but none of this is a contract.

When you'd rather not maintain the parsers

If you need more than a few designers, or you just don't want to fix two profile templates every time Dribbble ships a redesign, I also run ScrapeAtlas*, which has four Dribbble endpoints that return JSON for the same public pages:

Endpoint	Params	Returns
`GET /v1/dribbble/profile`	`username` (or a profile URL)	`profile`: name, headline, bio, location, `followersCount`, `followingCount`, `likesCount`, `skills`, `languages`, `website`, `socialLinks`, `isPro`, `isTeam`
`GET /v1/dribbble/shots`	`username`, `page` (1-100)	`shots[]` with `title`, `thumbnailUrl`, `videoUrl`, `author`, `likesCount`, `viewsCount`, plus `pagination.nextPage`
`GET /v1/dribbble/collections`	`username`, `page`	`collections[]` with `name`, `shotsCount`, `designersCount`, `thumbnailUrls`
`GET /v1/dribbble/collection`	`url` (a `/<user>/collections/<id>-<slug>` URL), `page`	`collection` (name, description, owner, counts) and one page of `shots[]`

Auth is an x-api-key header. Each successful request costs one credit, and failed requests cost nothing:

req, _ := http.NewRequest("GET", "https://api.scrapeatlas.com/v1/dribbble/shots?username=simplebits&page=2", nil)
req.Header.Set("x-api-key", os.Getenv("SCRAPEATLAS_API_KEY"))
res, err := http.DefaultClient.Do(req)
// Response: {"success":true,"status":"success"|"partial", "shots":[...],
//            "pagination":{"page":2,"nextPage":3},
//            "coverage":{"complete":false,"warnings":["metrics_unavailable"]}}

Read status and coverage.warnings. When a page doesn't show likes or views, likesCount/viewsCount come back as null with a metrics_unavailable warning instead of a made-up 0. That's the same honesty you want from your own scraper. It reads the same public pages as the DIY version, so it doesn't give you follower lists, comments, or anything robots.txt rules out.

For a handful of designers you track daily, though, the DIY scraper above is enough.

Wrapping up

The official Dribbble API is your-account-only. Great for showing your own shots on your site, no help for research.
Profiles and collections are server-rendered. Shots come as embedded JSON (newestShots), and ?page=N still works logged out.
Respect the walls. robots.txt rules out followers, tags, comments, and fans; AWS WAF guards shot pages and search. Stop, don't bypass.
Expect two templates, lazy images, and rounded counts.
Track changes, not snapshots. A tiny daily cron job turns a scraper into a portfolio tracker.

What would you track on Dribbble: your own shots, a list of designers you'd like to hire, or which topics get the most likes? Tell me in the comments.

*I build it.

How to Scrape Whatnot with Node.js: Seller Stats, Live Shows, Search, and a Viewer Tracker (2026)

Viktor — Wed, 28 Oct 2026 04:21:59 +0000

Whatnot is where a lot of trading cards, comics, Funko Pops, and sneakers get sold now, live, one auction at a time. If you sell there, or you're studying the market, you want numbers: which sellers move the most items, who is live right now, how many people are watching, and what a card goes for when it's listed.

The first thing most people search is "Whatnot API". There is one, but it probably isn't the one you want. So this post covers both: what the official Whatnot API gives you, and how to build a small Whatnot scraper in Node.js for the public data it doesn't cover.

The scraper is one file, no npm dependencies, Node 18+. It will:

Discover seller handles from the sitemap Whatnot publishes in robots.txt
Read a seller's profile: followers, items sold, rating, reviews, live status
List a seller's live and scheduled shows, and their shop listings
Run a search and read a category feed
Read a single listing with price, condition, and seller
Snapshot live viewers in a category every few minutes into a CSV

Everything below was run on October 28, 2026, and the output is real.

Before you start: Whatnot's Terms of Service prohibit scraping, crawling, and other automated access to the app. robots.txt allows the public pages used here, but robots rules and terms are different things. Treat this as a learning project or small-scale research, keep request rates very low, don't resell the data, and don't build profiles of individual buyers or sellers. If you're a seller who wants your own data, use the official API first.

Is there a Whatnot API?

Yes. Whatnot has an official Seller API (REST, https://api.whatnot.com/seller-api/rest/v2, docs at developers.whatnot.com). You create a personal access token in the developer dashboard and send it as Authorization: Bearer wn_pat_tk_.... It covers products, inventory, orders, shipments, your shows, and show analytics.

The catch is in the name. The Whatnot API is for your own store. It answers "what did I sell and what do I need to ship", not "how many followers does that seller have" or "who is live in Pokémon cards right now". There is no official public endpoint for other sellers, search, or category feeds.

So, a quick decision table:

You want	Use
Your own orders, inventory, show analytics	Official Seller API
Other sellers' public stats, live shows, search results, listing prices	The public website, which is what this Whatnot scraper reads

How whatnot.com looks from a scraper's point of view

Three findings shaped the code.

1. robots.txt is friendly and points to sitemaps. It allows everything except account pages, the player, and a few others (/account/, /dashboard, /messages, /order/, /player, /video/, ...). It also lists seller-sitemap.txt, users-sitemap-index.xml, and listings-sitemap-index.xml. That's your discovery layer.

2. The data is already in the HTML. whatnot.com is a Next.js app with Apollo GraphQL. When the server renders a page, it embeds the GraphQL results it used in two places:

(window[Symbol.for("ApolloSSRDataTransport")] ??= []).push({...}) script tags. These hold the query results for feeds: search, category, a seller's shows, a seller's shop.
self.__next_f.push([1,"..."]) script tags. That's the React Server Components payload, and it carries the seller profile and the listing details.

So you don't need a headless browser, and you don't need to call the GraphQL endpoint yourself. The browser's GraphQL calls go to /services/graphql, which is protected by Kasada bot detection. Leave it alone. The HTML is enough.

3. Cloudflare sits in front, and it cares about honesty. This surprised me. From Node's built-in fetch, a request that pretends to be Chrome gets Cloudflare's "Just a moment..." challenge every time. A request with an honest, descriptive User-Agent gets a normal 200:

round  user-agent                status  bytes   challenge?
0      Chrome 130 string         403     5768    true
0      whatnot-research-script   200     830788  false
0      (none, Node default)      403     5533    true
0      curl/8.5.0                403     5533    true
1      Chrome 130 string         403     5768    true
1      whatnot-research-script   200     830827  false

My guess is that a Chrome User-Agent with a non-Chrome TLS fingerprint looks like a disguised bot, while a script that says it's a script looks like what it is. That might change, but "say who you are" is good practice anyway.

Step 1: A polite fetch helper

One request every 3 to 4 seconds, an honest User-Agent, 404 treated as "gone", and a hard back-off when Cloudflare shows up.

// whatnot.mjs - a small Whatnot scraper in Node.js (18+), no dependencies.
// Reads only public, robots-allowed pages: sitemaps, /user/, /tag/, /search, /listing/.
import fs from "node:fs";

const BASE = "https://www.whatnot.com";
// Say who you are. A fake Chrome UA from Node gets Cloudflare's challenge (see "Pitfalls").
const UA = process.env.WHATNOT_UA ?? "whatnot-research-script/1.0 (+mailto:you@example.com)";
const MIN_DELAY_MS = 3000; // one request every ~3 s is plenty for this kind of job

// ---------- Step 1: a polite fetch helper ----------

let lastRequest = 0;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export class BlockedError extends Error {}

export async function get(url, { tries = 3 } = {}) {
  for (let attempt = 1; attempt <= tries; attempt++) {
    const wait = lastRequest + MIN_DELAY_MS + Math.random() * 1000 - Date.now();
    if (wait > 0) await sleep(wait);
    lastRequest = Date.now();

    const res = await fetch(url, {
      headers: { "user-agent": UA, "accept-language": "en-US,en;q=0.9", accept: "text/html,*/*" },
      redirect: "follow",
    });
    const body = await res.text();

    if (res.status === 404) return null;
    // Cloudflare's interstitial: 403 + "Just a moment..." (or a challenge platform script)
    if (res.status === 403 && /Just a moment|challenge-platform/.test(body)) {
      if (attempt === tries) throw new BlockedError(`Cloudflare challenge on ${url}`);
      await sleep(30_000 * attempt); // back off hard, don't hammer
      continue;
    }
    if (res.status === 429 || res.status >= 500) {
      await sleep(5_000 * attempt);
      continue;
    }
    if (!res.ok) throw new Error(`${res.status} on ${url}`);
    return body;
  }
  throw new Error(`gave up on ${url}`);
}

BlockedError matters. If you see a challenge after a long back-off, stop. Don't rotate User-Agents or proxies to get around it.

Step 2: Read the data the page already ships

Two small parsers. The Apollo blocks are JavaScript object literals, not strict JSON: they contain undefined, so we swap it for null before JSON.parse. The RSC payload is a series of JSON strings that we concatenate. Then we pull out typed objects by looking for {"__typename":"..." and slicing a balanced {...}.

// ---------- Step 2: get the data the page already ships ----------

// (a) Apollo SSR transport: the GraphQL results the server rendered with.
// They are JS object literals (they contain `undefined`), not strict JSON.
const APOLLO = '(window[Symbol.for("ApolloSSRDataTransport")] ??= []).push(';

export function apolloResults(html) {
  const out = [];
  let i = 0;
  while ((i = html.indexOf(APOLLO, i)) !== -1) {
    const start = i + APOLLO.length;
    const end = html.indexOf(")</script>", start);
    const chunk = JSON.parse(html.slice(start, end).replace(/:undefined\b/g, ":null"));
    for (const v of Object.values(chunk.rehydrate ?? {})) if (v?.data) out.push(v.data);
    for (const ev of chunk.events ?? []) if (ev.type === "next" && ev.value?.data) out.push(ev.value.data);
    i = end;
  }
  return out;
}

// (b) React Server Components payload: self.__next_f.push([1,"..."]) strings.
export function rscText(html) {
  const re = /self\.__next_f\.push\(\[1,("(?:[^"\\]|\\.)*")\]\)/g;
  let m, text = "";
  while ((m = re.exec(html))) text += JSON.parse(m[1]);
  return text;
}

// Pull one balanced {...} object out of a text blob, starting at `from`.
function sliceObject(text, from) {
  let depth = 0, inStr = false, esc = false;
  for (let i = from; i < text.length; i++) {
    const c = text[i];
    if (inStr) {
      if (esc) esc = false;
      else if (c === "\\") esc = true;
      else if (c === '"') inStr = false;
    } else if (c === '"') inStr = true;
    else if (c === "{") depth++;
    else if (c === "}" && --depth === 0) return text.slice(from, i + 1);
  }
  return null;
}

// Every object in the RSC text that starts with {"__typename":"<type>"
export function rscObjects(text, typename) {
  const needle = `{"__typename":"${typename}"`;
  const out = [];
  let i = 0;
  while ((i = text.indexOf(needle, i)) !== -1) {
    const raw = sliceObject(text, i);
    if (raw) { try { out.push(JSON.parse(raw)); } catch { /* skip partial chunks */ } }
    i += needle.length;
  }
  return out;
}

The brace slicer is a bit crude, but the payload is machine-generated JSON, so a string-aware depth counter is enough. If a slice fails to parse, we skip it.

Step 3: Find sellers from the sitemap

seller-sitemap.txt is a plain list of profile URLs. On October 28, 2026 it had 7,740 handles.

// ---------- Step 3: discover sellers from the sitemap robots.txt points to ----------

export async function sellerHandles() {
  const txt = await get(`${BASE}/seller-sitemap.txt`);
  return txt
    .split(/\r?\n/)
    .map((l) => l.trim().match(/\/user\/([^/?#]+)$/)?.[1])
    .filter(Boolean);
}

$ node whatnot.mjs sellers 5
{
  "count": 7740,
  "sample": ["001spades", "1099comics", "10k_sportscards", "112comics", "123readycomics"]
}

The sitemap is stale in places. In two random samples, 4 of 14 handles returned 404 (renamed or closed accounts), including 10k_sportscards from the sample above. That's normal. Treat 404 as data, not as an error.

Step 4: Seller profile and shows

A seller page renders whichever tab the seller picked as default. Some open on Shows, others on Shop, and the Apollo data changes with it (GetUserLiveStreams vs ProfileShop). Requesting /user/{handle}/shows forces the shows tab, so the data is always the same shape. The profile object (followers, sold count, rating) is in the RSC payload on every tab.

// ---------- Step 4: seller profile + shows from /user/{handle}/shows ----------

const money = (m) => (m ? { amount: m.amount / 100, currency: m.currency } : null); // amounts are in cents

function cleanShow(s) {
  return {
    id: s.id,
    title: s.title,
    status: s.status, // PLAYING = live now, CREATED = scheduled
    start_time: s.startTime ? new Date(s.startTime).toISOString() : null,
    active_viewers: s.activeViewers ?? null,
    bookmarks: s.totalWatchlistUsers ?? null,
    seller: s.user?.username ?? null,
    categories: (s.livestreamCategories ?? []).map((c) => c.label).join("|"),
    tags: (s.tags ?? []).map((t) => t.label).join("|"),
    url: `${BASE}/live/${s.id}`,
  };
}

export async function seller(handle) {
  // Plain /user/{handle} renders whichever tab the seller picked (shows or shop).
  // /shows forces the shows tab, so the Apollo data is always GetUserLiveStreams.
  const html = await get(`${BASE}/user/${encodeURIComponent(handle)}/shows`);
  if (!html) return null; // 404: renamed, banned, or deleted

  const user = rscObjects(rscText(html), "PublicUserNode").find(
    (u) => u.username?.toLowerCase() === handle.toLowerCase() && "followerCount" in u
  );
  if (!user) throw new Error(`no profile object on /user/${handle} (layout changed?)`);

  const streams = apolloResults(html).find((d) => d.getUser?.livestreams)?.getUser.livestreams;

  return {
    handle: user.username,
    id: Buffer.from(user.id, "base64").toString().split(":")[1] ?? user.id, // "PublicUserNode:190554"
    display_name: user.displayName?.trim() || null,
    bio: user.bio || null,
    followers: user.followerCount,
    following: user.followingCount,
    sold: user.soldCount, // null when the seller hides it / has none
    rating: user.sellerRating?.overall ?? null,
    reviews: user.sellerRating?.numReviews ?? 0,
    avg_ship_days: user.averageShipDays,
    is_live: user.isLive,
    verified: user.isVerifiedSeller,
    premier: !!user.premierShopStatus?.isPremierShop,
    shows_total: streams?.totalCount ?? null,
    shows: (streams?.edges ?? []).map((e) => cleanShow(e.node)), // live + scheduled, first 15
  };
}

Two things in there are easy to miss:

IDs are base64 Relay IDs. UHVibGljVXNlck5vZGU6MTkwNTU0 decodes to PublicUserNode:190554. Keep the numeric part. It doesn't change when a seller renames their account, and the handle does.
Money is in cents. {"amount": 5500, "currency": "USD"} is $55.00. There's also a localizedPrice converted to your currency based on your IP (I got VND from Vietnam). Always store price, never localizedPrice.

A seller who was live while I ran this:

$ node whatnot.mjs seller hititfromthepack
{
  "handle": "hititfromthepack",
  "id": "25944732",
  "display_name": "HitItFromThePack",
  "followers": 16358,
  "following": 128,
  "sold": 4407,
  "rating": 4.9,
  "reviews": 753,
  "avg_ship_days": 1,
  "is_live": true,
  "verified": false,
  "premier": true,
  "shows_total": 7,
  "shows": [
    {
      "id": "359ac220-1a62-435d-9e2a-50c00ab85c4c",
      "title": "🇺🇸 MY BIGGEST $1 START STREAM EVER! $100K+ MID ERA",
      "status": "PLAYING",
      "start_time": "2026-10-28T02:02:51.692Z",
      "active_viewers": 158,
      "bookmarks": 95,
      "categories": "Pokémon Cards",
      "tags": "$1 Starts|Sudden Death|Vintage",
      ...
    },
    {
      "id": "a1854a1a-e238-421c-b057-99162b365ae5",
      "status": "CREATED",
      "start_time": "2026-10-29T19:38:00.000Z",
      "active_viewers": 0,
      "bookmarks": 58,
      ...
    }
  ]
}

PLAYING means live now, CREATED means scheduled. The seven shows were one live and six upcoming, so the profile shows live and scheduled streams, not the archive. bookmarks (totalWatchlistUsers) on a scheduled show is a nice early signal of how big that show will be.

And a random sample from the sitemap, via node whatnot.mjs sellers-csv 6:

handle,followers,sold,rating,reviews,avg_ship_days,is_live,premier,shows_total
sundazebest,7839,13691,4.9,2878,,false,false,0
comicselitelive,25574,91897,5,17060,0,true,false,51
king_espeo,197,117,5,43,,false,false,0
bfunko_collectibles,1262,414,4.8,149,,false,false,0

(The other two of the six were 404s.) sold is lifetime items sold, which is the most useful single number for sizing a seller. avg_ship_days is often null.

The seller's shop

Many sellers also keep buy-it-now and auction listings in a shop. /user/{handle}/shop ships the first 20 of them:

export async function shop(handle) {
  const html = await get(`${BASE}/user/${encodeURIComponent(handle)}/shop`);
  if (!html) return null;
  const feed = apolloResults(html).find((d) => d.getUser?.searchShop)?.getUser.searchShop;
  return flattenFeed(feed).listings; // first 20 listings
}

$ node whatnot.mjs shop bangarangcards
[
  { "title": "Typhoid Mary", "price": 5, "currency": "USD", "type": "BUY_IT_NOW", "status": "ACTIVE", ... },
  { "title": "Thena", "price": 15, "currency": "USD", "type": "BUY_IT_NOW", "status": "ACTIVE", ... },
  ...
]

Step 5: Search and category feeds

Search (/search?query=...) and category pages (/tag/{name}) both render a Feed. Its edges are a mix: FeedEntity wrappers around listings or livestreams, plus Section carousels. We flatten them and keep the three types we care about.

// ---------- Step 5: search and category feeds ----------

function cleanListing(l) {
  return {
    id: l.id,
    title: l.title,
    subtitle: l.subtitle ?? null,
    price: money(l.price)?.amount ?? null,
    currency: l.price?.currency ?? null,
    type: l.transactionType, // BUY_IT_NOW or AUCTION
    status: l.listingStatus ?? l.publicStatus,
    current_bid: money(l.currentBid)?.amount ?? null,
    bids: l.currentBidCount ?? null,
    quantity: l.quantity,
    seller: l.user?.username ?? null,
    url: `${BASE}/listing/${encodeURIComponent(l.id)}`,
  };
}

// Feeds mix listings, livestreams, users and "Section" carousels. Flatten them.
function flattenFeed(feed) {
  const shows = [], listings = [], users = [];
  const visit = (o) => {
    if (!o) return;
    if (o.__typename === "LiveStream") shows.push(cleanShow(o));
    else if (o.__typename === "ListingNode") listings.push(cleanListing(o));
    else if (o.__typename === "PublicUserNode") users.push({ handle: o.username, followers: o.followerCount, sold: o.soldCount });
  };
  for (const { node } of feed?.objects?.edges ?? []) {
    if (node.__typename === "FeedEntity") visit(node.object);
    else visit(node);
  }
  const dedupe = (arr) => [...new Map(arr.map((x) => [x.id ?? x.handle, x])).values()];
  return { title: feed?.title, hasNextPage: feed?.objects?.pageInfo?.hasNextPage ?? false,
    shows: dedupe(shows), listings: dedupe(listings), users: dedupe(users) };
}

export async function search(query) {
  const html = await get(`${BASE}/search?query=${encodeURIComponent(query)}&referringSource=typed`);
  const feed = apolloResults(html).find((d) => d.search?.objects)?.search;
  return flattenFeed(feed);
}

export async function category(tag) {
  const html = await get(`${BASE}/tag/${encodeURIComponent(tag)}`);
  if (!html) return null;
  const feed = apolloResults(html).find((d) => d.feed?.objects)?.feed;
  return flattenFeed(feed);
}

$ node whatnot.mjs search charizard
# 18 listings, hasNextPage: true
{ "title": "Charizard Expedition Reverse Holo", "subtitle": "Moderately Played", "price": 175, "currency": "USD", "seller": "bigmacspokeshack", ... }
{ "title": "Charizard ex Special Collection", "subtitle": "New ∙ Journey Together", "price": 50, "currency": "USD", "seller": "takeflightcards", ... }

$ node whatnot.mjs category pokemon_cards
# "Pokémon Cards": 24 shows, all PLAYING
{ "seller": "hititfromthepack", "active_viewers": 160, "bookmarks": 95, "title": "🇺🇸 MY BIGGEST $1 START STREAM EVER! ..." }
{ "seller": "koheitcg_jp", "active_viewers": 22, "tags": "Pokémon|Pokemon 30th Anniversary|$1 Starts|Shop from Japan|...", ... }

The limit: you only get the first page. Search gives about 18 to 24 items, and a category gives 24 shows. Later pages and the vertical filters (shows only, sellers only) are loaded by the browser through the Kasada-protected GraphQL endpoint. Adding &vertical=LIVESTREAM to the URL did nothing in my tests. For monitoring "what's big right now" the first page is fine. For a full catalog it isn't.

Also, the category feed is ranked, not sorted by viewers. Sort it yourself.

Step 6: A single listing

Listing pages carry the full ListingNode in the RSC payload, including description, created date, condition and other attributes. The node doesn't include its seller. The page passes the seller as a separate "seller":{...} prop, so we grab that too.

// ---------- Step 6: a single listing ----------

export async function listing(idOrUrl) {
  const id = decodeURIComponent(idOrUrl.split("/listing/").pop());
  const html = await get(`${BASE}/listing/${encodeURIComponent(id)}`);
  if (!html) return null;
  const text = rscText(html);
  const node = rscObjects(text, "ListingNode").find((l) => l.id === id);
  if (!node) return null;
  // The listing node itself has no user; the page passes the seller as a separate prop.
  const at = text.indexOf('"seller":{"__typename":"PublicUserNode"');
  const sellerNode = at === -1 ? null : JSON.parse(sliceObject(text, at + '"seller":'.length));
  const attrs = Object.fromEntries((node.listingAttributeValues ?? []).map((a) => [a.attribute.label, a.value]));
  return { ...cleanListing(node), seller: sellerNode?.username ?? null, description: node.description, created_at: node.createdAt,
    offers_allowed: node.transactionProps?.isOfferable ?? null, attributes: attrs };
}

$ node whatnot.mjs listing "https://www.whatnot.com/listing/TGlzdGluZ05vZGU6MjQ0ODA3NTg0MQ=="
{
  "id": "TGlzdGluZ05vZGU6MjQ0ODA3NTg0MQ==",
  "title": "Charizard Expedition Reverse Holo",
  "subtitle": "Moderately Played",
  "price": 175,
  "currency": "USD",
  "type": "BUY_IT_NOW",
  "status": "ACTIVE",
  "quantity": 1,
  "seller": "bigmacspokeshack",
  "description": "Charizard Expedition Reverse Holo\nMP",
  "created_at": "Fri, 16 Oct 2026 20:25:26 GMT",
  "offers_allowed": true,
  "attributes": { "Condition": "Moderately Played" }
}

Listing IDs are base64 too (ListingNode:2448075841), and they can end in =. Encode them when you build URLs.

Step 7: A live-viewer tracker

Now the fun part. Live shopping is all about who's live and how many people are watching. Snapshot a category every few minutes and append each live show to a CSV:

// ---------- Step 7: live-viewer tracker ----------

export const csvCell = (v) => (v == null ? "" : /[",\n]/.test(String(v)) ? `"${String(v).replace(/"/g, '""')}"` : String(v));

export async function track(tag, { every = 10, times = 3, file = "live-snapshots.csv" } = {}) {
  const cols = ["snapshot_at", "tag", "show_id", "seller", "status", "active_viewers", "bookmarks", "start_time", "title"];
  if (!fs.existsSync(file)) fs.writeFileSync(file, cols.join(",") + "\n");
  for (let n = 1; n <= times; n++) {
    const at = new Date().toISOString();
    const feed = await category(tag);
    const live = (feed?.shows ?? []).filter((s) => s.status === "PLAYING");
    const rows = live.map((s) => [at, tag, s.id, s.seller, s.status, s.active_viewers, s.bookmarks, s.start_time, s.title]);
    fs.appendFileSync(file, rows.map((r) => r.map(csvCell).join(",")).join("\n") + (rows.length ? "\n" : ""));
    const viewers = live.reduce((a, s) => a + (s.active_viewers ?? 0), 0);
    console.error(`[${at}] ${tag}: ${live.length} live shows, ${viewers} viewers in the first feed page`);
    if (n < times) await sleep(every * 60_000);
  }
}

I ran it on Pokémon cards, every 2 minutes, 3 times:

$ node whatnot.mjs track pokemon_cards 2 3
[2026-10-28T04:12:14.735Z] pokemon_cards: 24 live shows, 6096 viewers in the first feed page
[2026-10-28T04:14:15.576Z] pokemon_cards: 24 live shows, 6150 viewers in the first feed page
[2026-10-28T04:16:16.457Z] pokemon_cards: 24 live shows, 7125 viewers in the first feed page

snapshot_at,tag,show_id,seller,status,active_viewers,bookmarks,start_time,title
2026-10-28T04:12:14.735Z,pokemon_cards,359ac220-...,hititfromthepack,PLAYING,155,95,2026-10-28T02:02:51.692Z,🇺🇸 MY BIGGEST $1 START ...
2026-10-28T04:12:14.735Z,pokemon_cards,729592e6-...,koheitcg_jp,PLAYING,24,21,2026-10-28T03:00:17.480Z,🇯🇵 Superbly Spectacular ...
2026-10-28T04:12:14.735Z,pokemon_cards,e3538101-...,clovrcards,PLAYING,208,152,2026-10-27T19:04:20.600Z,🇺🇸 🐉 EX EMERALD BOX ...

Grouping by seller across the three snapshots already tells a story:

spacenarwhalz      2043 → 1993 → 1963
novatcg            1075 → 1137 → 1252
saw_tcg             160 →  154 → 1096
shop_exp            531 →  551 →  529

A jump like saw_tcg's is exactly what you'd want an alert for. 27 different sellers showed up, and 22 of them were in all three snapshots. The other five moved in or out of the first page as the ranking shifted. That churn is a property of the feed, not of the shows, so don't read a missing row as "the show ended". To confirm that, check the seller's /shows page.

For a real tracker, run it from cron every 10 to 15 minutes, not every 2. That's 4 to 6 requests an hour per category, which is about as gentle as it gets.

Step 8: The CLI

A tiny command switch ties it together. Paste the blocks above into one whatnot.mjs (Node 18+, ES module, no dependencies), add this at the bottom, and run node whatnot.mjs for usage.

// ---------- CLI ----------

async function main() {
  const [cmd, ...args] = process.argv.slice(2);
  const print = (x) => console.log(JSON.stringify(x, null, 2));
  switch (cmd) {
    case "sellers": {
      const handles = await sellerHandles();
      print({ count: handles.length, sample: handles.slice(0, Number(args[0] ?? 10)) });
      break;
    }
    case "sellers-csv": {
      // Profile a random sample of sitemap sellers into sellers.csv
      const n = Number(args[0] ?? 10);
      const handles = (await sellerHandles()).sort(() => Math.random() - 0.5).slice(0, n);
      const cols = ["handle", "followers", "sold", "rating", "reviews", "avg_ship_days", "is_live", "premier", "shows_total"];
      const lines = [cols.join(",")];
      for (const h of handles) {
        const s = await seller(h).catch((e) => { if (e instanceof BlockedError) throw e; return null; });
        if (s) lines.push(cols.map((c) => csvCell(s[c])).join(","));
        console.error(`${h}: ${s ? `${s.followers} followers, ${s.sold ?? "?"} sold` : "skipped"}`);
      }
      fs.writeFileSync("sellers.csv", lines.join("\n") + "\n");
      break;
    }
    case "seller":
      for (const h of args) {
        try { print(await seller(h) ?? { handle: h, error: "not found" }); }
        catch (e) { print({ handle: h, error: e.message }); if (e instanceof BlockedError) return; }
      }
      break;
    case "shop": print(await shop(args[0])); break;
    case "search": print(await search(args.join(" "))); break;
    case "category": print(await category(args[0])); break;
    case "listing": print(await listing(args[0])); break;
    case "track": await track(args[0], { every: Number(args[1] ?? 10), times: Number(args[2] ?? 3) }); break;
    default:
      console.log("usage: node whatnot.mjs sellers [n] | sellers-csv [n] | seller <handle...> | shop <handle> | search <query> | category <tag> | listing <id|url> | track <tag> [everyMin] [times]");
  }
}

if (import.meta.url === `file://${process.argv[1]}`) main().catch((e) => { console.error(e.message); process.exit(1); });

Pitfalls, in one place

Terms of Service. Whatnot's terms prohibit automated access. robots.txt allowing a path doesn't change that. Keep it small, slow, and non-commercial, or use the official Seller API for your own data.
Cloudflare checks if your client is consistent. A fake browser UA from Node got challenged every time. An honest UA passed. If you get challenged anyway, back off and stop.
GraphQL is off-limits in practice. The browser's calls go through Kasada-protected /services/graphql, and introspection is disabled. Everything here comes from server-rendered HTML.
First page only. Search and category pages give roughly one page of results (about 24). No deep pagination without the browser's GraphQL calls.
Default tabs differ per seller. Use /user/{handle}/shows and /user/{handle}/shop explicitly.
Money is in cents, and localizedPrice follows your IP. Store price.amount / 100 with price.currency.
IDs are base64 Relay IDs. Decode user IDs and key on the number. Handles change.
Sitemaps are stale. Expect a share of 404s (4 of 14 in my samples).
Feeds are ranked and churn. Sort by viewers yourself, and don't infer "ended" from absence.
The markup will change. The parsers key on __typename and Apollo query results, which are more stable than CSS classes. If seller() throws "no profile object", look at the page again.

When DIY stops being worth it

The single-file scraper is fine for a few sellers and a category or two. It gets painful when you need page 2+ of search, a seller's full show history and reviews, or thousands of profiles a day without tripping Cloudflare.

For that I use ScrapeAtlas*, which has Whatnot endpoints with real cursor pagination. GET /v1/whatnot/search takes a query, a vertical (UNIVERSAL, LIVESTREAM, PRODUCT, USER, or CATEGORY), and a cursor, and returns 24 records per page. /v1/whatnot/profile takes a handle, user_id, or url, and there are /v1/whatnot/user/shows, /user/listings, /user/reviews, and /user/clips per seller, plus /v1/whatnot/category, /livestream, /livestream/listings, /listing, /product, and /categories. Auth is an x-api-key header, and failed requests don't cost credits.

If you only need what's on the first page, though, the DIY route above works fine.

Wrapping up

The official Whatnot API is a Seller API. Great for your own store, no help for market data.
The data is in the HTML. Apollo SSR blocks for feeds, the RSC payload for profiles and listings. No headless browser needed.
Be honest and slow. An honest User-Agent passed Cloudflare where a fake Chrome one didn't. Keep to one request every few seconds.
Know the limits. First page only, ranked feeds, prices in cents, base64 IDs, stale sitemaps.
Live viewers are the interesting metric. A tiny cron job gives you a category-level view of who's pulling a crowd.

What would you track on Whatnot: seller growth, prices for one card, or live viewers by category? Tell me in the comments.

*I build it.

How to Scrape Mastodon with Go: Profiles, Posts, Hashtags Across Servers, and Trends (2026)

Viktor — Mon, 26 Oct 2026 04:10:11 +0000

Scraping Mastodon is unusual in a good way: you don't need to parse HTML at all. Every Mastodon server exposes the same public REST API, and most read endpoints answer without a token. The hard part isn't getting data. It's that there is no single "Mastodon". There are thousands of servers, each with its own partial view of the network, its own rate limits, and its own settings.

In this post I'll build a small Mastodon scraper in Go, standard library only, that:

Looks up a profile on the account's home server
Pages through a user's original posts for the last N days and computes engagement
Collects a hashtag from several servers and de-duplicates the results
Reads trending hashtags per server
Writes everything to CSV

Everything below was run on October 26, 2026, and the output is real.

Before you start: read the server's rules and terms, keep request rates low, and respect people's choices. Mastodon accounts can set noindex (opt out of search engines) and indexable flags; the code below skips noindex accounts. Don't build profiles of individual people, and don't republish their posts out of context.

How the Mastodon API looks from a scraper's point of view

All the endpoints in this post are plain GET requests that return JSON:

What	Endpoint (on any server)	Auth?
Profile by handle	`/api/v1/accounts/lookup?acct=user`	No
A user's posts	`/api/v1/accounts/:id/statuses`	No
Hashtag timeline	`/api/v1/timelines/tag/:tag`	Usually no
Trending tags	`/api/v1/trends/tags`	No
Public/federated timeline	`/api/v1/timelines/public`	Often yes
Full-text post search	`/api/v2/search?type=statuses`	Yes

The last two rows are the first surprise. On mastodon.social, the public timeline answers HTTP 422 with {"error":"This method requires an authenticated user"}, and status search returns an empty statuses array when you're logged out. Server admins can turn these on or off, so check them on each server you target.

Pagination isn't in the JSON body. It's in the Link response header, like GitHub's API:

Link: <https://mastodon.social/api/v1/accounts/1/statuses?max_id=115...>; rel="next", <...>; rel="prev"

Rate limits are in headers too. Unauthenticated requests on mastodon.social got x-ratelimit-limit: 300 per five-minute window, with x-ratelimit-remaining and an x-ratelimit-reset timestamp.

Step 1: A polite HTTP helper

mkdir mastoscraper && cd mastoscraper
go mod init mastoscraper

The helper sends a real User-Agent (put your own contact URL in it), retries on network errors and 5xx, sleeps until the reset time on a 429, and slows down on its own when fewer than 20 requests are left in the window. It returns the response headers, because we need Link for pagination.

package main

import (
    "encoding/csv"
    "encoding/json"
    "errors"
    "flag"
    "fmt"
    "html"
    "net/http"
    "net/url"
    "os"
    "regexp"
    "sort"
    "strconv"
    "strings"
    "time"
)

const userAgent = "mastoscraper/0.1 (+https://example.com/contact)"

var client = &http.Client{Timeout: 20 * time.Second}

// getJSON fetches a URL, decodes JSON into out, and returns the response
// headers so callers can read Link (pagination) and X-RateLimit-*.
func getJSON(u string, out any) (http.Header, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, _ := http.NewRequest("GET", u, nil)
        req.Header.Set("User-Agent", userAgent)
        req.Header.Set("Accept", "application/json")
        res, err := client.Do(req)
        if err != nil {
            time.Sleep(time.Duration(attempt+1) * 2 * time.Second)
            continue
        }
        if res.StatusCode == 429 {
            wait := untilReset(res.Header)
            fmt.Fprintf(os.Stderr, "429 from %s, sleeping %s\n", res.Request.URL.Host, wait)
            res.Body.Close()
            time.Sleep(wait)
            continue
        }
        if res.StatusCode >= 500 {
            res.Body.Close()
            time.Sleep(time.Duration(attempt+1) * 3 * time.Second)
            continue
        }
        if res.StatusCode != 200 {
            res.Body.Close()
            return res.Header, fmt.Errorf("GET %s: HTTP %d", u, res.StatusCode)
        }
        err = json.NewDecoder(res.Body).Decode(out)
        res.Body.Close()
        // Be polite even when we have budget left.
        if rem, _ := strconv.Atoi(res.Header.Get("X-RateLimit-Remaining")); rem > 0 && rem < 20 {
            time.Sleep(untilReset(res.Header))
        }
        return res.Header, err
    }
    return nil, errors.New("gave up after retries: " + u)
}

func untilReset(h http.Header) time.Duration {
    if t, err := time.Parse(time.RFC3339Nano, h.Get("X-RateLimit-Reset")); err == nil {
        if d := time.Until(t); d > 0 {
            return d + time.Second
        }
    }
    return 30 * time.Second
}

// nextURL pulls the rel="next" URL out of a Link header like
// <https://host/api/v1/...?max_id=123>; rel="next", <...>; rel="prev"
var linkNext = regexp.MustCompile(`<([^>]+)>;\s*rel="next"`)

func nextURL(h http.Header) string {
    m := linkNext.FindStringSubmatch(h.Get("Link"))
    if m == nil {
        return ""
    }
    return m[1]
}

Step 2: Types and small helpers

Mastodon returns post bodies as HTML (<p>, links, mention spans), so plainText strips tags and unescapes entities. splitHandle accepts @user@server, user@server, or a profile URL.

type Account struct {
    ID             string `json:"id"`
    Acct           string `json:"acct"`
    DisplayName    string `json:"display_name"`
    URL            string `json:"url"`
    FollowersCount int    `json:"followers_count"`
    FollowingCount int    `json:"following_count"`
    StatusesCount  int    `json:"statuses_count"`
    LastStatusAt   string `json:"last_status_at"`
    Bot            bool   `json:"bot"`
    Indexable      *bool  `json:"indexable"`
    Noindex        *bool  `json:"noindex"`
}

type Status struct {
    ID              string    `json:"id"`
    URI             string    `json:"uri"`
    URL             string    `json:"url"`
    CreatedAt       time.Time `json:"created_at"`
    Content         string    `json:"content"`
    Language        string    `json:"language"`
    RepliesCount    int       `json:"replies_count"`
    ReblogsCount    int       `json:"reblogs_count"`
    FavouritesCount int       `json:"favourites_count"`
    Account         Account   `json:"account"`
    Reblog          *Status   `json:"reblog"`
    Tags            []struct {
        Name string `json:"name"`
    } `json:"tags"`
}

var tagRe = regexp.MustCompile(`<[^>]+>`)

func plainText(h string) string {
    h = strings.ReplaceAll(h, "</p><p>", "\n\n")
    h = strings.ReplaceAll(h, "<br />", "\n")
    return strings.TrimSpace(html.UnescapeString(tagRe.ReplaceAllString(h, "")))
}

// splitHandle turns "@user@server", "user@server" or a profile URL into (user, server).
func splitHandle(handle string) (string, string, error) {
    if strings.HasPrefix(handle, "https://") {
        u, err := url.Parse(handle)
        if err != nil {
            return "", "", err
        }
        return strings.TrimPrefix(strings.Trim(u.Path, "/"), "@"), u.Host, nil
    }
    parts := strings.Split(strings.TrimPrefix(handle, "@"), "@")
    if len(parts) != 2 {
        return "", "", fmt.Errorf("want user@server, got %q", handle)
    }
    return parts[0], parts[1], nil
}

Step 3: Profiles and posts, from the home server

Any server can look up any account it knows about. mastodon.social will happily answer lookup?acct=golang@hachyderm.io. But that answer is mastodon.social's cached copy, and it's only as fresh as the last time mastodon.social refreshed it. In my checks today the numbers matched, but nothing guarantees that, and copies of posts on other servers definitely lag behind on favourites and boosts. So the code asks the account's own server.

exclude_replies=true&exclude_reblogs=true keeps only original posts, limit=40 is the maximum page size, and the loop stops when posts get older than the cutoff. Posts come back newest first, so it can stop early instead of paging through years of history.

// lookup asks the account's HOME server, not mastodon.social, so the counts are authoritative.
func lookup(handle string) (Account, string, error) {
    user, server, err := splitHandle(handle)
    if err != nil {
        return Account{}, "", err
    }
    var a Account
    _, err = getJSON(fmt.Sprintf("https://%s/api/v1/accounts/lookup?acct=%s", server, url.QueryEscape(user)), &a)
    return a, server, err
}

func userPosts(handle string, maxPages int, since time.Time) ([]Status, error) {
    a, server, err := lookup(handle)
    if err != nil {
        return nil, err
    }
    if a.Noindex != nil && *a.Noindex {
        return nil, fmt.Errorf("%s has opted out of search indexing (noindex); skipping", handle)
    }
    next := fmt.Sprintf("https://%s/api/v1/accounts/%s/statuses?limit=40&exclude_replies=true&exclude_reblogs=true", server, a.ID)
    var all []Status
    for page := 0; next != "" && page < maxPages; page++ {
        var batch []Status
        h, err := getJSON(next, &batch)
        if err != nil {
            return all, err
        }
        stop := false
        for _, s := range batch {
            if s.CreatedAt.Before(since) {
                stop = true
                break
            }
            all = append(all, s)
        }
        if stop || len(batch) == 0 {
            break
        }
        next = nextURL(h)
    }
    return all, nil
}

$ ./mastoscraper profile @Gargron@mastodon.social
Eugen Rochko (Gargron) on mastodon.social
  followers 383376 · following 764 · posts 82664 · last post 2026-10-26 · bot=false

$ ./mastoscraper posts -days 30 -pages 3 @Gargron@mastodon.social
34 original posts in the last 30 days
avg favourites 184.7 · avg boosts 61.3
  1012 ★ 451 ⟳  Heh. I misremembered the date--thought it was today, but loo…
   961 ★ 450 ⟳  "AI" is a failed technology, in the sense that it's neither …
   468 ★  95 ⟳  #SilentSunday

Note that last_status_at is a date, not a timestamp. Mastodon rounds it on purpose.

Step 4: One hashtag, several servers, no duplicates

This is where federation matters. A hashtag timeline on a server only contains posts that server has seen: posts by its own users, plus posts from accounts someone on that server follows. #golang on mastodon.social, fosstodon.org, and hachyderm.io are three different, overlapping lists.

So the code reads the tag from each server and de-duplicates by uri. The uri is the post's canonical ActivityPub id, and it's the same on every server. The id field is not: each server assigns its own local id to its copy of a post. When the same post shows up more than once, it keeps the copy with the highest counts, since remote copies are the ones that lag.

// tagAcrossServers reads the same hashtag from several servers and de-duplicates by URI,
// because each server only sees the posts that federated to it.
func tagAcrossServers(tag string, servers []string, maxPages int) (map[string]Status, map[string]int) {
    seen := map[string]Status{}
    perServer := map[string]int{}
    for _, srv := range servers {
        next := fmt.Sprintf("https://%s/api/v1/timelines/tag/%s?limit=40", srv, url.PathEscape(tag))
        for page := 0; next != "" && page < maxPages; page++ {
            var batch []Status
            h, err := getJSON(next, &batch)
            if err != nil {
                fmt.Fprintf(os.Stderr, "%s: %v\n", srv, err)
                break
            }
            for _, s := range batch {
                perServer[srv]++
                prev, ok := seen[s.URI]
                // Keep the copy with the highest counts; remote copies lag behind.
                if !ok || s.FavouritesCount+s.ReblogsCount > prev.FavouritesCount+prev.ReblogsCount {
                    seen[s.URI] = s
                }
            }
            if len(batch) == 0 {
                break
            }
            next = nextURL(h)
        }
    }
    return seen, perServer
}

The acct field has its own trap. It's relative to the server you asked: an account local to hachyderm.io shows up as plain golang when you ask hachyderm.io, and as golang@hachyderm.io everywhere else. If you mix servers, normalize it:

// fullAcct makes "user" (local to whichever server we asked) into "user@server".
func fullAcct(a Account) string {
    if strings.Contains(a.Acct, "@") {
        return a.Acct
    }
    if u, err := url.Parse(a.URL); err == nil {
        return a.Acct + "@" + u.Host
    }
    return a.Acct
}

func short(s string, n int) string {
    s = strings.ReplaceAll(s, "\n", " ")
    if r := []rune(s); len(r) > n {
        return string(r[:n]) + "…"
    }
    return s
}

$ ./mastoscraper tag -pages 2 -out golang.csv golang
  mastodon.social  80 posts
  fosstodon.org    80 posts
  hachyderm.io     80 posts
240 posts fetched, 106 unique after de-duplicating by URI
    34 ★ resetreboot@masto.astropirad… Bueno, mastodones. Hoy por sorpresa y sin dar ni l…
    27 ★ golang@hachyderm.io          🥳 Go 1.27.3 and 1.26.10 are released!  🔏 Security:…
     7 ★ omkar_foss@mastodon.social   Dear #fediverse,  Introducing gracias, a #noai cli…
     6 ★ skybert@hachyderm.io         Lock down your website to a dedicated few browsers…
     5 ★ haitchfive@oldbytes.space    The Bathroom  Originally by Matthew Smith.  #zenpl…

240 posts fetched, 106 unique. More than half of what you download from a second and third server is stuff you already have. That's normal, and it's why de-duplicating by uri isn't optional. Also notice that the top post is in Spanish and not really about Go: hashtags are free text, so filter by language and your own keywords if you need precision.

Step 5: Trending hashtags per server

/api/v1/trends/tags returns each tag with a history array of daily uses and accounts. The first trap: history[0] is today, which is still being counted. My first run printed #tatort (0) and #FineArtAPlaceOrDestination (218) from today's partial numbers; switching to history[1] (yesterday, a complete day) gives comparable numbers. A tag can also trend today and still show 0 for yesterday, as you'll see below.

type TrendTag struct {
    Name    string `json:"name"`
    History []struct {
        Day      string `json:"day"`
        Uses     string `json:"uses"`
        Accounts string `json:"accounts"`
    } `json:"history"`
}

$ ./mastoscraper trends
mastodon.social  #SilentSunday (884), #audi (189), #OppositesAMovie (713), #FineArtAPlaceOrDestination (0), #tatort (72)
fosstodon.org    #SilentSunday (696), #Monsterdon (25), #Serbia (179), #CrispSandwichDay (51), #Tatort (31)
hachyderm.io     #silentsunday (717), #OppositesAMovie (641), #audi (113), #FineArtAPlaceOrDestination (0), #my9albums (45)

Trends are per server and moderated: admins review tags before they can trend. So "trending on mastodon.social" isn't "trending on Mastodon", and case varies (#SilentSunday vs #silentsunday). Lowercase tag names before you compare servers.

Step 6: CSV output and the CLI

func writeCSV(path string, posts []Status) error {
    f, err := os.Create(path)
    if err != nil {
        return err
    }
    defer f.Close()
    w := csv.NewWriter(f)
    w.Write([]string{"created_at", "author", "url", "favourites", "boosts", "replies", "language", "text"})
    for _, s := range posts {
        w.Write([]string{
            s.CreatedAt.Format(time.RFC3339), fullAcct(s.Account), s.URL,
            strconv.Itoa(s.FavouritesCount), strconv.Itoa(s.ReblogsCount), strconv.Itoa(s.RepliesCount),
            s.Language, plainText(s.Content),
        })
    }
    w.Flush()
    return w.Error()
}

func main() {
    if len(os.Args) < 2 {
        fmt.Println("usage: mastoscraper profile|posts|tag|trends ...")
        os.Exit(1)
    }
    cmd := os.Args[1]
    fs := flag.NewFlagSet(cmd, flag.ExitOnError)
    pages := fs.Int("pages", 3, "max pages per source")
    days := fs.Int("days", 30, "only posts from the last N days")
    servers := fs.String("servers", "mastodon.social,fosstodon.org,hachyderm.io", "comma-separated servers")
    out := fs.String("out", "", "write CSV here")
    fs.Parse(os.Args[2:])
    arg := fs.Arg(0)

    switch cmd {
    case "profile":
        a, server, err := lookup(arg)
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
        fmt.Printf("%s (%s) on %s\n  followers %d · following %d · posts %d · last post %s · bot=%v\n",
            a.DisplayName, a.Acct, server, a.FollowersCount, a.FollowingCount, a.StatusesCount, a.LastStatusAt, a.Bot)
    case "posts":
        since := time.Now().AddDate(0, 0, -*days)
        posts, err := userPosts(arg, *pages, since)
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
        }
        var fav, boost int
        for _, p := range posts {
            fav += p.FavouritesCount
            boost += p.ReblogsCount
        }
        fmt.Printf("%d original posts in the last %d days\n", len(posts), *days)
        if len(posts) > 0 {
            fmt.Printf("avg favourites %.1f · avg boosts %.1f\n", float64(fav)/float64(len(posts)), float64(boost)/float64(len(posts)))
            sort.Slice(posts, func(i, j int) bool { return posts[i].FavouritesCount > posts[j].FavouritesCount })
            for _, p := range posts[:min(3, len(posts))] {
                fmt.Printf("  %4d ★ %3d ⟳  %s\n", p.FavouritesCount, p.ReblogsCount, short(plainText(p.Content), 60))
            }
        }
        if *out != "" {
            writeCSV(*out, posts)
        }
    case "tag":
        seen, per := tagAcrossServers(strings.TrimPrefix(arg, "#"), strings.Split(*servers, ","), *pages)
        total := 0
        for srv, n := range per {
            fmt.Printf("  %-16s %d posts\n", srv, n)
            total += n
        }
        fmt.Printf("%d posts fetched, %d unique after de-duplicating by URI\n", total, len(seen))
        var list []Status
        for _, s := range seen {
            list = append(list, s)
        }
        sort.Slice(list, func(i, j int) bool { return list[i].FavouritesCount > list[j].FavouritesCount })
        for _, p := range list[:min(5, len(list))] {
            fmt.Printf("  %4d ★ %-28s %s\n", p.FavouritesCount, short(fullAcct(p.Account), 28), short(plainText(p.Content), 50))
        }
        if *out != "" {
            writeCSV(*out, list)
        }
    case "trends":
        for _, srv := range strings.Split(*servers, ",") {
            var tags []TrendTag
            if _, err := getJSON("https://"+srv+"/api/v1/trends/tags?limit=5", &tags); err != nil {
                fmt.Printf("%s: %v\n", srv, err)
                continue
            }
            names := []string{}
            for _, t := range tags {
                uses := "?"
                // History[0] is today (still counting); History[1] is yesterday, a full day.
                if len(t.History) > 1 {
                    uses = t.History[1].Uses
                }
                names = append(names, "#"+t.Name+" ("+uses+")")
            }
            fmt.Printf("%-16s %s\n", srv, strings.Join(names, ", "))
        }
    }
}

go build -o mastoscraper .
./mastoscraper posts -days 30 -out gargron.csv @Gargron@mastodon.social
./mastoscraper tag -servers mastodon.social,fosstodon.org -pages 2 -out golang.csv golang

Pitfalls, in one place

The public timeline and status search need auth on mastodon.social (422 and empty results when logged out). Server admins decide this.
Pagination lives in the Link header. If you only read the JSON body, you'll get one page and think you're done.
id is per server, uri is global. De-duplicate on uri.
acct is relative to the server you asked. Normalize to user@host.
Remote copies lag. Ask the home server for profiles, and keep the highest counts when merging posts.
Content is HTML. Strip it, and expect custom emoji shortcodes like :blobcat: in the text.
Trends history[0] is today, still counting.
Rate limits are per server and per IP (300 requests per 5 minutes unauthenticated on mastodon.social). Reading many servers spreads load, but be polite on each one.
Not every fediverse account is Mastodon. Misskey, Pleroma, and others federate with Mastodon but have different APIs. The lookup above only works on Mastodon-compatible servers.

When you don't want to manage servers yourself

If you'd rather not juggle per-server rate limits, Link headers, and federated copies, I also run ScrapeAtlas*, which has Mastodon endpoints that wrap the same public API. For example, /v1/mastodon/hashtag takes a tag and an optional server, and /v1/mastodon/user/posts takes a handle with filter=original and a since date:

req, _ := http.NewRequest("GET", "https://api.scrapeatlas.com/v1/mastodon/user/posts?handle=Gargron@mastodon.social&filter=original&since=2026-09-26", nil)
req.Header.Set("x-api-key", os.Getenv("SCRAPEATLAS_API_KEY"))
// The JSON has "posts" (same fields as Mastodon's API), a "cursor" for the next page,
// and "coverage.warnings" that tell you when results are partial.

There are also /v1/mastodon/profile, /post, /post/replies, /post/interactions, /user/connections, /search, /trends, and /server. But honestly, for Mastodon the DIY route above is very workable. The API is open, documented, and stable.

Wrapping up

Mastodon has a real public API. No HTML parsing, no headless browser.
Think in servers. Every server has a partial view; read several and de-duplicate by uri.
Go to the source for profiles and fresh counts.
Read the headers: Link for pages, X-RateLimit-* for pacing.
Respect opt-outs like noindex, and don't collect more than you need.

Which server-specific quirk have you hit while scraping the fediverse? Tell me in the comments and I'll add it to the list.

*I build it.

How to Scrape Poshmark with Python: Search Results, Sold Comps, Closets, and a Price-Drop Tracker (2026)

Viktor — Fri, 23 Oct 2026 04:09:15 +0000

If you resell clothes, you already know the most useful question on Poshmark: what does this actually sell for? Active listings tell you what people hope to get. Sold listings tell you what buyers paid. Put the two side by side in your own database and you can price inventory, spot underpriced items, and see which closets move stock.

Poshmark doesn't offer a public data API. But the website is a JavaScript app, and the data it shows comes from JSON endpoints that answer plain GET requests without a login. In this post I'll build a small Python Poshmark scraper that:

Searches listings with real filters (department, category, size, condition)
Pulls sold listings and turns them into price comps (25th percentile, median, 75th)
Reads a seller's closet and profile stats
Fetches a single listing by URL
Stores every run in SQLite and reports price drops

One file, one dependency (requests). Everything below was run against poshmark.com on October 23, 2026, and the output you see is real.

Before you start: read Poshmark's Terms of Service and robots.txt, keep request rates low, and only collect what you need. Profiles include sellers' real names, so don't store personal data you don't need. The endpoints in this post are internal ones the website uses. They aren't documented and can change without notice.

Your options, briefly

HTML pages. Search, listing, and closet pages all embed their data in a window.__INITIAL_STATE__ = {...} script blob (not __NEXT_DATA__). It works, but a search page is about 4 MB of HTML to get 48 listings. Also, robots.txt disallows /search.
The vm-rest JSON endpoints (this post). The site fetches its listings from https://poshmark.com/vm-rest/.... Responses are clean JSON. robots.txt doesn't mention /vm-rest (it does disallow /api, /search, /users, and a few others).
Headless browser. You don't need one. Everything here worked with plain HTTP from a server.

How a Poshmark search request looks

Here's the part that isn't obvious. The search endpoint doesn't use normal query parameters. You pass a JSON object, URL-encoded, in a single request parameter:

GET https://poshmark.com/vm-rest/posts?request={"filters":{"department":"All","inventory_status":["available"]},"query":"levis 501","experience":"all","count":48}&summarize=true

The response has a data array of listings, a total_listing_count, and a more object with a next_max_id cursor for the next page. Each listing comes with a lot of fields. These are the ones I use:

Field	What it is
`id`	24-character hex id, also the end of the listing URL
`title`, `brand`, `size`, `condition`	The basics (`brand` is free text, more on that below)
`price_amount.val`	Current price as a string, e.g. `"47.0"`
`original_price_amount.val`	The "retail" price the seller typed in, often `"0.0"`
`first_user_price_amount.val`	The price when the item was first listed
`inventory.status`	`available` or `sold_out`
`inventory.status_changed_at`	When the status last changed (for sold items: about when it sold)
`like_count`, `comment_count`, `share_count`	Engagement
`creator_username`	The seller's closet
`first_published_at`	When it was listed

Step 1: A request helper that knows Poshmark's error style

pip install requests

The first surprise: errors come back as HTTP 200. Ask for a listing that doesn't exist and you get status 200 with a body like this:

{"error":{"errorType":"NotFoundError","statusCode":404, ...},"meta":{"api_url":"/posts/000000000000000000000000"}}

If you only check res.status_code, your scraper will happily treat errors as empty results. So the helper looks inside the body:

import json, re, sqlite3, statistics, sys, time
from datetime import datetime, timezone
from decimal import Decimal

import requests

BASE = "https://poshmark.com/vm-rest"

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0 Safari/537.36",
    "Accept": "application/json",
})


class PoshmarkError(Exception):
    pass


def get(path, request=None, max_retries=4):
    """GET a vm-rest path. Errors come back as HTTP 200 with an "error" object."""
    params = {}
    if request is not None:
        params = {"request": json.dumps(request, separators=(",", ":")), "summarize": "true"}
    for attempt in range(max_retries):
        res = session.get(f"{BASE}{path}", params=params, timeout=30)
        if res.status_code == 429 or res.status_code >= 500:
            wait = 2 ** attempt * 2
            print(f"{res.status_code} on {path}, sleeping {wait}s", file=sys.stderr)
            time.sleep(wait)
            continue
        res.raise_for_status()
        data = res.json()
        if "error" in data:
            err = data["error"]
            raise PoshmarkError(f'{err.get("errorType")} ({err.get("statusCode")}) for {path}')
        return data
    raise PoshmarkError(f"gave up on {path} after {max_retries} tries")

About rate limits: I sent 30 search requests back to back (about one per second) and got 30 × 200, with no rate-limit headers in the responses. That doesn't mean there's no limit. It means I didn't hit it at that pace. I keep a 1.5-second pause between pages and the retry logic for 429/5xx, in case that changes.

Step 2: The core of the Poshmark scraper, search with filters and paging

These are the filter keys I verified by checking the results that came back:

def build_request(query=None, department="All", category=None, size=None,
                  condition=None, sold=False, sort=None, count=48):
    filters = {"department": department,
               "inventory_status": ["sold_out" if sold else "available"]}
    if category:
        filters["category_v2"] = category
    if size:
        filters["size"] = [size]
    if condition:
        filters["condition"] = [condition]
    req = {"filters": filters, "experience": "all", "count": count}
    if query:
        req["query"] = query
    if sort:
        req["sort_by"] = sort  # e.g. "added_desc", "price_asc", "price_desc"
    return req


def iter_pages(path, request, max_pages=3):
    """Yield listings page by page, following more.next_max_id."""
    for page in range(max_pages):
        data = get(path, request)
        yield from data.get("data", [])
        more = data.get("more") or {}
        if not more.get("is_next_max_id_present"):
            break
        request = {**request, "max_id": more["next_max_id"]}
        time.sleep(1.5)  # be polite


def search(query, max_pages=3, **filters):
    return iter_pages("/posts", build_request(query, **filters), max_pages)

department="Women", category="Jeans", size="28" and condition="nwt" all came back filtered correctly. Paging with max_id gave a second page of 48 with zero overlap with the first. sort_by: "added_desc" returned the newest listings first, and price_asc/price_desc sorted by price.

Things that didn't work the way I expected:

Price filters are silently ignored. I tried four plausible shapes ("price": ["25-50"], {"min":25,"max":50}, and others). Every one returned the exact same total_listing_count as no filter at all, with prices from $16 to $100. Poshmark doesn't say "unknown filter". It just ignores it. So I filter by price on my side. (The HTML search page does accept price[]=25-50 in its URL, but that's the /search path robots.txt disallows.)
count caps at 150. count: 100 returned 100 listings, count: 200 returned 150.
You probably can't page through everything. total_listing_count said 158,172 for "levis 501" and 4.7 million for "nike", but more.total was 5000 for every query I tried. That looks like a depth cap. I didn't page 5,000 deep to confirm it, but plan for narrow, filtered searches instead of "download everything".

Step 3: Clean rows (and three data quirks)

# Condition codes seen in live data. Older listings use "not_nwt", some have none.
CONDITIONS = {"nwt": "New with tags", "uln": "Like new", "ug": "Good",
              "uf": "Fair", "not_nwt": "Used (older listing)"}


def money(obj, zero_is_missing=False):
    if not obj or not obj.get("val"):
        return None
    value = Decimal(obj["val"])
    return None if zero_is_missing and value == 0 else value


def to_row(p):
    inv = p.get("inventory") or {}
    return {
        "id": p["id"],
        "title": p.get("title", "").replace("\u200e", "").strip(),
        "brand": p.get("brand"),
        "size": p.get("size"),
        "condition": CONDITIONS.get(p.get("condition"), p.get("condition") or "n/a"),
        "price": money(p.get("price_amount")),
        "original_price": money(p.get("original_price_amount"), zero_is_missing=True),
        "first_price": money(p.get("first_user_price_amount")),
        "availability": inv.get("status"),
        "availability_changed_at": inv.get("status_changed_at"),
        "likes": p.get("like_count", 0),
        "comments": p.get("comment_count", 0),
        "seller": p.get("creator_username"),
        "listed_at": p.get("first_published_at"),
        "url": f"https://poshmark.com/listing/{p['id']}",
    }

What each workaround is for:

Condition is a code. In a 150-listing "nike" sample I saw ug (96), uln (23), nwt (15), uf (6), and 10 listings with no condition key at all. The labels ("Like New", "Good", "Fair") come from a config blob inside the page HTML. Filtering on not_nwt returned listings whose condition really is not_nwt, which looks like the older used/not-used scheme.
"Retail" price is usually zero. 111 of those 150 listings had original_price_amount of "0.0", meaning the seller left it blank. If you compute "discount from retail" without handling this, you'll divide by zero or report 100% discounts.
Invisible characters in titles. Some sellers' titles contain U+200E (left-to-right mark), e.g. "Levis 501\u200e Jeans...". It's invisible on screen, but it breaks exact matching and deduping, so I strip it.

brand is free text, too. Filtering on "brand": ["Levi's"] changed the result count, but the brands that came back included Levi's 501, Levis 501, and levi's 501. Normalize brands yourself before you group by them.

The short https://poshmark.com/listing/<id> URL works. It returns a 301 redirect to the full slug URL.

Step 4: Sold comps

This is the part resellers care about. Same endpoint, with inventory_status: ["sold_out"]:

def sold_comps(query, max_pages=2, **filters):
    rows = [to_row(p) for p in search(query, max_pages=max_pages, sold=True, **filters)]
    prices = sorted(float(r["price"]) for r in rows if r["price"] is not None)
    if not prices:
        return rows, None
    q = statistics.quantiles(prices, n=4) if len(prices) >= 4 else [prices[0], prices[len(prices) // 2], prices[-1]]
    return rows, {"n": len(prices), "p25": q[0], "median": statistics.median(prices), "p75": q[-1]}

Real output for women's Levi's 501 jeans in size 28:

$ python poshmark_scraper.py comps "levis 501" --department Women --category Jeans --size 28
sold comps for 'levis 501' {'department': 'Women', 'category': 'Jeans', 'size': '28'}: {'n': 96, 'p25': 22.0, 'median': 30.0, 'p75': 40.0}
  sold 2026-10-22  $   45.0  Levi’s 501 She’s Crafty 150th Anniversary Floral Patch Jeans
  sold 2026-10-22  $   33.0  Levi's 501 90s Womens Jeans 28x32 Light Wash Straight Leg De
  sold 2026-10-18  $   30.0  Levi's 501 Light Blue Distressed Straight Women’s Jeans W28 
  sold 2026-10-21  $   44.0  Levi’s 501s
  sold 2026-10-06  $   30.0  Levi's 501 Straight Leg Button Fly Jeans Patchwork Raw Hem W

Read those numbers with care. price_amount on a sold listing is the last listed price. If the buyer paid with an accepted offer, I didn't find that amount anywhere in the response. Your "median sold price" is really a median of list prices at the time of sale, so it's probably a bit high.

Step 5: Closets and single listings

A closet uses the same request format at a different path. The profile is a separate call:

def closet(username, max_pages=3, **filters):
    return iter_pages(f"/users/{username}/posts/filtered",
                      build_request(None, **filters), max_pages)


def listing(id_or_url):
    m = re.search(r"([0-9a-f]{24})/?(?:[?#].*)?$", id_or_url)
    if not m:
        raise ValueError("expected a 24-character listing id or a /listing/ URL")
    return get(f"/posts/{m.group(1)}")  # no "data" wrapper on this one


def user(username):
    return get(f"/users/{username}")["data"]

Watch the response shapes. /users/{username} wraps the profile in data, but /posts/{id} returns the listing at the top level. The profile's aggregates object has the useful numbers: posts, followers, following, shares, seller_orders_shipped, items_sold_display (a rounded string like "100+", not an exact number), and last_active_date.

$ python poshmark_scraper.py closet shopladydeals --pages 1
shopladydeals: 617 listings, 14411 followers, items sold: 100+, last active 2026-10-21
    11 likes  $   19.0  Motherhood Maternity Pants Women's Yoga z Small Gray Full Pa
     4 likes  $   24.0  Duluth Trading Men's Long Sleeve Beige Waffle Henley Shirt S
     3 likes  $   35.0  Tommy Hilfiger Sunglasses for men Matte Black and Tortoise s

The profile also returns the seller's first and last name and some linked-account IDs. My closet command only prints aggregate counts. Do the same unless you have a real reason to keep personal data.

Step 6: A price-drop tracker in SQLite

Poshmark sellers drop prices all the time. To catch drops, save a snapshot on every run and compare each listing with its price in the previous run:

def track(query, db_path="poshmark.db", max_pages=2, **filters):
    key = json.dumps({"q": query, **filters}, sort_keys=True)
    taken_at = datetime.now(timezone.utc).replace(microsecond=0).isoformat()
    rows = [to_row(p) for p in search(query, max_pages=max_pages, **filters)]
    con = sqlite3.connect(db_path)
    con.executescript(SCHEMA)  # snapshots(taken_at, search_key, id, title, ..., price, availability, likes)
    con.executemany(
        "INSERT OR IGNORE INTO snapshots VALUES (?,?,?,?,?,?,?,?,?,?)",
        [(taken_at, key, r["id"], r["title"], r["brand"], r["size"], r["seller"],
          str(r["price"]) if r["price"] is not None else None, r["availability"], r["likes"])
         for r in rows],
    )
    con.commit()
    changes = con.execute(
        """
        WITH runs AS (
          SELECT DISTINCT taken_at FROM snapshots WHERE search_key = ?
          ORDER BY taken_at DESC LIMIT 2
        ),
        cur  AS (SELECT * FROM snapshots WHERE search_key = ? AND taken_at = (SELECT MAX(taken_at) FROM runs)),
        prev AS (SELECT * FROM snapshots WHERE search_key = ? AND taken_at = (SELECT MIN(taken_at) FROM runs))
        SELECT cur.id, cur.title, prev.price, cur.price
        FROM cur JOIN prev ON cur.id = prev.id
        WHERE (SELECT COUNT(*) FROM runs) = 2 AND CAST(cur.price AS REAL) < CAST(prev.price AS REAL)
        ORDER BY CAST(prev.price AS REAL) - CAST(cur.price AS REAL) DESC
        """, (key, key, key)).fetchall()
    con.close()
    return rows, changes

Run it:

$ python poshmark_scraper.py track "levis 501" --department Women --category Jeans --size 28 --pages 2
96 available listings for 'levis 501' {'department': 'Women', 'category': 'Jeans', 'size': '28'}
  $  16.0 (no retail)       28 Good          501 Levi's denim
  $  16.0 (no retail)       28 Good          Levis Premium 501 Light Wash Straight Leg Raw
  $  17.0 (no retail)       28 Like new      Levi's 501 High Rise Straight Cropped Jeans L
  $  19.0 (retail $75.0)    28 Good          Levi's 501 high waisted skinny women's jeans
  $  19.0 (no retail)       28 Fair          Levi’s 501 Curve Jeans
0 price drops since the previous run

Honest note: I ran it a few times a couple of minutes apart, and it found zero drops, which is what you'd expect over that short a window. Put it in cron once or twice a day and the drop list fills up. --max-price 25 filters on your side (remember that the server ignores price filters). With that flag, the same search came down to 26 listings.

Two design choices worth knowing about:

Only listings that appear in both runs are compared. Search results move around, so a listing that drops off page 2 isn't "removed". It's just out of view. Narrow filters with fewer total results make the tracker more reliable.
Prices are stored as text so Decimal values round-trip exactly. Cast them in SQL when comparing.

The full script, with a small CLI (track, comps, closet, listing), is about 240 lines.

The walls you'll hit

Undocumented endpoints. vm-rest is the website's internal API. The JSON-in-a-query-parameter format, filter names, and response shapes can change at any time. Log loudly when a field you expect is missing.
Silent filter failures. An unknown or wrong filter is ignored, not rejected. After every change, check that total_listing_count actually moved.
HTTP 200 errors. Check for the error key every time.
Search depth. more.total says 5000 even when there are millions of matches. Split big searches by category, size, or condition.
Sold prices aren't transaction prices. Accepted offers and bundle discounts aren't visible in what I pulled.
Free-text brands and missing fields. Normalize brands, and expect condition or retail price to be missing.
Scale. 30 requests at about one per second went through fine from a single server IP. Polling thousands of searches is a different story, and I didn't test it.

When DIY stops being worth it

For a few saved searches checked once or twice a day, the script above is plenty. It gets tedious when you need server-side price filters, many closets, or data the search endpoint doesn't give you, like seller reviews or Posh Parties. Then you're maintaining a reverse-engineered client.

That's the point where I'd use a hosted API. I use ScrapeAtlas* for this. Its GET /v1/poshmark/search takes query, department, category, size, condition, availability (available, sold, or all), sort, minPrice/maxPrice (so price filtering happens on the server), plus cursor and limit:

import os
import requests

def api_sold(query, cursor=None):
    params = {"query": query, "department": "Women", "category": "Jeans",
              "size": "28", "availability": "sold", "maxPrice": "60"}
    if cursor:
        params["cursor"] = cursor
    res = requests.get(
        "https://api.scrapeatlas.com/v1/poshmark/search",
        params=params,
        headers={"x-api-key": os.environ["SCRAPEATLAS_API_KEY"]},
        timeout=60,
    )
    res.raise_for_status()
    data = res.json()
    # data["status"] is "success" or "partial"; data["coverage"]["warnings"] explains gaps
    return data.get("listings", []), data.get("nextCursor")

The listings keep Poshmark's field names (price_amount, original_price_amount, inventory, like_count, ...), so most of to_row() keeps working. There are also /v1/poshmark/listing, /closet, /profile, /love-notes (the five-star reviews on a closet's About page), and /parties with /party-listings.

Wrapping up

Skip the HTML. Poshmark's site loads its data from vm-rest JSON endpoints that answer plain GET requests.
Send a JSON request parameter, and check that every filter actually changes the result count. Price filters don't.
Look for error in 200 responses.
Clean up the data: condition codes, "0.0" retail prices, free-text brands, and invisible characters in titles.
Sold comps are list prices, not exact transaction prices. They're still the best pricing signal you can get for free.

Found a filter key I missed, or a field that moved? Tell me in the comments and I'll fold it into the script.

*I build it.

How to Scrape Kalshi with Python: Markets, Order Books, Price History, and a Snapshot Tracker (2026)

Viktor — Wed, 21 Oct 2026 04:05:50 +0000

Kalshi has become one of the more interesting public datasets around. Every market is a running, money-backed estimate of how likely something is: an election, a Fed decision, tomorrow's high temperature in New York. If you do research, build dashboards, or want to compare Kalshi against polls or other prediction markets, you'll want that data in your own database.

The good news is that you don't need to scrape HTML at all. The Kalshi API serves market data publicly, with no key and no login. The less good news is that it has a handful of quirks that will quietly give you wrong numbers if you don't know about them.

In this post I'll build a small Python Kalshi scraper that:

Pages through every open market in a series
Reads the order book the right way (it only lists bids, more on that below)
Exports hourly price candles to CSV
Stores a snapshot on every run in SQLite, so you can see which markets moved

Everything here was run against the live API in October 2026.

Before you start: read Kalshi's API docs and Terms of Service, keep request rates low, and don't redistribute the data commercially without checking what you're allowed to do. This tutorial only reads public market data. It doesn't place orders and doesn't need an account.

Your options, briefly

Public market data endpoints (this post). GET requests under https://api.elections.kalshi.com/trade-api/v2 for markets, events, series, order books, trades, and candlesticks. No auth needed.
Authenticated API. You need an account and an RSA-signed API key for anything that touches your portfolio: orders, fills, balance, and a few account endpoints. You don't need it for reading markets.
WebSocket feed. Better if you want real-time ticks, but it needs an authenticated connection and you have to handle reconnects yourself. For a snapshot every few minutes, REST is simpler.
Scraping kalshi.com HTML. Don't bother. The site is a JavaScript app that calls the same API anyway.

How Kalshi data is shaped

There are three levels, and the tickers tell you where you are:

Series   KXHIGHNY                 "Highest temperature in NYC" (recurring)
Event    KXHIGHNY-26OCT21         one occurrence: Oct 21, 2026
Market   KXHIGHNY-26OCT21-T66     one yes/no contract: "above 66°?"

A market is always a binary yes/no contract that pays $1. Prices are in dollars between 0 and 1, so a YES price of 0.23 means the market puts it at roughly 23%.

To get a series ticker from a market page, look at the URL. On kalshi.com/markets/kxhighny/..., the first path segment uppercased (KXHIGHNY) is the series ticker.

Step 1: A polite request helper

Install the one dependency:

pip install requests

The Kalshi docs say a rate-limited request returns 429 with {"error": "too many requests"}, no Retry-After header, and no penalty. So you just back off and try again. Authenticated Basic-tier accounts get a read budget of about 20 requests per second. I couldn't find a documented number for anonymous calls, so I stay far below that.

import csv
import sqlite3
import sys
import time
from datetime import datetime, timezone
from decimal import Decimal

import requests

BASE = "https://api.elections.kalshi.com/trade-api/v2"

session = requests.Session()
session.headers.update({"User-Agent": "kalshi-scraper-tutorial/1.0 (you@example.com)"})


def get(path, params=None, max_retries=5):
    """GET with backoff on 429 and 5xx."""
    for attempt in range(max_retries):
        res = session.get(f"{BASE}{path}", params=params, timeout=20)
        if res.status_code == 429 or res.status_code >= 500:
            wait = 2 ** attempt
            print(f"{res.status_code} on {path}, sleeping {wait}s", file=sys.stderr)
            time.sleep(wait)
            continue
        res.raise_for_status()
        return res.json()
    raise RuntimeError(f"gave up on {path} after {max_retries} tries")

Setting a descriptive User-Agent with a contact address costs nothing and makes it easier for the other side if your script ever misbehaves.

Step 2: The core of the Kalshi scraper, paging markets with a cursor

GET /markets takes series_ticker, event_ticker, status, limit (I use 200), and cursor. Each response includes a cursor. When it's empty, you're done.

def iter_markets(series_ticker, status="open", page_size=200):
    cursor = None
    while True:
        params = {"series_ticker": series_ticker, "status": status, "limit": page_size}
        if cursor:
            params["cursor"] = cursor
        data = get("/markets", params)
        yield from data.get("markets", [])
        cursor = data.get("cursor")
        if not cursor:
            break
        time.sleep(0.25)  # be polite

Two things surprised me here.

The status you filter by isn't the status you get back. You ask for status=open, but the records come back with "status": "active". If you filter again on your side with == "open", you'll get zero rows and spend ten minutes thinking pagination is broken.

Unfiltered listings are full of combo markets. If you call /markets?status=open without a series or event, the first pages are auto-generated multi-leg "combo" markets with tickers like KXMVECROSSCATEGORY-..., mostly with zero volume. Add mve_filter=exclude to leave them out, or better, always scope by series_ticker or event_ticker.

Step 3: Turn markets into clean rows

Here's the part that bites people who copy older tutorials: Kalshi sends prices and sizes as strings, with suffixes on the field names. Older posts use integer-cent fields like yes_bid and volume. What you get now looks like this:

"yes_bid_dollars": "0.0100",
"yes_ask_dollars": "0.0200",
"last_price_dollars": "0.0200",
"volume_fp": "2428.47",
"open_interest_fp": "2252.09"

_dollars is a price, _fp is a fixed-point count (contracts can be fractional now). I parse them as Decimal so I never get 0.30000000000000004 in a report:

def d(value):
    """Kalshi sends prices and sizes as strings. Keep them exact."""
    return Decimal(value) if value not in (None, "") else None


def to_row(m):
    return {
        "ticker": m["ticker"],
        "event_ticker": m["event_ticker"],
        "title": m.get("title"),
        "subtitle": m.get("yes_sub_title"),
        "status": m.get("status"),
        "close_time": m.get("close_time"),
        "yes_bid": d(m.get("yes_bid_dollars")),
        "yes_ask": d(m.get("yes_ask_dollars")),
        "last_price": d(m.get("last_price_dollars")),
        "volume": d(m.get("volume_fp")),
        "volume_24h": d(m.get("volume_24h_fp")),
        "open_interest": d(m.get("open_interest_fp")),
    }

If you read a field that doesn't exist anymore, .get() just returns None. So when your dashboard shows empty columns after a Kalshi update, check the field names first.

Step 4: Read the order book the right way

GET /markets/{ticker}/orderbook returns this:

{"orderbook_fp": {
  "no_dollars":  [["0.9600","441.00"],["0.9700","592.00"],["0.9800","398.00"]],
  "yes_dollars": [["0.0100","286.82"]]
}}

Notice that there are no asks. Kalshi only lists bids for YES and bids for NO. Because a YES and a NO together always pay $1, a NO bid at 0.98 works the same as a YES ask at 0.02. So the best YES ask is 1 - best NO bid:

def best_levels(ticker):
    ob = get(f"/markets/{ticker}/orderbook", {"depth": 5})["orderbook_fp"]
    yes_bids = ob.get("yes_dollars") or []
    no_bids = ob.get("no_dollars") or []
    best_yes_bid = max((Decimal(p) for p, _ in yes_bids), default=None)
    best_no_bid = max((Decimal(p) for p, _ in no_bids), default=None)
    # Only bids are listed. A NO bid at 0.97 is a YES ask at 0.03.
    implied_yes_ask = (Decimal("1") - best_no_bid) if best_no_bid is not None else None
    return best_yes_bid, implied_yes_ask

Levels are sorted with the best price last, which is why I use max() instead of trusting the order. An empty side comes back as missing or empty, so handle None. In my run, one of the busiest markets had no YES bids at all, because it was all but settled at zero.

Step 5: Snapshots in SQLite

The market list only tells you the current state. To see what moved, store a snapshot on every run:

SCHEMA = """
CREATE TABLE IF NOT EXISTS snapshots (
    taken_at     TEXT NOT NULL,
    ticker       TEXT NOT NULL,
    event_ticker TEXT,
    title        TEXT,
    yes_bid      TEXT,
    yes_ask      TEXT,
    last_price   TEXT,
    volume       TEXT,
    open_interest TEXT,
    PRIMARY KEY (taken_at, ticker)
);
"""


def snapshot(series_ticker, db_path="kalshi.db"):
    taken_at = datetime.now(timezone.utc).replace(microsecond=0).isoformat()
    rows = [to_row(m) for m in iter_markets(series_ticker)]
    con = sqlite3.connect(db_path)
    con.executescript(SCHEMA)
    con.executemany(
        """INSERT OR IGNORE INTO snapshots VALUES
           (:taken_at, :ticker, :event_ticker, :title, :yes_bid, :yes_ask,
            :last_price, :volume, :open_interest)""",
        [{**r, **{k: str(r[k]) if r[k] is not None else None
                  for k in ("yes_bid", "yes_ask", "last_price", "volume", "open_interest")},
          "taken_at": taken_at} for r in rows],
    )
    con.commit()
    con.close()
    return rows

I store numbers as text so the Decimal values round-trip exactly. Cast them when you query. Here's "which markets moved the most since the last run":

WITH ranked AS (
  SELECT ticker, title, taken_at, CAST(last_price AS REAL) AS price,
         ROW_NUMBER() OVER (PARTITION BY ticker ORDER BY taken_at DESC) AS rn
  FROM snapshots
)
SELECT a.ticker, b.price AS before, a.price AS now,
       ROUND(a.price - b.price, 4) AS move
FROM ranked a JOIN ranked b ON a.ticker = b.ticker AND b.rn = 2
WHERE a.rn = 1
ORDER BY ABS(a.price - b.price) DESC
LIMIT 10;

Run it from cron every 15 minutes and you have a basic movers report.

Step 6: Hourly price history to CSV

For history you don't need your own snapshots. The candlesticks endpoint needs the series ticker in the path as well as the market ticker. That's easy to miss:

def candles(series_ticker, ticker, hours=48, interval=60):
    end = int(time.time())
    start = end - hours * 3600
    data = get(
        f"/series/{series_ticker}/markets/{ticker}/candlesticks",
        {"start_ts": start, "end_ts": end, "period_interval": interval},
    )
    out = []
    for c in data.get("candlesticks", []):
        price = c.get("price", {})
        out.append({
            "end": datetime.fromtimestamp(c["end_period_ts"], timezone.utc).isoformat(),
            "close": price.get("close_dollars"),  # None when nothing traded
            "yes_bid_close": c.get("yes_bid", {}).get("close_dollars"),
            "yes_ask_close": c.get("yes_ask", {}).get("close_dollars"),
            "volume": c.get("volume_fp"),
            "open_interest": c.get("open_interest_fp"),
        })
    return out

period_interval is in minutes and accepts 1, 60, or 1440. Each candle has price (trades), yes_bid and yes_ask (quotes), plus volume and open interest. In hours with no trades, the price values can come back empty. Don't fill those gaps with zeros, or your chart will show a crash that never happened.

Putting the Kalshi scraper together

if __name__ == "__main__":
    series = sys.argv[1] if len(sys.argv) > 1 else "KXHIGHNY"
    rows = snapshot(series)
    rows.sort(key=lambda r: r["volume_24h"] or 0, reverse=True)
    print(f"{len(rows)} open markets in {series}")
    for r in rows[:5]:
        print(f'{r["ticker"]:<28} bid {r["yes_bid"]}  ask {r["yes_ask"]}  24h vol {r["volume_24h"]}  {r["subtitle"]}')

    if rows:
        top = rows[0]["ticker"]
        bid, ask = best_levels(top)
        print(f"\nOrder book for {top}: best YES bid {bid}, implied YES ask {ask}")
        history = candles(series, top)
        with open(f"{top}.csv", "w", newline="") as f:
            w = csv.DictWriter(f, fieldnames=history[0].keys() if history else ["end"])
            w.writeheader()
            w.writerows(history)
        print(f"wrote {len(history)} hourly candles to {top}.csv")

Real output from a run on the NYC temperature series:

12 open markets in KXHIGHNY
KXHIGHNY-26OCT20-B69.5       bid 0.0000  ask 0.0100  24h vol 63785.26  69° to 70°
KXHIGHNY-26OCT20-B71.5       bid 0.0000  ask 0.0100  24h vol 45825.52  71° to 72°
KXHIGHNY-26OCT20-B65.5       bid 0.0000  ask 0.0100  24h vol 33171.92  65° to 66°
KXHIGHNY-26OCT20-B67.5       bid 0.9900  ask 1.0000  24h vol 27363.99  67° to 68°

Order book for KXHIGHNY-26OCT20-B69.5: best YES bid None, implied YES ask 0.0100
wrote 38 hourly candles to KXHIGHNY-26OCT20-B69.5.csv

One more quirk shows up in that output. Yesterday's markets are still in the open list, priced at 0.01 or 0.99, because trading has stopped but they haven't been settled yet. If you're looking for live opinion, filter on close_time in the future, or you'll be ranking markets that are already decided.

The walls you'll hit

Schema drift. The move from integer cents to _dollars and _fp strings broke many older scripts without any error message. Log a warning when an expected field is missing instead of quietly writing None.
Rate limits without hints. A 429 doesn't tell you how long to wait. Use exponential backoff and a small sleep between pages. If you're polling hundreds of order books, you'll hit the limit faster than you expect.
Volume and pagination. Some series, like sports and crypto, have thousands of markets. Scope by series or event and limit columns to what you need.
Settled vs. closed vs. active. Status values in filters and records don't match one to one. Write down which states you care about and test them.
What isn't in the public API. Trader profiles, the leaderboard, and comments are on the website but not in the public market data endpoints. Scraping those means dealing with the web app itself, which is much more fragile.
Terms. Market data is public, but that doesn't make every use OK. Read Kalshi's terms before you publish or resell anything.

When DIY stops being worth it

For a few series checked every 15 minutes, the script above is all you need, and it's free. It gets tedious when you track a lot of series, want Polymarket and Kalshi in the same format, or need the profile and leaderboard data that the public market endpoints don't have.

At that point I'd use a hosted API. I use ScrapeAtlas* for this. Its /v1/kalshi/markets endpoint takes series, event, tickers, status, closesAfter/closesBefore, cursor, and limit (1 to 200). It leaves out combo markets by default (combos=include brings them back) and returns the same _dollars/_fp field names you saw above, plus a nextCursor:

import os
import requests

def api_markets(series, cursor=None):
    params = {"series": series, "status": "open", "limit": 200}
    if cursor:
        params["cursor"] = cursor
    res = requests.get(
        "https://api.scrapeatlas.com/v1/kalshi/markets",
        params=params,
        headers={"x-api-key": os.environ["SCRAPEATLAS_API_KEY"]},
        timeout=30,
    )
    res.raise_for_status()
    data = res.json()
    # data["status"]: "success" or "partial"; data["coverage"]["warnings"] explains gaps
    return data.get("markets", []), data.get("nextCursor")

Because the field names match, to_row() works unchanged. There are also /v1/kalshi/price-history, /v1/kalshi/orderbook, /v1/kalshi/trades, /v1/kalshi/leaderboard, and /v1/kalshi/profile endpoints if you need history, depth, or trader data.

Wrapping up

Use the public API, not HTML. Kalshi market data is a GET request away, with no key.
Parse _dollars and _fp strings as Decimal, and don't trust tutorials that use integer-cent fields.
The order book only has bids. YES ask = 1 − best NO bid.
Scope by series or event, exclude combos, and filter out markets that are closed but not yet settled.
Snapshot on a schedule if you care about movement. Use candlesticks if you only need history.

If something in the API changes and the script breaks, leave a comment and I'll update the post.

*I build it.

Substack Scraper in Node.js: Post Archives, Likes, Comments, and the Walls You'll Hit (2026)

Viktor — Mon, 19 Oct 2026 04:09:05 +0000

Substack has quietly become one of the more useful datasets on the web. If you do competitor research for a newsletter, track a niche, or want to know which topics actually get people to hit the heart button, the numbers are sitting right there on every publication.

In this post I'll build a small Node.js script that:

Pages through any publication's post archive
Exports title, date, author, paywall status, word count, likes, comments, and restacks to CSV
Prints a quick summary (paywalled share, top posts by likes)
Pulls the comment thread for a single post

No headless browser and no dependencies: Node 18+ has fetch built in. Everything below was tested against live publications in October 2026, and I'll point out the quirks I ran into along the way.

Before you start: stick to public data, keep request rates low, and read Substack's terms. The endpoints below are the ones Substack's own web app uses. They're not a documented public API, so they can change without notice.

Your options, briefly

RSS. Every publication has /feed. Great for "what's new", but you only get the most recent posts and no engagement numbers.
HTML pages. Works, but you'd be parsing markup for numbers that are already available as JSON.
The archive JSON (this post). The archive page loads its data from /api/v1/archive. It returns clean JSON with engagement counts, and it paginates.

Step 1: Find the archive endpoint

Open any publication's /archive page, scroll, and watch the network tab. You'll see requests like this:

https://www.lennysnewsletter.com/api/v1/archive?sort=new&offset=0&limit=12

Each item is a post object with a lot of fields. The ones I actually use:

Field	What it is
`title`, `slug`, `canonical_url`	The basics
`post_date`	ISO timestamp
`audience`	`everyone`, `only_paid`, or `founding`
`type`	`newsletter`, `podcast`, `thread`
`wordcount`	Full length, even for paywalled posts
`reaction_count`, `comment_count`, `restacks`	Engagement
`publishedBylines`	Authors (guest posts have more than one)

sort=top also works and gives you the most-liked posts first, which is handy if you only care about what performed.

Step 2: Page through it politely

Here's the whole scraper. Save it as scrape.mjs:

// usage: node scrape.mjs https://www.lennysnewsletter.com [maxPosts]
import { writeFile } from "node:fs/promises";

const base = new URL(process.argv[2] ?? "https://www.lennysnewsletter.com").origin;
const maxPosts = Number(process.argv[3] ?? 60);
const PAGE = 12; // larger values return odd page sizes, see below
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function getJSON(url, tries = 4) {
  for (let i = 0; i < tries; i++) {
    const res = await fetch(url, {
      headers: { "user-agent": "substack-research-script/1.0 (contact: you@example.com)" },
      redirect: "follow",
    });
    if (res.ok) return res.json();
    if (res.status === 429 || res.status >= 500) {
      const wait = Number(res.headers.get("retry-after")) * 1000 || 2 ** i * 2000;
      console.warn(`${res.status} on ${url}, waiting ${wait}ms`);
      await sleep(wait);
      continue;
    }
    throw new Error(`${res.status} ${res.statusText} for ${url}`);
  }
  throw new Error(`gave up on ${url}`);
}

async function fetchArchive(sort = "new") {
  const posts = [];
  for (let offset = 0; posts.length < maxPosts; offset += PAGE) {
    const url = `${base}/api/v1/archive?sort=${sort}&offset=${offset}&limit=${PAGE}`;
    const page = await getJSON(url);
    if (!Array.isArray(page) || page.length === 0) break;
    posts.push(...page);
    await sleep(1000); // one request per second
  }
  return posts.slice(0, maxPosts);
}

The retry loop backs off on 429 and 5xx, and respects Retry-After if Substack sends one. I didn't hit a rate limit at one request per second over a few dozen pages, but I wouldn't run this in parallel across hundreds of publications without a lot more care.

Step 3: Flatten and export to CSV

const toRow = (p) => ({
  id: p.id,
  date: p.post_date?.slice(0, 10),
  title: p.title,
  type: p.type,
  audience: p.audience,
  words: p.wordcount,
  likes: p.reaction_count,
  comments: p.comment_count,
  restacks: p.restacks,
  author: p.publishedBylines?.map((b) => b.name).join(" & "),
  url: p.canonical_url,
});

const csvCell = (v) => {
  const s = v == null ? "" : String(v);
  return /[",\n]/.test(s) ? `"${s.replaceAll('"', '""')}"` : s;
};

const posts = (await fetchArchive()).map(toRow);
const cols = Object.keys(posts[0]);
const csv = [cols.join(","), ...posts.map((r) => cols.map((c) => csvCell(r[c])).join(","))].join("\n");
await writeFile("posts.csv", csv + "\n");

Titles contain commas and quotes all the time, so the csvCell escaping isn't optional. Spreadsheet apps will happily shift every column otherwise.

Step 4: A quick summary

const paid = posts.filter((p) => p.audience !== "everyone");
const avg = (xs) => Math.round(xs.reduce((a, b) => a + b, 0) / (xs.length || 1));

console.log(`paywalled: ${paid.length} (${Math.round((100 * paid.length) / posts.length)}%)`);
console.log(`avg likes free: ${avg(posts.filter((p) => p.audience === "everyone").map((p) => p.likes))}, paid: ${avg(paid.map((p) => p.likes))}`);
for (const p of [...posts].sort((a, b) => b.likes - a.likes).slice(0, 5))
  console.log(`  ${p.likes} likes, ${p.comments} comments  ${p.date}  ${p.title}`);

Running it against two very different publications shows why this is interesting:

$ node scrape.mjs https://www.lennysnewsletter.com 36
paywalled: 29 (81%)
avg likes free: 335, paid: 496

$ node scrape.mjs https://astralcodexten.substack.com 24
paywalled: 2 (8%)
...

One is mostly paywalled with comparatively few comments per post; the other is almost entirely free, and its top posts had hundreds of comments each. Same platform, completely different playbooks, and you can see it from 30 lines of output.

Step 5: Comments for a single post

Comments live on a separate endpoint keyed by the post's numeric id:

async function fetchComments(postId) {
  const url = `${base}/api/v1/post/${postId}/comments?all_comments=true&sort=best_first`;
  const { comments = [] } = await getJSON(url);
  const flat = [];
  const walk = (list, depth = 0) => {
    for (const c of list) {
      flat.push({ depth, name: c.name, likes: c.reaction_count, date: c.date, body: c.body });
      walk(c.children ?? [], depth + 1);
    }
  };
  walk(comments);
  return flat;
}

Replies are nested under children, so a small recursive walk flattens them into rows with a depth column. Threads go deep: one popular post in my test nested 28 levels. Deleted comments stay in the tree with deleted: true and a null body (so their replies keep their place), and the number you collect won't exactly match the post's comment_count. Filter deleted rows before any text analysis.

Where this approach hits a wall

The limit parameter is weird. Up to 20 gave me exactly what I asked for. 25 to 49 all returned 23 posts, and limit=100 returned a single post. Keep pages small (I use 12, which is what the web app uses) and paginate with offset.
Paywalled posts are previews. For an only_paid post, /api/v1/posts/<slug> returns a body_html that stops at the paywall. On one 4,153-word post I got about 2,300 words. wordcount still reports the full length, which is useful on its own.
Custom domains redirect. astralcodexten.substack.com answers with a 301 to www.astralcodexten.com. fetch follows it, but if you dedupe publications by hostname, normalize after the redirect.
Cross-publication search is a different beast. Everything above works per publication. Finding publications by keyword means hitting substack.com's own internal search endpoints, which are even less stable.
It's undocumented. If a field disappears, your CSV quietly fills with blanks. Check that reaction_count is a number on the first page and fail loudly if it isn't.

When you'd rather not maintain it

If you're tracking a lot of publications, or you need search, notes, and writer profiles too, a scraping API can take over the fetching. I use ScrapeAtlas* for this. Its /v1/substack/posts endpoint takes the publication url plus optional limit (1 to 100), offset, and sort (new or top), and returns a posts array with the same field names you saw above, plus a nextOffset for the next page:

async function apiPosts(pubUrl, offset = 0) {
  const q = new URLSearchParams({ url: pubUrl, limit: "50", offset: String(offset), sort: "new" });
  const res = await fetch(`https://api.scrapeatlas.com/v1/substack/posts?${q}`, {
    headers: { "x-api-key": process.env.SCRAPEATLAS_API_KEY },
  });
  const data = await res.json();
  // data.status: "success" or "partial"; data.coverage.warnings explains gaps
  return { posts: data.posts ?? [], next: data.nextOffset };
}

Because the field names match, toRow and the CSV code work unchanged. There are also /v1/substack/publication, /v1/substack/post/comments, and /v1/substack/search endpoints if you need publication details, comment threads, or keyword search.

Wrapping up

A few publications, checked weekly: the archive endpoint and the script above are all you need, and it costs nothing.
Keep pages small and paginate with offset; don't trust large limit values.
Treat paywalled bodies as previews, and lean on wordcount and engagement fields instead.
Many publications, search, or anything you need to rely on: that's where maintaining it yourself stops being fun.

If the archive endpoint starts returning something unexpected, leave a comment and I'll update the post.

*I build it.

Pinterest Scraper in Go: Board RSS, Pin Pages, and a Saves Tracker in SQLite (2026)

Viktor — Fri, 16 Oct 2026 04:08:32 +0000

Pinterest is quietly one of the best datasets for anyone working in e-commerce, home decor, food, or fashion. Saves are a pretty honest signal of what people actually want to buy or make, and they show up weeks before a trend hits other platforms.

Getting that data out is a different story. In this post I'll build a small Go tool that:

Reads the latest pins from any public board or profile
Opens each pin and pulls out title, outbound link, domain, creation date, repins, and saves
Stores a snapshot on every run in SQLite, so you can see which pins are picking up saves

Everything here was tested against live Pinterest pages in October 2026. I'll also point out the parts that are flaky, because a few of them surprised me.

Before you start: stick to public data, keep request rates low, and read Pinterest's Terms of Service. If you only need data for accounts you own or manage, the official API v5 is the cleaner route (more on that below).

Your options, briefly

Official API v5. Needs a registered app, OAuth, and approval; new apps start on limited trial access. It's built around accounts that authorize your app, so it isn't a way to read arbitrary public boards.
Internal /resource/... endpoints. The web app's own JSON calls. They work until they don't: parameters change without notice and they're much more likely to get you rate limited.
RSS + pin pages (this post). Every public profile and board still has an RSS feed, and every pin page ships its data as embedded JSON. No login, no headless browser.

Step 1: The RSS feed gives you pin URLs

This part is pleasantly boring. Two URL patterns:

https://www.pinterest.com/<username>/feed.rss         # a profile's latest pins
https://www.pinterest.com/<username>/<board-slug>.rss  # one board

Each feed returns the 25 most recent pins with a title, link, thumbnail, and pubDate. That limit matters, and I'll come back to it. Parsing it in Go needs nothing beyond the standard library:

type rssFeed struct {
    Channel struct {
        Title string    `xml:"title"`
        Items []rssItem `xml:"item"`
    } `xml:"channel"`
}

type rssItem struct {
    Title   string `xml:"title"`
    Link    string `xml:"link"`
    PubDate string `xml:"pubDate"`
}

Step 2: A polite fetch helper

Pin pages are big (more than 1 MB of HTML each), and Pinterest does return 429 if you hammer it. Retries with exponential backoff and jitter keep things civil:

// fetch does a polite GET with retries and exponential backoff.
func fetch(url string) ([]byte, error) {
    var lastErr error
    for attempt := 0; attempt < 4; attempt++ {
        if attempt > 0 {
            wait := time.Duration(1<<attempt) * time.Second
            time.Sleep(wait + time.Duration(rand.Intn(1000))*time.Millisecond)
        }
        req, _ := http.NewRequest("GET", url, nil)
        req.Header.Set("User-Agent", userAgent)
        req.Header.Set("Accept-Language", "en-US,en;q=0.9")
        resp, err := client.Do(req)
        if err != nil {
            lastErr = err
            continue
        }
        body, err := io.ReadAll(resp.Body)
        resp.Body.Close()
        if err != nil {
            lastErr = err
            continue
        }
        switch {
        case resp.StatusCode == 200:
            return body, nil
        case resp.StatusCode == 429 || resp.StatusCode >= 500:
            lastErr = fmt.Errorf("%s: HTTP %d", url, resp.StatusCode)
            continue
        default:
            return nil, fmt.Errorf("%s: HTTP %d", url, resp.StatusCode)
        }
    }
    return nil, lastErr
}

Setting Accept-Language matters more than you'd think. Pinterest localizes heavily, and without it you may get titles and SEO descriptions in whatever language it guesses from your IP.

func boardFeed(user, board string) ([]rssItem, error) {
    url := fmt.Sprintf("https://www.pinterest.com/%s/%s.rss", user, board)
    if board == "" {
        url = fmt.Sprintf("https://www.pinterest.com/%s/feed.rss", user)
    }
    body, err := fetch(url)
    if err != nil {
        return nil, err
    }
    var feed rssFeed
    if err := xml.Unmarshal(body, &feed); err != nil {
        return nil, fmt.Errorf("parse rss: %w", err)
    }
    return feed.Channel.Items, nil
}

Step 3: Where the pin data actually lives

My first attempt parsed the application/ld+json block on the pin page. It has a nice SocialMediaPosting object with a "Saves" counter. It worked on the first pin I tried, and then failed on 25 out of 25 pins in my first real run.

Fetching the same pin six times in a row explained it: the JSON-LD block was there once. Pinterest streams the page, and which blocks make it into the HTML varies from request to request.

What is consistently there is one or more <script data-relay-completed-request="true"> tags. Each one calls window.__PWS_RELAY_REGISTER_COMPLETED_REQUEST__(key, json), and the JSON contains a v3GetPinQueryv2.data object with the pin. A single pin is often split across several of these blocks, and some fields (especially aggregatedPinData.aggregatedStats.saves) only appear in one of them. So the trick is to parse all of them and merge:

type Pin struct {
    ID        string
    URL       string
    Title     string
    Link      string // outbound destination, if any
    Domain    string
    Image     string
    Author    string
    Board     string
    Published string
    Repins    int
    Saves     *int // nil when Pinterest didn't include it in this response
}

var relayRe = regexp.MustCompile(`(?s)<script data-relay-completed-request="true"[^>]*>window\.__PWS_RELAY_REGISTER_COMPLETED_REQUEST__\("[^"]*",(.*?)\);?</script>`)

type pinData struct {
    EntityID   string `json:"entityId"`
    Title      string `json:"title"`
    SeoTitle   string `json:"seoTitle"`
    Link       string `json:"link"`
    Domain     string `json:"domain"`
    CreatedAt  string `json:"createdAt"`
    RepinCount int    `json:"repinCount"`
    ImageLarge string `json:"imageLargeUrl"`
    Pinner     *struct {
        Username string `json:"username"`
    } `json:"pinner"`
    Board *struct {
        URL string `json:"url"`
    } `json:"board"`
    AggregatedPinData *struct {
        AggregatedStats struct {
            Saves *int `json:"saves"`
        } `json:"aggregatedStats"`
    } `json:"aggregatedPinData"`
}

type relayPayload struct {
    Data struct {
        Pin *struct {
            Data *pinData `json:"data"`
        } `json:"v3GetPinQueryv2"`
    } `json:"data"`
}

var errNoPinData = errors.New("no pin data in page")

func first(vals ...string) string {
    for _, v := range vals {
        if v != "" {
            return v
        }
    }
    return ""
}

// scrapePin merges every relay block on the page: Pinterest splits one pin
// across several blocks, and some fields (like saves) only show up in one.
func scrapePin(url string) (*Pin, error) {
    body, err := fetch(url)
    if err != nil {
        return nil, err
    }
    p := &Pin{URL: url}
    found := false
    for _, m := range relayRe.FindAllSubmatch(body, -1) {
        var r relayPayload
        if json.Unmarshal(m[1], &r) != nil || r.Data.Pin == nil || r.Data.Pin.Data == nil {
            continue
        }
        d := r.Data.Pin.Data
        found = true
        p.ID = first(p.ID, d.EntityID)
        p.Title = first(p.Title, d.Title, d.SeoTitle)
        p.Link = first(p.Link, d.Link)
        p.Domain = first(p.Domain, d.Domain)
        p.Published = first(p.Published, d.CreatedAt)
        p.Image = first(p.Image, d.ImageLarge)
        if d.RepinCount > p.Repins {
            p.Repins = d.RepinCount
        }
        if d.Pinner != nil {
            p.Author = first(p.Author, d.Pinner.Username)
        }
        if d.Board != nil {
            p.Board = first(p.Board, d.Board.URL)
        }
        if d.AggregatedPinData != nil && d.AggregatedPinData.AggregatedStats.Saves != nil {
            p.Saves = d.AggregatedPinData.AggregatedStats.Saves
        }
    }
    if !found {
        return nil, errNoPinData
    }
    return p, nil
}

Two field notes:

repinCount is not the save count. On one pin I saw repinCount: 9 next to saves: 417. The aggregated saves number is the one that matches what users see.
Saves are often missing. In a 25-pin run, saves came back on only about a third of the pins. That's why Saves is a pointer: I'd rather store NULL than a fake zero that wrecks your growth charts.

Step 4: Store snapshots, not rows

The value is in the change over time, so pins go in once and every run appends a snapshot:

const schema = `
CREATE TABLE IF NOT EXISTS pins (
  id TEXT PRIMARY KEY, url TEXT, title TEXT, link TEXT, domain TEXT,
  image TEXT, author TEXT, board TEXT, published TEXT, first_seen TEXT
);
CREATE TABLE IF NOT EXISTS snapshots (
  pin_id TEXT, taken_at TEXT, repins INTEGER, saves INTEGER,
  PRIMARY KEY (pin_id, taken_at)
);`

func save(db *sql.DB, p *Pin) error {
    now := time.Now().UTC().Format(time.RFC3339)
    _, err := db.Exec(`INSERT INTO pins (id,url,title,link,domain,image,author,board,published,first_seen)
        VALUES (?,?,?,?,?,?,?,?,?,?) ON CONFLICT(id) DO NOTHING`,
        p.ID, p.URL, p.Title, p.Link, p.Domain, p.Image, p.Author, p.Board, p.Published, now)
    if err != nil {
        return err
    }
    _, err = db.Exec(`INSERT INTO snapshots (pin_id,taken_at,repins,saves) VALUES (?,?,?,?)`,
        p.ID, now, p.Repins, p.Saves) // a nil *int is stored as NULL
    return err
}

And the main loop, with a random 2 to 4 second pause between pins:

func main() {
    if len(os.Args) < 2 {
        log.Fatal("usage: pinscrape <user> [board-slug]")
    }
    user, board := os.Args[1], ""
    if len(os.Args) > 2 {
        board = os.Args[2]
    }

    db, err := sql.Open("sqlite", "pins.db")
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()
    if _, err := db.Exec(schema); err != nil {
        log.Fatal(err)
    }

    items, err := boardFeed(user, board)
    if err != nil {
        log.Fatal(err)
    }
    log.Printf("feed returned %d pins", len(items))

    for _, it := range items {
        p, err := scrapePin(it.Link)
        if err != nil {
            log.Printf("skip %s: %v", it.Link, err)
            continue
        }
        if err := save(db, p); err != nil {
            log.Printf("db: %v", err)
        }
        saves := "?"
        if p.Saves != nil {
            saves = fmt.Sprint(*p.Saves)
        }
        fmt.Printf("%-20s saves=%-6s repins=%-4d %s\n", p.ID, saves, p.Repins, p.Title)
        time.Sleep(time.Duration(2000+rand.Intn(2000)) * time.Millisecond)
    }
}

Install the SQLite driver (pure Go, no cgo) and run it against a board:

go mod init pinscrape
go get modernc.org/sqlite
go build -o pinscrape .
./pinscrape pinterest                 # profile feed
./pinscrape <username> <board-slug>   # a single board

Real output from Pinterest's own profile feed (trimmed):

feed returned 25 pins
424605071146015538   saves=418    repins=2    Burgundy food table
424605071146015535   saves=?      repins=9    The Christmas Centerpiece That Makes a Table Feel Instantly Magical
skip https://www.pinterest.com/pin/424605071146015533/: no pin data in page
424605071146015511   saves=6462   repins=5    Aesthetic gift wrapping idea
424605071146015505   saves=3063   repins=6    Christmas champagne

Run it from cron once a day, then ask SQLite which pins are gaining traction:

SELECT p.title,
       MIN(s.saves) AS first_saves,
       MAX(s.saves) AS latest_saves,
       MAX(s.saves) - MIN(s.saves) AS gained
FROM snapshots s
JOIN pins p ON p.id = s.pin_id
WHERE s.saves IS NOT NULL
GROUP BY p.id
ORDER BY gained DESC
LIMIT 20;

MIN and MAX skip NULLs, which is exactly what you want when saves only show up on some runs.

Where this approach hits a wall

25 pins per feed, no pagination. RSS only shows the newest 25. To build history for a big board you have to start tracking early and keep running; you can't backfill old pins this way.
No keyword search. RSS covers profiles and boards. Searching Pinterest for "christmas tablescape" means the search page, which renders client-side and leans on the internal endpoints.
Flaky fields. Around 1 in 8 pin pages in my test had no pin data at all, and saves were missing more often than not. Retry missing pins on the next run instead of hammering them immediately.
Bandwidth. 25 pins is roughly 30 MB of HTML. Fine for a few boards, painful for thousands.
Markup drift. v3GetPinQueryv2 is an internal query name. When it changes, your scraper returns zero pins. Alert on that instead of silently storing nothing.

When you'd rather not maintain it

If you need keyword search, deeper pagination, or more volume than a cron job can handle politely, a scraping API takes over the fetching. I use ScrapeAtlas* for this. Its /v1/pinterest/search endpoint takes a query and an optional cursor, and /v1/pinterest/board takes a board url. Both return a pins array plus a cursor for the next page:

type apiResp struct {
    Status string  `json:"status"` // success, partial, failed, challenged
    Cursor *string `json:"cursor"`
    Pins   []struct {
        ID        string `json:"id"`
        Title     string `json:"title"`
        URL       string `json:"url"`
        CreatedAt string `json:"created_at"`
    } `json:"pins"`
}

func searchPins(query, cursor string) (*apiResp, error) {
    q := url.Values{"query": {query}}
    if cursor != "" {
        q.Set("cursor", cursor)
    }
    req, _ := http.NewRequest("GET", "https://api.scrapeatlas.com/v1/pinterest/search?"+q.Encode(), nil)
    req.Header.Set("x-api-key", os.Getenv("SCRAPEATLAS_API_KEY"))
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    var r apiResp
    return &r, json.NewDecoder(resp.Body).Decode(&r)
}

Check status either way: partial means you got usable pins but some fields were missing (the response's coverage.warnings tells you which, for example missing_pin_counts). The same pins and snapshots tables work for both sources.

Wrapping up

A handful of boards, tracked daily: RSS plus pin pages, as above, works well and costs nothing.
Make missing data explicit: store NULL for saves you didn't get and merge every relay block on the page.
Search or scale: that's where the DIY version stops being fun, and an API or a lot of proxy work comes in.

If your run starts returning "no pin data" on every pin, Pinterest has probably renamed the query. Leave a comment and I'll update the parser.

*I build it.

Vinted Scraper in Go: Search, Prices, and the Walls You'll Hit (2026)

Viktor — Wed, 14 Oct 2026 08:03:42 +0000

Vinted is one of the biggest second-hand fashion marketplaces in Europe, and its public catalog is useful for price tracking, brand research, and spotting what actually sells in a niche. There is no official public product API for third-party tools, so most people end up scraping the HTML or reverse-engineering the same JSON the website uses.

This guide builds a small Go scraper that:

Searches Vinted for a keyword on a chosen country domain
Pulls out item id, title, brand, size, price, and seller
Pages through results politely
Stores snapshots in SQLite so you can track prices over time

I'll also be honest about where the DIY approach breaks.

Legal note: Only scrape publicly visible data, keep your request rate low, and read Vinted's terms for the country you target. If you need this data for a commercial product at scale, assume you'll need proxies and ongoing maintenance — or an API that already handles them.

How Vinted search is structured

A search URL looks like this:

https://www.vinted.fr/catalog?search_text=nike+zoom&page=1

Country matters. vinted.fr, vinted.de, vinted.co.uk, vinted.pl, and vinted.com are separate marketplaces with different inventories and currencies. Don't assume a listing on .fr exists on .com.

The page is heavily client-rendered. The HTML shell is mostly empty; the useful data arrives as JSON from Vinted's internal catalog API. You can either:

Parse that internal JSON (fragile: paths and auth headers change)
Or render the page in a browser and scrape the DOM (slow, still fingerprintable)

For a small DIY tool, listening to the catalog JSON from a headless browser is more reliable than hand-rolling their signed requests. For a pure-Go path without a browser, people often replay the catalog endpoint with careful headers — until it breaks. We'll start with the pure-Go approach against the public catalog URL and fall back honestly when it fails.

Setup

mkdir vinted-scraper && cd vinted-scraper
go mod init vinted-scraper
# no third-party deps for the basic version

Step 1: Fetch a search page without looking like a bot on day one

At minimum, send a real browser User-Agent, an Accept-Language that matches the domain, and reuse one http.Client so cookies persist:

package main

import (
    "fmt"
    "io"
    "math/rand"
    "net/http"
    "net/url"
    "strings"
    "time"
)

var client = &http.Client{Timeout: 25 * time.Second}

func headers(domain string) http.Header {
    h := make(http.Header)
    h.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36")
    h.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
    h.Set("Accept-Language", langFor(domain))
    h.Set("Connection", "keep-alive")
    return h
}

func langFor(domain string) string {
    switch domain {
    case "fr":
        return "fr-FR,fr;q=0.9,en;q=0.8"
    case "de":
        return "de-DE,de;q=0.9,en;q=0.8"
    case "co.uk":
        return "en-GB,en;q=0.9"
    case "pl":
        return "pl-PL,pl;q=0.9,en;q=0.8"
    default:
        return "en-US,en;q=0.9"
    }
}

func fetch(rawURL string, domain string) ([]byte, error) {
    var last error
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequest(http.MethodGet, rawURL, nil)
        if err != nil {
            return nil, err
        }
        req.Header = headers(domain)
        resp, err := client.Do(req)
        if err != nil {
            last = err
        } else {
            body, readErr := io.ReadAll(resp.Body)
            resp.Body.Close()
            if readErr != nil {
                last = readErr
            } else if resp.StatusCode == 403 || resp.StatusCode == 429 || resp.StatusCode == 503 {
                last = fmt.Errorf("blocked: HTTP %d", resp.StatusCode)
            } else if resp.StatusCode >= 400 {
                last = fmt.Errorf("HTTP %d", resp.StatusCode)
            } else if looksLikeChallenge(body) {
                last = fmt.Errorf("challenge / captcha page")
            } else {
                return body, nil
            }
        }
        wait := time.Duration(5*(1<<attempt))*time.Second + time.Duration(rand.Intn(3000))*time.Millisecond
        time.Sleep(wait)
    }
    return nil, last
}

func looksLikeChallenge(body []byte) bool {
    s := strings.ToLower(string(body))
    return strings.Contains(s, "captcha") || strings.Contains(s, "cf-challenge") || strings.Contains(s, "access denied")
}

A 200 with a Cloudflare challenge page is still a failure. Check the body, not only the status code.

Step 2: Prefer the JSON the site already loads

When you open a catalog page in DevTools, you'll see requests to something like /api/v2/catalog/items. That payload is cleaner than the DOM. The hard part is that those calls often need cookies and anti-bot tokens minted by the first HTML load.

A practical DIY pattern in Go:

Fetch the HTML once to get cookies into your client jar (use net/http/cookiejar).
Replay the catalog API URL with the same jar and headers.
Parse the JSON into your own structs.

type CatalogResponse struct {
    Items []CatalogItem `json:"items"`
    Pagination struct {
        CurrentPage int  `json:"current_page"`
        TotalPages  int  `json:"total_pages"`
    } `json:"pagination"`
}

type CatalogItem struct {
    ID    int64  `json:"id"`
    Title string `json:"title"`
    URL   string `json:"url"`
    Price string `json:"price"` // sometimes a string; sometimes nested — normalize below
    Brand struct {
        Title string `json:"title"`
    } `json:"brand"`
    SizeTitle string `json:"size_title"`
    User      struct {
        ID    int64  `json:"id"`
        Login string `json:"login"`
    } `json:"user"`
    Photo struct {
        URL string `json:"url"`
    } `json:"photo"`
}

type Item struct {
    ID     int64
    Title  string
    Brand  string
    Size   string
    Price  float64
    URL    string
    Seller string
}

Field names and nesting change. Keep a saved fixture from a successful response and a unit test that fails loudly when Vinted renames size_title again.

Price parsing is locale-dependent ("12,50 €" vs "£12.50"). Don't ParseFloat the raw string — strip currency symbols and normalize commas first.

Step 3: Paginate politely across country domains

func search(domain, query string, maxPages int) ([]Item, error) {
    var out []Item
    seen := map[int64]bool{}
    for page := 1; page <= maxPages; page++ {
        u := url.URL{
            Scheme: "https",
            Host:   "www.vinted." + domain,
            Path:   "/catalog",
        }
        q := u.Query()
        q.Set("search_text", query)
        q.Set("page", fmt.Sprintf("%d", page))
        u.RawQuery = q.Encode()

        body, err := fetch(u.String(), domain)
        if err != nil {
            return out, fmt.Errorf("page %d: %w", page, err)
        }
        items, err := parseCatalog(body, domain) // your JSON or HTML parser
        if err != nil {
            return out, err
        }
        if len(items) == 0 {
            break
        }
        for _, it := range items {
            if !seen[it.ID] {
                seen[it.ID] = true
                out = append(out, it)
            }
        }
        fmt.Printf("page %d: %d items\n", page, len(items))
        time.Sleep(time.Duration(3+rand.Intn(5)) * time.Second)
    }
    return out, nil
}

Random delays of a few seconds matter more than clever headers. Burst traffic from one datacenter IP is the fastest way onto a challenge wall.

Step 4: Store snapshots for price history

A single scrape is rarely useful. SQLite is enough:

import (
    "database/sql"
    "time"
    _ "modernc.org/sqlite" // or github.com/mattn/go-sqlite3
)

func save(db *sql.DB, keyword string, items []Item) error {
    _, err := db.Exec(`CREATE TABLE IF NOT EXISTS snapshots (
        id INTEGER, keyword TEXT, title TEXT, brand TEXT, size TEXT,
        price REAL, url TEXT, seller TEXT, scraped_at TEXT
    )`)
    if err != nil {
        return err
    }
    now := time.Now().UTC().Format(time.RFC3339)
    tx, err := db.Begin()
    if err != nil {
        return err
    }
    stmt, err := tx.Prepare(`INSERT INTO snapshots VALUES (?,?,?,?,?,?,?,?,?)`)
    if err != nil {
        return err
    }
    defer stmt.Close()
    for _, it := range items {
        if _, err := stmt.Exec(it.ID, keyword, it.Title, it.Brand, it.Size, it.Price, it.URL, it.Seller, now); err != nil {
            tx.Rollback()
            return err
        }
    }
    return tx.Commit()
}

Run it on a cron and you can ask:

SELECT id, date(scraped_at) AS day, MIN(price) AS price
FROM snapshots
WHERE keyword = 'nike zoom' AND brand = 'Nike'
GROUP BY id, day
ORDER BY id, day;

Where DIY falls apart

The script above is fine for a few pages a day from a home connection. Past that you'll hit:

Cloudflare / bot challenges at volume. From AWS, GCP, or a cheap VPS you often get blocked on the first request. Residential proxies cost real money and still need care.
Country domains. Inventory, currency, and anti-bot behavior differ per host. A scraper tuned on vinted.fr will mis-parse prices on vinted.co.uk.
Internal API drift. Catalog JSON field names and required cookies change. Budget time for fixtures and alerts when a page returns zero items.
Sold / reserved items. Listings disappear or flip to reserved quickly in popular niches. If you only store the latest scrape, you lose the price history that made the project worth doing.
Headless browsers aren't free wins. Playwright helps with client-rendered pages but is slower, heavier, and still fingerprinted.

Option: use an API for the hard part

If you'd rather not maintain proxies and selectors, a scraping API can return the same catalog as JSON. For example, ScrapeAtlas* has a Vinted search endpoint that takes the query, domain, and page:

func searchAPI(apiKey, query, domain string, page int) ([]Item, error) {
    u, _ := url.Parse("https://api.scrapeatlas.com/v1/vinted/search")
    q := u.Query()
    q.Set("query", query)
    q.Set("domain", domain) // fr, de, co.uk, com, ...
    q.Set("page", fmt.Sprintf("%d", page))
    q.Set("per_page", "24")
    u.RawQuery = q.Encode()

    req, _ := http.NewRequest(http.MethodGet, u.String(), nil)
    req.Header.Set("x-api-key", apiKey)
    resp, err := client.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    // Decode their JSON into your Item slice; check status for
    // "failed" / "challenged" / "partial" the same way you would
    // for any upstream that can be rate-limited.
    return decodeScrapeAtlas(resp.Body)
}

There are also endpoints for a single item by id/URL, a member profile, and a member's wardrobe. domain selects which marketplace the data was read from. That output can feed the same save() function, so you can swap fetchers without changing storage.

Wrapping up

For small, occasional scrapes: Go net/http + careful headers + slow delays can work from a residential IP.
For daily tracking of a few keywords: add a cookie jar, response fixtures, zero-result alerts, and per-domain locale parsing.
For anything at scale or from cloud servers: plan for residential proxies and ongoing breakage, or an API that absorbs that cost.

Whichever route you take, store snapshots instead of overwriting rows. On Vinted the listing that was €40 yesterday may be gone tomorrow — the history is the product.

Questions or a selector that broke? Leave a comment and I'll update the guide.

*I build it.

Amazon Scraper in Python: An Honest Developer Guide (2026)

Viktor — Mon, 12 Oct 2026 10:12:49 +0000

Amazon is probably the most-scraped e-commerce site there is, and it's also one of the best defended. People scrape it to track prices, monitor competitors, research product niches, or check what's actually ranking for a keyword.

This guide covers what works in 2026, what breaks, and why. We'll build a small Python scraper that:

Searches Amazon for a keyword
Pulls out ASIN, title, price, rating, and review count
Pages through the results
Stores snapshots in SQLite so you can track prices over time

I'll also be upfront about where the DIY approach stops working.

Legal note: Only scrape publicly visible data, keep your request rate low, and read Amazon's Conditions of Use. If you need product data at scale for a commercial product, look at the official Product Advertising API first (it requires an Associates account with qualifying sales).

How Amazon search pages are structured

A search URL looks like this:

https://www.amazon.com/s?k=wireless+earbuds&page=2

k is the keyword
page is the 1-based page number (Amazon usually caps search at around 7–20 pages)

Every organic result is a div with data-component-type="s-search-result" and a data-asin attribute. The ASIN (Amazon Standard Identification Number) is the stable ID you want: https://www.amazon.com/dp/<ASIN> always resolves to the product.

Inside each result card:

Field	Selector (2026, subject to change)
Title	`h2 span`
Price	`span.a-price > span.a-offscreen` (first one is the current price)
List price	`span.a-price.a-text-price > span.a-offscreen`
Rating	`span.a-icon-alt` (text like "4.4 out of 5 stars")
Review count	`span.s-underline-text` or `a[href*="customerReviews"] span`

Amazon changes class names regularly and A/B tests layouts, so write your parser defensively: every field may be missing.

Setup

pip install requests beautifulsoup4 lxml

Step 1: Fetch a search page without getting blocked immediately

The bare requests.get() with the default python-requests/2.x user agent usually gets a 503 or a CAPTCHA page on the first try. At minimum, send realistic browser headers and reuse a session so cookies persist:

import random
import time
import requests

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6) AppleWebKit/605.1.15 "
    "(KHTML, like Gecko) Version/17.6 Safari/605.1.15",
]

session = requests.Session()

def headers():
    return {
        "User-Agent": random.choice(USER_AGENTS),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
        "Connection": "keep-alive",
    }

class Blocked(Exception):
    pass

def fetch(url, params=None, retries=3):
    for attempt in range(retries):
        resp = session.get(url, params=params, headers=headers(), timeout=20)
        blocked = (
            resp.status_code in (503, 429)
            or "validateCaptcha" in resp.text
            or "Robot Check" in resp.text
        )
        if not blocked and resp.ok:
            return resp.text
        wait = 2 ** attempt * 5 + random.uniform(0, 3)
        print(f"Blocked or error ({resp.status_code}), sleeping {wait:.0f}s")
        time.sleep(wait)
    raise Blocked(f"Gave up on {resp.url}")

Note that a CAPTCHA page often comes back with HTTP 200, so checking only the status code isn't enough. Look for validateCaptcha in the body.

Step 2: Parse the results

import re
from bs4 import BeautifulSoup

def to_float(text):
    if not text:
        return None
    m = re.search(r"[\d,]+(?:\.\d+)?", text)
    return float(m.group().replace(",", "")) if m else None

def text_of(el):
    return el.get_text(strip=True) if el else None

def parse_search(html):
    soup = BeautifulSoup(html, "lxml")
    products = []
    for card in soup.select('div[data-component-type="s-search-result"]'):
        asin = card.get("data-asin")
        if not asin:
            continue
        # Skip sponsored placements if you only want organic results
        sponsored = card.select_one('span.puis-label-popover-default, span:-soup-contains("Sponsored")')

        rating_text = text_of(card.select_one("span.a-icon-alt"))  # "4.4 out of 5 stars"
        products.append({
            "asin": asin,
            "title": text_of(card.select_one("h2 span")),
            "price": to_float(text_of(card.select_one("span.a-price > span.a-offscreen"))),
            "list_price": to_float(text_of(card.select_one("span.a-price.a-text-price > span.a-offscreen"))),
            "rating": to_float(rating_text.split(" ")[0]) if rating_text else None,
            "reviews": to_float(text_of(card.select_one("span.s-underline-text"))),
            "sponsored": sponsored is not None,
            "url": f"https://www.amazon.com/dp/{asin}",
        })
    return products

A few gotchas I hit:

Prices are locale-formatted. amazon.de gives you 12,99 €. If you scrape other marketplaces, parse per locale instead of stripping commas.
Review counts can be abbreviated ("1.2K") in some layouts. Handle that if you see it.
Sponsored results are mixed in. Decide up front whether you want them; for rank tracking you usually don't.
Duplicate ASINs show up across pages. Dedupe by ASIN.

Step 3: Paginate politely

SEARCH_URL = "https://www.amazon.com/s"

def search(keyword, max_pages=5):
    seen, results = set(), []
    for page in range(1, max_pages + 1):
        html = fetch(SEARCH_URL, params={"k": keyword, "page": page})
        items = parse_search(html)
        if not items:
            break  # end of results, or a layout we don't recognize
        for item in items:
            if item["asin"] not in seen:
                seen.add(item["asin"])
                results.append(item)
        print(f"page {page}: {len(items)} items")
        time.sleep(random.uniform(3, 7))  # don't hammer it
    return results

Random delays of a few seconds matter more than any header trick. Burst traffic from one IP is the fastest way to get a CAPTCHA wall.

Step 4: Store snapshots for price tracking

The useful thing is usually the history, not a single scrape. SQLite is all you need:

import sqlite3
from datetime import datetime, timezone

def save(products, keyword, db="amazon.db"):
    con = sqlite3.connect(db)
    con.execute("""
        CREATE TABLE IF NOT EXISTS snapshots (
            asin TEXT, keyword TEXT, title TEXT, price REAL, list_price REAL,
            rating REAL, reviews REAL, sponsored INTEGER, position INTEGER,
            scraped_at TEXT
        )""")
    now = datetime.now(timezone.utc).isoformat()
    con.executemany(
        "INSERT INTO snapshots VALUES (?,?,?,?,?,?,?,?,?,?)",
        [(p["asin"], keyword, p["title"], p["price"], p["list_price"], p["rating"],
          p["reviews"], int(p["sponsored"]), i, now)
         for i, p in enumerate(products, 1)],
    )
    con.commit()
    con.close()

if __name__ == "__main__":
    kw = "wireless earbuds"
    data = search(kw, max_pages=3)
    save(data, kw)
    print(f"saved {len(data)} products")

Run it daily with cron and you have price and rank history:

SELECT asin, date(scraped_at) AS day, MIN(price) AS price, MIN(position) AS rank
FROM snapshots
WHERE keyword = 'wireless earbuds'
GROUP BY asin, day
ORDER BY asin, day;

Where DIY falls apart

Here's the honest part. The script above works for a few pages a day from a home connection. Past that, you'll hit:

CAPTCHAs at volume. From datacenter IPs (AWS, GCP, a VPS), you'll often get blocked on the first request. You need residential proxies, which cost real money.
TLS / browser fingerprinting. Amazon looks at more than headers. Libraries like curl_cffi (which impersonates Chrome's TLS fingerprint) help a lot over plain requests.
Layout drift. Selectors break every few weeks. Budget time for maintenance and add an alert when a page returns zero items.
Location-dependent prices. Prices and availability depend on the delivery ZIP code (stored in a cookie). Your numbers may not match what your users see.
Headless browsers aren't a silver bullet. Playwright gets you past some checks but is 10–50x slower and still gets fingerprinted.

Option: use an API for the hard part

If you'd rather not maintain proxies and selectors, a scraping API can return the same data as JSON. For example, ScrapeAtlas* has an Amazon product search endpoint that takes the keyword, marketplace, and a pagination cursor:

import os
import requests

API = "https://api.scrapeatlas.com/v1/amazon/product-search"

def search_api(keyword, country="US", max_pages=3):
    results, cursor = [], None
    for _ in range(max_pages):
        params = {"query": keyword, "country": country}
        if cursor:
            params["cursor"] = cursor
        resp = requests.get(
            API,
            headers={"x-api-key": os.environ["SCRAPEATLAS_API_KEY"]},
            params=params,
            timeout=60,
        )
        data = resp.json()
        if data["status"] in ("failed", "challenged"):
            raise RuntimeError(f"request failed: {data['requestId']}")
        for item in data["items"]:
            p = item["product"]
            results.append({
                "asin": p["id"],
                "title": p["title"],
                "price": (p.get("price") or {}).get("current"),
                "list_price": (p.get("price") or {}).get("original"),
                "rating": (p.get("rating") or {}).get("average"),
                "reviews": (p.get("rating") or {}).get("count"),
                "sponsored": False,
                "url": p["url"],
            })
        if not data["pagination"]["has_more"]:
            break
        cursor = data["pagination"]["next_cursor"]
    return results

The output fits the same save() function, so you can swap one fetcher for the other. Check status: partial means you got usable data but some fields weren't available. There are also endpoints for single products by ASIN, reviews, offers/sellers, and best-seller lists, and country covers marketplaces like GB, DE, and JP.

Wrapping up

For small, occasional scrapes: requests + BeautifulSoup + slow, random delays works fine.
For daily tracking of a few keywords: add curl_cffi, retry logic, and a zero-results alert.
For anything at scale or from cloud servers: you'll need residential proxies and ongoing maintenance, or an API that handles them for you.

Whichever route you take, store snapshots rather than overwriting rows. The price history is where the value is.

Questions or a selector that broke? Drop a comment and I'll update the guide.

*I build it.