large language model traffic

From IndieWeb

This article is a stub. You can help the IndieWeb wiki by expanding it with relevant information.


Large language model traffic is (AKA LLM crawling or AI scraping) the phenomenon of web requests made by LLM operators to ingest data for use in training AI models, or to answer user queries in “search” mode (like ChatGPT Search), which tend to ignore robots directives and overwhelm personal blogs, community sites, and other smaller websites.

Increasing usage of LLM tools corresponds with increased crawler traffic from web robots. website owners have noticed traffic spikes in traffic which have made some public sites unavailable. Some crawlers appear not to respect robots.txt directives.

IndieWeb Examples

Several members of the IndieWeb community have reported AI traffic becoming an increasing burden on their servers.

  • Joe Crawford was told by system administrators from his web host that bot spikes accounted for multiple crashes of Apache.
  • As a result of heavy traffic on the indieweb.org wiki, web server nginx rules were added to prevent access to "diff" pages for individual wiki pages for not-logged in user agents
  • capjamesg prefers AI bots not to crawl his website. Because of accounts that some AI bots do not adhere to robots.txt (i.e. https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/), James blocks many AI bot user agents using nginx. Matching user agents are returned a 404 for every web page.
  • Jak2k does not want AI bots to crawl their website. They used Anubis in the past, but switched to Iocaine because it does not require client-side JavaScript and can also poison AIs.
  • Aaron Parecki added an nginx rule to block access to the indieweb.org history pages for not-logged-in users in order to combat traffic slowing down the wiki
  • The OpenID.net website has been regularly taken offline throughout September-October 2025 and the admins have been working to add a CDN in front of the site.
  • Ross A. Baker is using Iocaine to protect his static site and self-hosted services. In November, 2025, he is averaging over 20 million requests per day, 97.8% of which are deemed garbage. Because his code forge gets hit hardest, he generates fake responses from BASIC Computer Games, which can be sampled on his public maze.
  • 2026-02-12 Tantek Çelik's main site tantek.com went down for ~7 hours due to a spike in traffic primarily from LLM crawlers that exhausted all available PHP processes on his shared server. Solution was to enable Cloudflare’s "Control AI crawlers" feature of "Block AI training bots" with a setting of "Block on all pages".
  • fluffy has been experiencing an ever-increasing arms race against crawlers; currently they use a number of anti-bot measures, including requiring log-in for suspicious URLs, gating expensive endpoints behind a sentience check, and running an anti-crawler tarpit on a dedicated server to try to disrupt the various models. They previously also used Cloudflare's anti-bot protection but found this to be largely ineffective, and felt it was contributing too much to the centralization of the web.
  • 2026-05-26 WinnerWind's server was taken offline due to Claude despite setting up Iocaine, user agent based blocking and even Cloudflare's Anti bot protection. Claude ignored robots.txt directives and continued to scrape the garbage pages generated by Iocaine until the server couldn't take it anymore. Iocaine says that in the past month, 99.9% of requests are deemed garbage. He noticed that Claude's crawlers would sometimes masquerade themselves as fully compliant Chromium browsers.
  • 2026-05-29 WinnerWind after much trial and error found that using a mixture of Iocaine and upstream blocking brought down the average traffic noise down to almost zero. Instead of using Iocaine (see above) to poison all the bots all the time, which causes horrible network spikes, let upstream (be it a VPS with more capable network hardware or Cloudflare etc.) redirect flagged bots to another URL, such as NFSHost. This redirection causes bots to ignore your site for long periods of time, and also stops bots from making more than one or two requests on your site. In case of Iocaine, this can be done by checking the URI and seeing if it matches the path prefix generated by it.
    Graph of network traffic where each point is the cumulative network traffic in the past six hours (in GB). Red line shows the maximum capacity that the server can handle for brief bursts of time before slowdowns or crashes.


News reporting

Traffic reports

The inside story of how one of film data's most trusted sites vanished overnight, and why every website you rely on is more fragile than you think.

Goodmorning to the Anthropic Claude AI training scraper that suddenly decided to request 660 thousand pages (exactly the number I had remaining on the starter plan) and brought Pikiwedia down.

The traffic to my blog and my books (which were free to read online) increased beyond what I can currently afford. Virtually all of it comes from AI crawlers, so there is no ad income.

Tools to combat

This section is a stub. You can help the IndieWeb wiki by expanding it.

…we can confirm that at least Huawei networks now send the challenge responses and they actually do seem to take a few seconds to actually compute the answers… we assume that AI crawlers leveled up their computing power to emulate more of real browser behaviour

Proposed Tools

Robots.txt Guidance to Scrapers

Since 1994, Robots.txt has been the sidefile mechanism site operators may use to suggest to web spiders how to crawl their sites.

  • User agent values can be called out by specific name and told whether they are allowed or disallowed.
  • Content-Signals, from Cloudflare proposes a robots.txt a specific directive called Content-Signal with values of ai-train, search, ai-input with values of yes or no to steer bots and scrapers, should they respect the value.
    • As of 30 September 2025 the IndieWeb wiki implements Content-signal directives.
  • Related: llms.txt proposal
  • There was a 2024 proposal New Internet Rules Will Block AI Training Bots to add AllowAITraining and DisallowAITraining as directives inside robots.txt but the proposal has since expired.
  • Dark Visitors keeps a list of robot user agents. This includes AI bots which are listed in categories like "AI Agent", "AI Assistant", "AI Data Scrapers" and more general bots like "Archivers" and link preview "Fetchers".

Brainstorming

Types of LLM bot

There are several types of bot that act on behalf of or collect data for an LLM. The types of bot vary by service provider, but are mainly distinguished by their intent.

Some bots crawl content (crawlers); others retrieve content for use in answering a user's query ("search bots" or bots that retrieve a web page in service of answering a user's question); others download data for the purpose of training AI models.

Security firm DataDome distinguishes 3 types:

  • Training scrapers, which ingest large volumes of public content to improve model performance. These are typically used to build or refine foundation models and may not honor robots.txt or rate limits.
  • Prompt-time fetchers, which retrieve real-time data to supplement LLM outputs—think of AI copilots or search assistants querying web pages on-the-fly to answer user prompts.
  • Agentic crawlers, which act more like users. These bots are capable of clicking, scrolling, submitting forms, and navigating complex UIs, often as part of a RAG (retrieval-augmented generation) pipeline or testing framework.

Source: AI Agents at the Gate: Understanding & Securing Against LLM Crawlers

See Also

  • large language model
  • robots.txt
  • common crawl is intended to share site data with crawlers so they would avoid inflicting onerous web traffic
  • https://mastodon.social/@st3phvee/116415952485784541
    • "If you also run a website in the #indieweb / #smolweb / #personalweb space, are you seeing similar bot onslaughts? If so, how are you dealing with it? I'm getting so frustrated... I don't pay for my bandwidth, so it doesn't affect me financially, but it makes me so irritated to know that the majority of the traffic I receive is from bots and scrapers trying to exploit my site and harvest my data for AI training. 🧵2/2" @st3phvee April 16, 2026
  • https://mastodon.social/@st3phvee/116415940185404566
    • "More depressing stats. In the last 24 hours, only 14 real people visited my site, as evidenced by the challenge solve rate. Approximately 2000 requests from known botnets were blocked outright. 332 requests made it past the block list, but couldn't solve the challenge (meaning they were also most likely bots). 18 AI bots disregarded their robots.txt instructions and attempted to access other areas of my site. 21 requests were made to hidden links designed to catch scrapers. 🧵1/2#bots #indieweb" @st3phvee April 16, 2026
  • another casualty of LLM traffic costs: https://2ality.com/

    …traffic to my blog and my books (which were free to read online) increased beyond what I can currently afford. Virtually all of it comes from AI crawlers…

Retrieved from "https://indieweb.org/wiki/index.php?title=large_language_model_traffic&oldid=114328"