common crawl
This article is a stub. You can help the IndieWeb wiki by expanding it with relevant information.
common crawl is an open repository of web crawl data, extensively used for web analysis and generative text large language model training.
How to Remove Content from Common Crawl
You can request content to be removed from the Common Crawl dataset by emailing their team directly.
IndieWeb Examples
capjamesg requested removal of his personal website from common crawl.
Criticisms
- 2025-11-04 The Nonprofit Doing the AI Industryβs Dirty Work Common Crawl claims to provide a public benefit, but it lies to publishers about its activities by Alex Reisner
See Also
- Internet Archive
- See https://microformats.org/wiki/any23 for microformats related work on the parser used by common crawl
- large language model traffic
