DEV Community

#robotstxt

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
One robots.txt Line Will Opt Your Site Out of AI Training at Google, OpenAI, Anthropic and Microsoft
neuraitool profile

One robots.txt Line Will Opt Your Site Out of AI Training at Google, OpenAI, Anthropic and Microsoft

#robotstxt #ai #webdev #seo
Comments 1
3 min read
Blocking AI crawlers: robots.txt vs Cloudflare vs Anubis, and what each one actually does
rozoomnews_editor profile

Blocking AI crawlers: robots.txt vs Cloudflare vs Anubis, and what each one actually does

#aicrawlers #robotstxt #cloudflare #anubis
Comments
8 min read
We read the robots.txt of 151 big websites. The syntax was fine. The problems were elsewhere.
crawlerpeek profile

We read the robots.txt of 151 big websites. The syntax was fine. The problems were elsewhere.

#seo #webdev #ai #robotstxt
Comments
2 min read
Say no to AI training, not to AI search: robots.txt per bot
richicodes profile

Say no to AI training, not to AI search: robots.txt per bot

#robotstxt #ai #seo #crawlers
Comments 1
5 min read
111 of the top 1,000 sites block GPTBot. 35 never mention the bot behind ChatGPT search.
mahirhir profile

111 of the top 1,000 sites block GPTBot. 35 never mention the bot behind ChatGPT search.

#ai #seo #webdev #robotstxt
Comments 2
9 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.