Your hosting bill has new tenants, and they don’t pay rent. AI crawlers — GPTBot, ClaudeBot, Bytespider, and a swarm of anonymous scrapers — are hammering websites at a scale that infrastructure providers now openly describe as a problem. Kinsta has a good breakdown of how AI bot traffic became an infrastructure problem: on some sites, bots consume the majority of server resources while delivering exactly zero human readers. If your WordPress site has felt slower or your bandwidth graph looks haunted, this might be why — and here’s how to block AI crawlers without shooting yourself in the foot.
First, Check If You Actually Have a Bot Problem
Don’t fix what you haven’t measured. Look at your hosting dashboard’s bandwidth stats, or grep your access logs for the obvious user-agents: GPTBot, ClaudeBot, CCBot, Bytespider, PerplexityBot, meta-externalagent. If bots are a rounding error, close this tab and go optimize something that matters. If they’re a third of your traffic — keep reading.
Before You Block AI Crawlers: The Real Trade-Off
Here’s the honest tension: block every AI bot and you also block being cited by AI assistants — the exact opportunity I covered in my getting-cited-by-ChatGPT post. My take: distinguish between bots. Reputable training/search crawlers that respect robots.txt (GPTBot, ClaudeBot, PerplexityBot) are a strategic choice — allow them if you want AI visibility. Anonymous scrapers that ignore robots.txt and slam your server? Those get the wall, no debate.
Layer 1: robots.txt (the Polite Fence)
For the respectful bots, robots.txt works and costs nothing:
User-agent: BytespiderDisallow: /
Repeat per bot you want out. Know its limits: robots.txt is a request, not a lock. The polite bots honor it; the problem bots use it as a site map.
Layer 2: Cloudflare (the Actual Wall)
This is the layer with teeth, and the free plan is enough. Cloudflare can identify and block AI crawlers at the edge — before they ever touch your server or burn your bandwidth. Turn on its AI-bot controls, enable Bot Fight Mode, and add rate-limiting rules for aggressive paths. On a site I manage, this single change cut bot bandwidth dramatically overnight — the server never even sees the blocked requests.
Layer 3: Server-Level Blocks (for the Stubborn)
For named offenders that dodge everything else, block their user-agents in .htaccess or your Nginx config. Hosting providers like Pressable have documented recipes for this. It’s whack-a-mole — scrapers rotate agents — but it raises their cost, and that’s the game.
My Setup, for the Record
- Allowed: GPTBot, ClaudeBot, PerplexityBot — I want assistants citing my posts.
- Blocked: Bytespider and the anonymous scraper crowd, at the Cloudflare edge.
- Watched: monthly log check, five minutes, calendar reminder. Bots change behavior; so should the list.
The point isn’t bot hatred — it’s that your server exists for readers. Every gigabyte a scraper burns is speed and money taken from the humans (and yes, the useful robots) you actually built the site for. Guard it accordingly.
