Anthropic's ClaudeBot crawled some publishers' sites roughly 500,000 times for every one reader it sent back in 2025, according to Cloudflare's own year-end traffic data. OpenAI's GPTBot managed a comparatively better 3,700 to 1. Both numbers grew over the course of the year, not shrank. This is the crawl-to-refer ratio: how many times an AI crawler requests your pages against how many actual visitors that same platform later sends you. It matters because most GEO advice quietly assumes getting crawled is progress. Increasingly, it just means your content trained a model that never points anyone back to you.
What the crawl-to-refer ratio actually measures
The crawl-to-refer ratio compares how many times an AI crawler requests pages from a domain to how many visits that platform later refers back to it. A ratio of 1,000 to 1 means a bot read a thousand pages for every reader it delivered. It is the clearest single number for how much a given AI platform actually gives back to the sites it depends on for answers.
Search engines have always crawled more than they referred, but the gap stayed close to parity because ranking existed to drive clicks somewhere. AI answer engines break that link. A model can extract everything it needs for a synthesized answer and never surface a source page at all, which is what makes this ratio worth tracking on its own rather than assuming it moves with your search rankings.
How lopsided the ratio has become
Cloudflare's Radar team measured Anthropic's ClaudeBot at roughly 50,000 requests for every referred visit in the first week of August 2025, against 887 to 1 for OpenAI's GPTBot and 118 to 1 for Perplexity. All three ratios grew before the year was out.
Cloudflare's 2025 Year in Review, published December 31, 2025, put Anthropic's full-year ratio at roughly 500,000 to 1 and OpenAI's at 3,700 to 1, and described crawl-to-refer ratios industry-wide as having increased over the prior year. Perplexity kept the lowest ratio of the three named platforms, though Cloudflare didn't publish an exact year-end figure for it.
A crawl-to-refer ratio of 500,000 to 1 means a bot read half a million pages to help write its answers, and sent one person back to see where any of it came from.
Why AI platforms crawl so much more than they refer
Most AI crawling exists to train models, not to answer a live query. Cloudflare found that training-purpose requests made up nearly 80% of all AI bot crawling activity in the periods it measured, with real-time user queries and undeclared purposes together accounting for under 5%. A crawler built to ingest a training corpus has little reason to link back to any single source page.
That skew shows up unevenly by industry. In Cloudflare's snapshot, Anthropic's crawlers were far more extractive in computers and electronics (8,800 to 1) than in news and publications (2,500 to 1), where publishers have pushed hardest for licensing deals and pay-per-crawl arrangements. Much of this activity counts as dark AI traffic: crawling that happens with no attributable session, no referral, and often no visible line item in a publisher's own analytics.
What the falling referral side means for attribution
Referral traffic from AI platforms isn't just growing slowly. On a per-visit basis, it's shrinking. TollBit, which tracks bot and referral activity across a network of publisher sites, measured the ratio of AI bot visits to human visits rising from 1 in 200 in the first quarter of 2025 to 1 in 31 by the fourth quarter, a 60% jump in twelve months.
Click-through rates fell alongside that. Sites without any AI licensing deal saw referral click-through drop from 0.8% to 0.27% across 2025, TollBit found, and even sites with paid licensing agreements fell from 8.8% to 1.33%. Averaged across the open web, TollBit put AI applications' overall referral contribution at 0.12% of traffic, against more than 80% from Google. That's the attribution gap in numbers: the distance between how much content an AI answer draws on and how much credit, in visits, it hands back.
None of that makes the traffic AI does send worthless. We've covered why AI referral sessions convert roughly 5 times better than organic Google traffic: someone who clicked through from a synthesized answer has usually already done most of their research before they arrive.
How to check your own site, and why the number won't sit still
You don't need Cloudflare-scale data to see your own ratio. Pull AI bot requests from your server or CDN logs for GPTBot, ClaudeBot, and PerplexityBot, then compare that count to the AI-referred sessions your analytics shows for the same window. Most sites are surprised by how one-sided the number is even at a small scale.
The ratio also moves faster than most teams check it. A monthly crawler-traffic tracker from websearchapi.ai found Anthropic's combined crawlers improved to roughly 1,683 to 1 in July 2026, down from about 2,947 to 1 in June, while Perplexity's worsened to 322 to 1 over the same stretch. Both remain far behind Google's own crawl-to-refer ratio, which that tracker put at roughly 4.8 to 1. A single snapshot, Cloudflare's or anyone else's, is already dated by the time most teams read it.
Whether it's worth restricting a low-ratio bot depends on what you're protecting. We've covered which crawlers robots.txt actually stops and which ones ignore it anyway. It changes what getting crawled even means for a given site.
A ratio you can't see is one you can't manage. Sites that have accidentally blocked AI crawlers through their CDN or firewall settings, a more common problem than most teams realize, end up with an artificially clean-looking ratio for the wrong reason: no crawling and no referrals at all.
Tools built for AI visibility monitoring, including VizibleAI, track the referral and citation side of that equation across engines, which is the half most server log tools were never built to see.
Frequently Asked Questions
What is a good crawl-to-refer ratio?
There is no benchmark yet that calls any AI platform's ratio good. One July 2026 tracker put Google's own crawl-to-refer ratio at roughly 4.8 to 1 for comparison. Named AI platforms have ranged from the hundreds to the hundreds of thousands to 1 across 2025 and 2026. Lower and falling is the trend worth watching, not any fixed target.
Does a high crawl-to-refer ratio hurt my Google rankings?
No, it isn't a Google ranking signal. It measures how much a specific AI platform gives back in referred visits relative to how much of your content it consumes, which matters for content strategy and licensing decisions, not for organic search position.
How do I calculate my own site's crawl-to-refer ratio?
Count AI bot requests by user agent, such as GPTBot, ClaudeBot, or PerplexityBot, from your server or CDN logs over a set period. Divide that number by the sessions your analytics attributes to that same platform's referral traffic over the same period.
Which AI platform has the worst crawl-to-refer ratio?
Anthropic's ClaudeBot has posted the highest ratios in Cloudflare's data, reaching roughly 500,000 to 1 for 2025 overall. Monthly trackers show individual platforms' ratios shifting significantly month to month, so treat any single snapshot as temporary, not permanent.
Should I block AI crawlers to improve my ratio?
Blocking a crawler removes it from the ratio rather than improving it, and it likely removes you from that platform's answers too. Whether that trade is worth making depends on whether the platform sends you any referral traffic or citations worth keeping.
Track the referral side AI platforms don't advertise
Crawl logs tell you how much AI platforms take. They don't tell you whether your brand actually gets mentioned, cited, or sent traffic when someone asks an AI engine a question in your category, and that gap is where GEO strategy actually happens.
VizibleAI tracks brand mentions, citations, and share of voice across ChatGPT, Claude, Gemini, and Perplexity, so you can see the referral side of the ratio instead of guessing at it.