Crawl-to-Refer Ratio: The One Metric That Tells You Whether AI Is Worth It to Your Site

Last updated: June 2026.

Every site owner watching their logs in 2026 sees the same picture: AI bots everywhere. GPTBot reading the docs, ClaudeBot in the archives, PerplexityBot fetching last week's post. The reflexive question is "should I block them?" and the honest answer is: you cannot decide without a number.

This page defines that number. We call it the crawl-to-refer ratio, and we think it belongs in every analytics dashboard built after 2025.

Definition

Crawl-to-refer ratio = AI crawler requests received ÷ human visits referred back by that engine, over the same period.

It measures the bargain each AI engine offers you. A ratio of 50:1 means an engine made 50 crawl requests for every visitor it sent you. The lower the ratio, the better the deal. An engine that crawls heavily and refers nothing is extracting; an engine that crawls lightly and refers steadily is distributing.

The ratio only makes sense per engine, because the deal varies wildly between them, and that variance is exactly the decision-making information you need.

How to calculate it

You need two numbers per engine, same time window (a month is right):

1. Crawl requests : from server or CDN logs, count requests by user agent family. Group them by company: OpenAI = GPTBot + OAI-SearchBot + ChatGPT-User; Anthropic = ClaudeBot + Claude-SearchBot + Claude-User; and so on (the full user agent table is here ↗).

2. Referred visits : from analytics, count sessions by AI referrer (chatgpt.com, perplexity.ai, claude.ai...) per engine ↗.

Example, one month:
OpenAI crawls: 14,200 requests  | ChatGPT referrals: 310 visits → ratio 46:1
Perplexity crawls: 1,900        | Perplexity referrals: 95     → ratio 20:1
ByteDance crawls: 22,000        | referrals: 0                 → ratio ∞ (pure extraction)

Two honest caveats. First, referrer-based counts undercount real influence (app clicks and post-answer brand searches carry no AI fingerprint), so your true ratios are better than measured. Second, training crawlers pay you back on a delay measured in model versions, not weeks. The ratio is a flashlight, not a courtroom.

The trap in the numerator: user agents are not evidence

Almost everyone computing this ratio computes the top half wrong, and it makes every engine look worse than it is.

A user agent is a claim, not an identity. curl -A "GPTBot" https://yoursite.com takes five seconds, and scrapers do it constantly, sometimes to look legitimate, sometimes so that the block they earn lands on OpenAI's name instead of theirs. Grep your logs for GPTBot and you are counting OpenAI plus an unknown number of impostors. Your crawl count is inflated, so your ratio is pessimistic, and you may be about to block an engine over crawling it never did.

The verification is at the network layer, and there is no shortcut:

• Published IP ranges for the engines that publish them (OpenAI, Perplexity). A GPTBot request from outside OpenAI's range is a forgery.
• Reverse DNS for the ones that do not (Google, Bing, Apple): resolve the requesting IP to a hostname, then resolve that hostname back to the same IP.

A hit that fails the check is not a crawl by that engine, and it does not belong in the numerator. Datalenk runs both checks on every bot hit and marks the failures as spoofed, so the crawl number you divide by is the number that actually happened. It is a dull detail that changes the answer.

Benchmarks: the takers and the givers

Public data is young but directionally consistent, and the numbers are stark. Cloudflare's June 2026 data put the spread in black and white: Anthropic's ClaudeBot crawled about 11,122 pages for every single human visit it referred back (week of May 25 to June 1, 2026), and as high as ~24,000:1 across Q1, while OpenAI's GPTBot sat near 1,276:1. In the same window, bots accounted for 57.4% of web traffic to HTML content, with humans at 42.6%: the web now serves more machines than people, which is the whole reason this ratio matters. SEOmator's GEO data report and our own observations sketch the same hierarchy:

Engine	Typical behavior	Deal quality
Perplexity	Light crawl, citation-driven referrals	Best ratio of the majors: citations are its product
OpenAI (search + user bots)	Heavy crawl, growing referrals since ChatGPT search and the May 2026 live-link update	Improving fast
Google (AI Overviews / Gemini)	Crawl bundled with Googlebot, referrals partly cannibalized from your own organic clicks	Hard to isolate, watch closely
Anthropic	Moderate crawl, modest but real referrals via Claude citations	Middling, improving
Common Crawl (CCBot)	Bulk crawl, zero direct referrals	Indirect only (feeds many labs)
ByteDance (Bytespider)	Very heavy crawl (doubled by May 2026), no referral mechanism	Worst deal on the table

A site-level rule of thumb until standardized benchmarks exist: under 25:1 for a search-oriented engine is a good deal; over 200:1 with no trend toward improvement means that engine is a cost center, and a robots.txt decision is warranted.

What to do with your ratio

1. Good ratio (engine refers meaningfully) : let everything in, and invest in citable content for that engine; it is buying distribution with your bandwidth, cheaply.
2. Bad ratio on a search/user engine : usually a visibility problem on your side rather than malice: blocked search bots, client-rendered content the bots cannot read, or pages that are not quotable. Fix access first (check your robots.txt against the current bot list ↗ and confirm your key pages are server-rendered), content shape second.
3. Bad ratio on a training-only crawler : a values-and-bandwidth decision. Blocking GPTBot does not touch your ChatGPT search visibility; blocking Bytespider costs you approximately nothing today.
4. Track the trend, not the snapshot. Engines change their referral behavior in steps (ChatGPT search launch, the May 2026 live links). A taker can become a giver in one product release, which is why this belongs in a dashboard rather than a quarterly log dive.

Where the two halves come from (and what Datalenk actually gives you)

Let us be precise, because the whole point of this page is a number people get wrong.

Datalenk does not print a "crawl-to-refer ratio" figure on a card in the dashboard. What it does is put both halves of the fraction in the same place, on the same time window, per engine, which is the part that is genuinely hard:

• The numerator: verified crawls per engine. This is collected server-side, through a small beacon in your middleware or edge function, because a crawler never runs your analytics script (JavaScript-only tools are structurally blind here, no matter what they claim). Every hit is checked against published IP ranges or reverse DNS, and spoofs are flagged rather than counted.
• The denominator: referral visits per engine, the AI Assistant channel broken out by ChatGPT, Perplexity, Claude, Gemini and the rest. Plus, because visits are not the real prize, the revenue those visits produced.

The division is yours to do, and it takes five seconds once the two columns are side by side and honest. What took the industry two years was getting them side by side and honest.

The metric is bigger than any tool, including ours: if you build analytics, steal it. The web is renegotiating its deal with AI, and you cannot negotiate what you do not measure ↗.

Get both halves of your ratio in one dashboard. Verified AI crawls per engine, referral visits per engine, and the revenue behind them. 14-day trial, card required. Start your trial.

FAQ

What is a crawl-to-refer ratio? The number of requests an AI engine's crawlers make to your site divided by the human visits that engine refers back, over the same period, per engine. It measures what each AI engine takes versus what it gives.

What is a good crawl-to-refer ratio? Directionally: under 25:1 for search-oriented engines is healthy, over 200:1 with no improvement is extraction. Standardized benchmarks do not exist yet; per-engine trends matter more than absolutes.

Why is my ratio so bad? Four usual causes, and check the last one first: you are counting spoofed user agents as real crawls, which inflates the numerator (verify by IP, not by user agent string). Then the real ones: search bots blocked in robots.txt while training bots roam free, JavaScript-rendered content AI crawlers cannot read, or content that answers nothing quotable. Access problems first, content problems second.

Can I compute this from my analytics tool? Only if it sees bots, and a JavaScript snippet does not: crawlers never execute the script. You need a server-side source for the crawl half (raw logs, a CDN export, or a beacon in your middleware) and your analytics for the referral half, on the same window, per engine.

Does blocking high-ratio bots hurt my AI visibility? Blocking training bots (GPTBot, ClaudeBot, Bytespider, CCBot) does not affect AI search citations. Blocking search and user-fetch bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) absolutely does. Know which type you are blocking ↗.