What AI Crawler Traffic Actually Looks Like on My Sites cover image
Prateek Zare

Written by Prateek Zare

Software Developer with ML and Data Expertise, 8+ years of experience

Last updated

What AI Crawler Traffic Actually Looks Like on My Sites

Every site gets visits from AI crawlers now. Most owners never see them, because standard analytics either hides bots or counts them as people.

My plugin records every machine by name, so I can show you exactly what this looks like on convertnow.tools. These are the last 30 days, 7 October to 6 November 2026.

Who shows up

AI crawlers and fetchers made 1,854 requests in that window:

AI machineRequestsShare
GoogleOther48426.1%
ClaudeBot48326.1%
Common Crawl26214.1%
ChatGPT Search18910.2%
PerplexityBot1085.8%
You.com804.3%
Google Inspection683.7%
ChatGPT user fetch633.4%
Claude user fetch472.5%
Meta ads check412.2%
Meta AI search160.9%
Google Vertex60.3%
GPTBot50.3%

For scale, the site had 849 human visitors in the same period.

Two kinds of AI traffic

The names split into two groups.

Training and index crawlers, such as ClaudeBot, Common Crawl and GPTBot, collect pages in bulk. They read a lot and send nobody back directly.

Answer fetchers, such as ChatGPT Search, PerplexityBot and the user fetches from ChatGPT and Claude, load a page because someone asked a question right now. That is the closest thing to a real reader that a machine gets.

What surprised me: ClaudeBot read almost exactly as much as GoogleOther, while GPTBot, the crawler most people worry about, made just 5 requests.

Who actually sends people

Crawling is one side. Visits are the other. In the same 30 days, AI assistants sent 26 real visitors who viewed 65 pages, about 6% of all site views. Every one of them came from ChatGPT. No other assistant sent a single visitor.

Those visitors stayed 55 seconds on average and read about 23% of a page.

Which pages AI sends people to

PageVisitorsShare
Combine Strava Activities tool2079.5%
ConvertDash210.3%
ConvertNow Analytics12.6%
MP3 to WAV converter12.6%
Compound interest calculator12.6%
Background remover12.6%

One page does almost all the work. The Strava tool solves a specific problem that people ask ChatGPT about, and ChatGPT links to it. Generic tools that a hundred other sites also offer barely show up.

Crawling is not the same as being cited

The plugin also lines up the two sides for each page: how often AI crawlers fetched it, and how often an assistant sent a real person to it.

PageAI crawlsVisitors from AIPer 100 crawls
Homepage33200
ConvertDash5547.3
Combine Strava Activities323196.9
ConvertNow Analytics2314.3

Across the whole site, AI crawlers made 1,615 requests and AI assistants sent 39 visits. That is 2.4 visits for every 100 crawls. AI crawlers read 514 different pages, and only 6 of them ever got a visitor back.

The homepage was the most crawled page by far, mostly by ClaudeBot, and it sent nobody. The Strava tool was crawled 32 times and sent 31 visitors. Being read by AI is easy. Being the page an assistant picks as the answer is rare, and it goes to pages that solve one specific problem.

What happened to llms.txt

llms.txt is a proposed file that tells AI tools which pages on a site matter most. Rank Math generates one for this site, so I could check who reads it. In 30 days it was fetched 80 times: 53 of those were Google Lighthouse running a page audit, and the rest came from unnamed crawlers. Not one request came from a named AI crawler like GPTBot, ClaudeBot or PerplexityBot.

That does not mean the file is useless, but on this site the AI crawlers ignored it and went straight to the pages.

What this means for site owners

  • Your human numbers stay honest only if bot visits sit in their own report
  • Heavy crawling does not mean visits. ClaudeBot read 483 times and Claude sent nobody this month
  • AI tools link to pages that answer one clear question better than anyone else
  • With names in front of you, robots.txt becomes a choice instead of a guess

See it on your own site

The AI and bot reports come with the free ConvertNow Analytics plugin for WordPress, and with ConvertDash for any other site. Give it a week and look at who has been reading your pages.