You published llms.txt, cleaned up your structured data, and now you want to know one thing: is it working, are ChatGPT, Perplexity and Gemini actually citing your site. Open Google Analytics and the honest answer is that it cannot tell you. GA4 only sees a visit if someone clicks a link from an AI answer through to your site, and Cloudflare's own 2026 crawl data shows why that view is so incomplete: GPTBot alone crawls roughly 1,255 pages for every one referral it sends back, and ClaudeBot's ratio is closer to 20,583:1. This is a practical, cited setup guide for the three data sources that actually answer the question: server logs, GA4 channel grouping, and manual prompt probing.
Why Google Analytics does not show you this
GA4 is a client-side, click-based measurement tool: it fires when a browser loads your page and can attribute a referrer if one is present in the request, but an AI engine that answers a question by summarizing your content and citing it as a source, without the user ever clicking through, leaves no trace in GA4 at all. This is the single biggest blind spot founders have about their own GEO performance, and it is not a small gap: Cloudflare's analysis of AI bot traffic found that over the past year, 80% of all AI crawling was for training data collection, 18% for search indexing, and just 2% resulted in an actual user visit.
Separately, the crawl itself, the AI engine's bot visiting your page to read and index it before it can ever cite you, happens entirely outside the browser and therefore outside anything GA4's JavaScript snippet can see. To see crawl activity at all you need to look at server logs, not analytics, which is exactly what step 1 and step 2 below set up.
How big is AI crawl traffic, really?
It is large and growing fast, which is why ignoring it in your analytics setup is increasingly expensive. Cloudflare's mid-2026 data shows Googlebot still leads AI-adjacent crawl volume at roughly 27.3% of AI-related bot requests, with GPTBot and ClaudeBot trading second place month to month, ClaudeBot briefly doubling to nearly 20% of requests in one month alone. Separately, AI-driven search visits grew from 15.6 billion to 27.4 billion between Q1 2025 and Q1 2026, a 42.8% year-over-year increase, per the same Cloudflare research.
Put together, this means a meaningful and rising share of how people discover your business now happens inside an AI answer, not a search results page, which is the entire premise behind generative engine optimization and answer engine optimization as disciplines distinct from classic SEO.
Step 1: identify AI crawlers in your server logs
- GPTBot (OpenAI) — user-agent string contains 'GPTBot', the primary crawler that indexes content for ChatGPT and OpenAI's models, documented in OpenAI's crawler overview.
- ChatGPT-User (OpenAI) — a separate user-agent that fires when a live ChatGPT session with browsing fetches a page in response to a specific user question, distinct from GPTBot's background crawling.
- PerplexityBot — user-agent contains 'PerplexityBot', Perplexity's crawler for indexing and real-time answer generation, per Perplexity's own documentation.
- ClaudeBot (Anthropic) — user-agent contains 'ClaudeBot', used for both training data collection and, increasingly, live answer retrieval; Cloudflare's 2026 data shows its share of AI bot requests now regularly overtakes GPTBot's.
- Google-Extended — not a separate crawler but a token in robots.txt that controls whether Googlebot's data can feed Gemini and AI Overviews, worth confirming it is not accidentally blocked.
Step 2: set up log-based crawl tracking
- Get access to raw request logs: on Vercel this is the Logs tab or a Log Drain to a storage bucket, on Cloudflare it is Logpush, on a self-managed Nginx server it is the access log file directly, any of these expose the user-agent string per request.
- Filter for the AI crawler user-agents from step 1 using a simple grep or a log-query filter, for example `grep -E 'GPTBot|ChatGPT-User|PerplexityBot|ClaudeBot' access.log`, and export the matches with timestamp and requested path.
- Aggregate into a weekly count by bot and by page: a simple Google Sheet with columns for date, bot name, path and status code is enough to start, you do not need a full data warehouse for the first few months of tracking.
- For ongoing tracking at scale, pipe logs through a Cloudflare Worker or a scheduled export into BigQuery or a similar store, then build a small dashboard (even a pivot table) showing crawl frequency per bot over time and which pages get crawled most.
- Watch for a rising trend on your highest-value pages specifically, not just total crawl volume, since a bot crawling your homepage repeatedly matters less than it consistently reaching your service and FAQ pages where citable answers actually live.
Step 3: track AI referral traffic and citation proxies in GA4
- Build a custom channel group in GA4 (Admin > Channel Groups) that buckets referrers from chatgpt.com, perplexity.ai, gemini.google.com and copilot.microsoft.com into a single 'AI Answer Engines' channel, so this traffic is not lost inside 'Referral' or 'Direct'.
- Where possible, add UTM parameters to links you control that AI engines might pull from, like your llms.txt entries or press mentions, so any resulting click carries clear attribution instead of falling back to a bare referrer.
- Watch for unexplained spikes in Direct/(none) sessions that correlate with a known mention or a period of active publishing, since a user who reads an AI-generated answer and then manually types your URL or searches your brand name shows up as Direct, not as an AI referral. Given GPTBot's roughly 1,255:1 crawl-to-refer ratio cited above, most AI-driven awareness will never show up as a clean referral at all, this Direct spike is often the closest proxy you get.
- Segment by landing page, not just by session count, since a citation is only valuable if it lands users on the page actually relevant to their query. A flood of AI-referred sessions all landing on your homepage suggests the AI is citing you as a brand, not for a specific answer.
Step 4: manually probe for citations, on a schedule
- Build a fixed list of 15-20 target queries your ideal customer would actually type, phrased as questions, not keywords, for example 'how much does an AI marketing agency in India cost' rather than 'AI marketing agency India'.
- Run the same list weekly through ChatGPT (with browsing/search enabled), Perplexity and Gemini, and log in a spreadsheet whether your domain appears as a cited source, which specific page it links to, and where in the answer it appears.
- Log competitor presence on the same queries in the same pass, since relative position (are you cited alongside three competitors, or not at all) is a more actionable signal than a single yes/no for your own site.
- Track change over time, not a single snapshot, since AI answer engines update their retrieval and ranking frequently and a page that was not cited in week one can appear after a content or schema change, the trend is the useful data point, not any one week's result.
Common mistakes when measuring GEO performance
- Relying on GA4 alone and concluding 'AI traffic is basically zero', when the real gap is that GA4 cannot see crawling or citation-without-click at all. Only the log data and manual probing in steps 1-4 answer that, and a 1,255:1 crawl-to-refer ratio means the absence of referral traffic tells you almost nothing about whether you are being read and cited.
- Testing an AI engine once and treating the result as final, when these systems' retrieval and citation behavior shifts week to week as they update their indexes and ranking.
- Only testing with the browsing-enabled or search-grounded mode of a tool and assuming the base model behaves the same, when a non-browsing ChatGPT answer draws only from training data and will never cite a page published after its training cutoff.
- Ignoring which specific page gets cited in favor of just tracking whether the domain appears at all, since GEO improvements should be judged page by page, the same way you would judge SEO rankings by URL, not just by domain.
- Blocking AI crawlers by accident and never noticing, a real risk given that one academic analysis found the share of reputable sites blocking AI crawlers in robots.txt rose from 23% in September 2023 to nearly 60% by May 2025, often without the site owner realizing their own configuration was the cause of zero citations.
How AIBOOTSTRAPPER solved this for a client: PropLock
This is the same discipline behind PropLock, a UK real estate platform AIBOOTSTRAPPER built with a GEO-optimized site from the ground up: AI-written listings, real-time lead scoring, and content structured specifically so both search engines and AI assistants could parse and surface it for local property intent.
The client went from a traditional agency with no measurement of its own digital footprint to 12,000 organic visitors a month within 90 days of launch, and 47% more qualified viewings, because the site was built to be crawlable and citable from day one rather than retrofitted later. That is the same gap this guide's log-tracking and citation-probing setup is designed to catch early. Full results are on the case studies page.
How AIBOOTSTRAPPER helps
Measuring GEO performance is only useful if you act on what it shows. If your logs show ClaudeBot crawling frequently but Perplexity never appears in your manual probes, that points to a specific, fixable gap in structured data or crawler access, not a reason to give up on GEO entirely.
AIBOOTSTRAPPER sets up this full measurement stack, log-based crawl tracking, GA4 channel grouping and a recurring manual citation audit, for clients alongside the underlying llms.txt, schema and crawler-access work, so you can see whether GEO investment is actually moving the needle. If you want this built and monitored for your site, get in touch and we will run a first crawl-and-citation audit before recommending anything.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.
