How to detect AI agent traffic

As ChatGPT, Claude, Perplexity and other AI tools become mainstream, their visits to your site are growing fast. But traditional analytics (Google Analytics, Baidu Tongji, etc.) often lump these visits into “direct” or generic “bot” categories — leaving you blind to the real AI traffic picture.

This guide walks through the practical techniques for identifying AI agent visits.

Step 1: User-Agent strings

The simplest signal is the User-Agent HTTP header:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/2.0; +https://openai.com/bot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; [email protected]
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot

Common AI agent User-Agents:

Provider	User-Agent keywords
OpenAI	ChatGPT-User, GPTBot, OAI-SearchBot
Anthropic	ClaudeBot, Claude-Web
Google	Google-Extended, Gemini-Deep-Research
Meta	Meta-ExternalAgent, Meta-ExternalFetcher
Perplexity	PerplexityBot, Perplexity-User
Apple	Applebot-Extended

Step 2: Behavioral signals

User-Agent strings are easy to spoof. More reliable: analyze access behavior.

Signal 1: extremely fast consecutive requests

AI agents often crawl multiple pages within a few seconds:

14:23:01 GET /
14:23:01 GET /blog/post-1
14:23:01 GET /blog/post-2
14:23:02 GET /blog/post-3

Signal 2: no JavaScript execution

AI agents usually don’t run JavaScript, so client-side tracking fires nothing:

• server logs the request
• frontend emits no matching event

Signal 3: stable visitor pattern

A single AI agent’s IP range is fairly stable. You can validate via reverse DNS:

$ host 23.98.142.10
10.142.98.23.in-addr.arpa domain name pointer crawl-23-98-142-10.openai.com.

Step 3: reverse DNS verification

The most authoritative check is the IP’s PTR record:

func isAIBot(ip string) bool {
    names, err := net.LookupAddr(ip)
    if err != nil {
        return false
    }
    for _, name := range names {
        if strings.HasSuffix(name, ".openai.com.") ||
           strings.HasSuffix(name, ".anthropic.com.") ||
           strings.HasSuffix(name, ".googlebot.com.") {
            return true
        }
    }
    return false
}

Step 4: robots.txt control

After identifying AI agents, you can choose:

• 🤖 Allow training: AI learns your content (more exposure)
• 🚫 Block training: via robots.txt (e.g. User-agent: GPTBot\nDisallow: /)
• 🔍 Allow search but block training: User-agent: Google-Extended\nDisallow: /

In practice: Site Distiller

Site Distiller combines all the above, automatically categorizing visits from 20+ AI agents into:

• AI Bot (e.g. GPTBot) — training-data crawlers
• AI User (e.g. ChatGPT-User, Perplexity-User) — on-demand retrieval for user queries
• AI Search (e.g. OAI-SearchBot) — search-engine-style AI

This way you can finally see: is AI training on your content, or are users discovering it through AI tools?

Next steps

Want to see how much AI traffic your site gets? Sign up free for Site Distiller — 30-second setup, first AI-agent visit visible within 5 minutes.