---
title: "AI SEO Log File Analyzer Tool | Justin Hà"
description: "Free alternative to Screaming Frog Log File. Save €99 analyze AI crawlers like GPTBot & ClaudeBot for GEO/ AI SEO website."
url: https://justinha.info.vn/ai-log-file-analyzer-tool/
---

# AI SEO Log File Analyzer Tool | Justin Hà

Free GEO / AI SEO Tool

# Free SEO Log File *Analyzer* Tool

Upload your server access log to discover how 147 AI crawlers - GPTBot, ClaudeBot, PerplexityBot, Grok, DeepSeek and more - are indexing your site. Get your GEO Score™ in seconds. 100% free, runs in your browser.

📁

Drop your log file(s) here

or click to browse from your computer - select multiple files (e.g. daily or monthly rotated logs) to combine them into one analysis

.log
.gz
.txt
.csv
.tsv

Choose Files

Where to get your log file

📁 cPanel

File Manager → logs → access-logs
/home/username/logs/domain-ssl_log

☁️ Cloudflare

Analytics → Logs → Enable Logpush
Enterprise or via API

🌐 AWS CloudFront

Distribution → Logs → S3 Bucket
s3://your-bucket/cf-logs/

💻 SSH / Terminal

cat /var/log/nginx/access.log
| gzip > access_log.gz

€0

100% Free - Forever

Replaces Screaming Frog Log File Analyser at €99/yr

0

AI Bot Patterns

Every major LLM crawler - GPTBot, ClaudeBot, Grok & more

0

Total Bot Patterns

AI + search engines + social media + SEO tools

5

Millions of Log Rows

Up to 1 GB - all processed locally in your browser

.gz

Native .log & .gz

Drop compressed files - no conversion, no upload

## Why use our *free* tool?

Everything Screaming Frog's Log File Analyser offers - and much more AI intelligence - at zero cost. No install, no account, no data sent to any server.

Save €99 / year

💰

### 100% *Free* Forever

Screaming Frog Log File Analyser costs €99/year. Our tool is completely free - no licence, no subscription, no paywall.

↓ Save €99 vs Screaming Frog

📦

### Up to *1 GB* & 5 Million Rows

Handle large enterprise log files up to 1 GB per file, with up to 5 million rows - all processed locally inside your browser. Upload multiple files at once (e.g. daily or monthly rotated logs) and they're combined into one analysis. No upload limits.

🤖

### *147* AI Bots · 74 Companies

Detect every major AI crawler: OpenAI, Anthropic, Google, Perplexity, xAI, Meta, Mistral, DeepSeek, Cohere, Kagi, Brave, Exa, Qwen and more - by name and product.

🔒

### Native *.log* & *.gz* Support

Drop in raw `.log`, compressed `.gz`, `.txt`, `.csv` or `.tsv` - no conversion needed. Your data never leaves your device.

## How it *works*

Analyse millions of log rows in seconds - completely in your browser. No data leaves your device.

01

### Upload Your Log File(s)

Drag and drop or browse for any server access log - Apache, Nginx, Cloudflare, AWS, cPanel. Supports .log, .gz, .txt, .csv up to 1 GB per file. Select several files at once to combine daily or monthly rotated logs into one analysis.

02

### Instant Analysis

Our browser-side engine parses every row, identifies 147 AI bot patterns across 74 AI companies, calculates your GEO Score™, builds 20+ charts and surfaces key recommendations.

03

### Export Your Report

Download a full PDF report with charts, scores and action items - ready to share with clients or your team. No account required.

## Why GEO & AI SEO *require* log file analysis

Log analysis has always been a technical SEO skill. In the GEO era, it becomes indispensable - because AI platforms have no equivalent of Google Search Console. Your server logs are the only source of truth.

⚠️

### AI search has *no Search Console.*

Google gives you Search Console - impressions, clicks, index coverage, crawl stats. ChatGPT, Perplexity, Claude, Grok and every other AI platform give you **nothing**. No dashboard. No reports. No API. Your server access log is the only place you can see whether GPTBot, ClaudeBot or PerplexityBot has ever visited your site - and what happened when they did. If you're not reading your logs, you're flying blind in the most important new channel in search.

### The AI *discovery pipeline*

Before an AI platform can cite your content in an answer, it must first crawl it. Log analysis is the only way to verify where your content sits in this pipeline.

Step 01 - You control this

🕷️

#### AI Bot *Crawls* Your Page

GPTBot, ClaudeBot or PerplexityBot visits your URL. Visible only in your server logs. If blocked by robots.txt, WAF, or rate limiting - nothing else in the pipeline fires.

→

Step 02 - AI decides

🧠

#### Content Gets *Indexed* / Trained

The AI platform processes your content into its knowledge base or live index. Log frequency, recrawl rate and status codes all influence how much of your content is absorbed.

→

Step 03 - Outcome

💬

#### AI *Cites* Your Content

Your brand, article or expertise surfaces in ChatGPT, Perplexity or Claude answers. This is the GEO goal - and it starts at Step 01. No crawl = no citation.

### What *only* your log file can tell you

No third-party tool, no rank tracker, no AI visibility monitor can give you this data. It lives only on your server.

01

#### Which AI bots are actually crawling you - right now

Not estimates. Not averages. Real bot names, real timestamps, real IPs. You'll know if GPTBot crawled your site yesterday, which 14 pages it read, and whether it got a 200 or a 403.

02

#### Whether your robots.txt rules are working for AI bots

You might have `Allow: /` for GPTBot - but is it actually crawling? Or is a WAF rule, IP block or rate limit silently rejecting every request? Logs expose the truth robots.txt validation tools can't see.

03

#### Which pages AI bots ignore - your GEO coverage gap

Google might index 3,000 of your pages. GPTBot might only be reading 12. The pages AI bots never visit are your GEO dead zones - no crawl means no training data means no citations. Logs are the only way to find them.

04

#### How AI crawler behaviour is changing over time

Is PerplexityBot crawling you more this month than last? Did ClaudeBot suddenly stop after a server change? Trend data from logs lets you correlate technical events with changes in AI crawl activity - before it hits your citation rate.

05

#### The real cost of errors to your AI visibility

Every 404 or 500 that an AI bot hits wastes its crawl budget on your site and teaches it your content is unreliable. Logs quantify exactly how much of your AI crawl budget is being destroyed by errors - and on which pages.

06

#### AI referral traffic - proof that GEO is working

When someone clicks a citation in Perplexity or a ChatGPT link and lands on your site, that referral appears in your logs. It's the clearest signal that your GEO efforts are converting into real traffic.

### Traditional SEO *vs* GEO: a new diagnostic skill

Log analysis existed in technical SEO - but GEO makes it a front-line skill, not a specialist one. Here's how the mindset shifts.

Traditional SEO Approach

- Check Google Search Console for coverage

- Monitor Googlebot crawl stats dashboard

- Fix 404s to protect Google crawl budget

- Optimise for one primary crawler (Googlebot)

- Log files = occasional advanced audit

- Rank tracking tells you if SEO is working

vs

GEO / AI SEO Approach

- No Search Console exists - logs are your only data

- Monitor 147 AI bot patterns across 74 AI companies

- Fix errors to protect crawl budget across all AI platforms

- Optimise for 332 total bot patterns simultaneously

- Log files = weekly core GEO health check

- AI referral traffic & citation tracking shows GEO ROI

## Which bots are *tracked?*

332 user-agent patterns across every category - AI crawlers, search engines, social media bots and SEO tools. All detected and classified automatically.

147

AI Bot Patterns

from 74 AI companies

72

Search Engine Patterns

Google, Bing, Baidu, Yandex & 20+ more

23

Social Media Bots

Twitter, LinkedIn, TikTok, WhatsApp & more

48

SEO Tool Bots

Ahrefs, Semrush, Screaming Frog & more

42

Scrapers & Other

Research crawlers, cloud/security scanners & more

332

Total Patterns

all categories combined

🤖 AI & LLM Bots 147

🔍 Search Engines 78

📱 Social & SEO 73

📦 Scrapers & Other 36

**74 AI companies tracked:**
ChatGPT · Claude · Perplexity · Gemini · Grok (xAI) · Llama (Meta) · Copilot (Microsoft) · Mistral · DeepSeek · Cohere · You.com · Kagi · Phind · Brave AI · Exa AI · Qwen (Alibaba) · Doubao (ByteDance) · Apple Intelligence · HuggingFace · Manus · Linkup · WRTN · Kimi (Moonshot AI) · Devin (Cognition AI) and more

**147** AI & LLM Bot patterns listed below

AddSearchBot
`AddSearchBot`
Operator: **AddSearch**

adidxbot
`adidxbot`
Operator: **Microsoft**

AI2Bot
`AI2Bot`
Operator: **Allen AI**

AI2Bot-DeepResearchEval
`AI2Bot-DeepResearchEval`
Operator: **Allen AI**

Ai2Bot-Dolma
`Ai2Bot-Dolma`
Operator: **Allen AI**

aiHitBot
`aiHitBot`
Operator: **aiHit**

AIWebIndex
`AIWebIndex`
Operator: **Lyrenth**

AIWebIndex-Agent
`AIWebIndex-Agent`
Operator: **Lyrenth**

amazon-QBusiness
`amazon-QBusiness`
Operator: **Amazon**

AmazonBuyForMe
`AmazonBuyForMe`
Operator: **Amazon**

Amzn-User
`Amzn-User`
Operator: **Amazon**

Andibot
`Andibot`
Operator: **Andi**

Anomura
`Anomura`
Operator: **Direqt**

anthropic-ai
`anthropic-ai`
Operator: **Anthropic**

ApifyBot
`ApifyBot`
Operator: **Apify**

ApifyWebsiteContentCrawler
`ApifyWebsiteContentCrawler`
Operator: **Apify**

atlassian-bot
`atlassian-bot`
Operator: **Atlassian**

AzureAI-SearchBot
`AzureAI-SearchBot`
Operator: **Microsoft**

bedrockbot
`bedrockbot`
Operator: **Amazon**

Big_Sur_AI
`Big_Sur_AI`
Operator: **Big Sur AI**

BingBot
`BingBot`
Operator: **Microsoft**

brave-search-bot
`brave-search-bot`
Operator: **Brave**

Brightbot
`Brightbot`
Operator: **Bright Data**

Bytespider
`Bytespider`
Operator: **ByteDance**

CCBot
`CCBot`
Operator: **CommonCrawl**

Channel3Bot
`Channel3Bot`
Operator: **Channel3**

ChatGLM-Spider
`ChatGLM-Spider`
Operator: **Zhipu AI**

ChatGPT Agent
`ChatGPT Agent`
Operator: **OpenAI**

ChatGPT Operator
`ChatGPT Operator`
Operator: **OpenAI**

ChatGPT-User
`ChatGPT-User`
Operator: **OpenAI**

Claude-Code
`Claude-Code`
Operator: **Anthropic**

Claude-SearchBot
`Claude-SearchBot`
Operator: **Anthropic**

Claude-User
`Claude-User`
Operator: **Anthropic**

Claude-Web
`Claude-Web`
Operator: **Anthropic**

ClaudeBot
`ClaudeBot`
Operator: **Anthropic**

Cloudflare-AutoRAG
`Cloudflare-AutoRAG`
Operator: **Cloudflare**

Code
`Code`
Operator: **GitHub**

cohere-ai
`cohere-ai`
Operator: **Cohere**

cohere-training-data-crawler
`cohere-training-data-crawler`
Operator: **Cohere**

CohereBot
`CohereBot`
Operator: **Cohere**

CohereForAI
`CohereForAI`
Operator: **Cohere**

CragCrawler
`CragCrawler`
Operator: **CragSoftware**

Cursor
`Cursor`
Operator: **Cursor**

DeepSeekBot
`DeepSeekBot`
Operator: **DeepSeek**

Devin
`Devin`
Operator: **Cognition AI**

Diffbot
`Diffbot`
Operator: **Diffbot**

DigitalOceanGenAICrawler
`DigitalOceanGenAICrawler`
Operator: **DigitalOcean**

Doubaobot
`Doubaobot`
Operator: **ByteDance**

DuckAssistBot
`DuckAssistBot`
Operator: **DuckDuckGo**

ERNIEBot
`ERNIEBot`
Operator: **Baidu**

ExaBot
`ExaBot`
Operator: **Exa AI**

ExaSearchBot
`ExaSearchBot`
Operator: **Exa AI**

FirecrawlAgent
`FirecrawlAgent`
Operator: **Firecrawl**

FriendlyCrawler
`FriendlyCrawler`
Operator: **FriendlyCrawler**

Gemini-Deep-Research
`Gemini-Deep-Research`
Operator: **Google**

Google-Agent
`Google-Agent`
Operator: **Google**

Google-Extended
`Google-Extended`
Operator: **Google**

Google-Firebase
`Google-Firebase`
Operator: **Google**

Google-Gemini-CLI
`Google-Gemini-CLI`
Operator: **Google**

Google-NotebookLM
`Google-NotebookLM`
Operator: **Google**

Google-Safety
`Google-Safety`
Operator: **Google**

GoogleAgent-Mariner
`GoogleAgent-Mariner`
Operator: **Google**

GoogleAgent-URLContext
`GoogleAgent-URLContext`
Operator: **Google**

Googlebot
`Googlebot`
Operator: **Google**

Googlebot-Discovery
`Googlebot-Discovery`
Operator: **Google**

Googlebot-Image
`Googlebot-Image`
Operator: **Google**

Googlebot-News
`Googlebot-News`
Operator: **Google**

Googlebot-Video
`Googlebot-Video`
Operator: **Google**

GoogleOther
`GoogleOther`
Operator: **Google**

GoogleOther-Image
`GoogleOther-Image`
Operator: **Google**

GoogleOther-Video
`GoogleOther-Video`
Operator: **Google**

GPTBot
`GPTBot`
Operator: **OpenAI**

Grok
`Grok`
Operator: **xAI**

GrokBot
`GrokBot`
Operator: **xAI**

HenkBot
`HenkBot`
Operator: **Valyu**

huggingface
`huggingface`
Operator: **HuggingFace**

HuggingFaceBot
`HuggingFaceBot`
Operator: **HuggingFace**

iAskBot
`iAskBot`
Operator: **iAsk.ai**

iaskspider
`iaskspider`
Operator: **iAsk.ai**

imageSpider
`imageSpider`
Operator: **ByteDance**

img2dataset
`img2dataset`
Operator: **LAION**

kagi-fetcher
`kagi-fetcher`
Operator: **Kagi**

KagiBot
`KagiBot`
Operator: **Kagi**

Kimi-SearchBot
`Kimi-SearchBot`
Operator: **Moonshot AI**

Kimi-User
`Kimi-User`
Operator: **Moonshot AI**

KimiBot
`KimiBot`
Operator: **Moonshot AI**

KimiCrawler
`KimiCrawler`
Operator: **Moonshot AI**

KlaviyoAIBot
`KlaviyoAIBot`
Operator: **Klaviyo**

laion-huggingface-processor
`laion-huggingface-processor`
Operator: **LAION**

LINER Bot
`LINER Bot`
Operator: **LINER**

LINER_Bot
`LINER_Bot`
Operator: **LINER**

LinerBot
`LinerBot`
Operator: **LINER**

LinkupBot
`LinkupBot`
Operator: **Linkup**

Magpie-Crawler
`Magpie-Crawler`
Operator: **Magpie**

Manus-User
`Manus-User`
Operator: **Manus**

meta-externalagent
`meta-externalagent`
Operator: **Meta**

Meta-ExternalAgent
`Meta-ExternalAgent`
Operator: **Meta**

meta-webindexer
`meta-webindexer`
Operator: **Meta**

micro-crawl
`micro-crawl`
Operator: **Reflection AI**

mistral-ai
`mistral-ai`
Operator: **Mistral**

MistralAI-Index
`MistralAI-Index`
Operator: **Mistral**

MistralAI-User
`MistralAI-User`
Operator: **Mistral**

MistralBot
`MistralBot`
Operator: **Mistral**

MoonshotBot
`MoonshotBot`
Operator: **Moonshot AI**

MoonSpider
`MoonSpider`
Operator: **Moonshot AI**

Mozilla-Tabstack
`Mozilla-Tabstack`
Operator: **Mozilla**

MyCentralAIScraperBot
`MyCentralAIScraperBot`
Operator: **MyCentral AI**

Neevabot
`Neevabot`
Operator: **Snowflake**

NotebookLM
`NotebookLM`
Operator: **Google**

OAI-AdsBot
`OAI-AdsBot`
Operator: **OpenAI**

OAI-SearchBot
`OAI-SearchBot`
Operator: **OpenAI**

omgili
`omgili`
Operator: **Webz.io**

omgilibot
`omgilibot`
Operator: **Webz.io**

OpenAI
`OpenAI`
Operator: **OpenAI**

openai-searchbot
`openai-searchbot`
Operator: **OpenAI**

opencode
`opencode`
Operator: **Open Source**

Operator
`Operator`
Operator: **OpenAI**

PanguBot
`PanguBot`
Operator: **Huawei**

Perplexity-User
`Perplexity-User`
Operator: **Perplexity**

PerplexityBot
`PerplexityBot`
Operator: **Perplexity**

PhindBot
`PhindBot`
Operator: **Phind**

Poggio-Citations
`Poggio-Citations`
Operator: **Poggio**

QuillBot
`QuillBot`
Operator: **QuillBot**

quillbot.com
`quillbot.com`
Operator: **QuillBot**

QwenBot
`QwenBot`
Operator: **Alibaba**

Reflectionbot
`Reflectionbot`
Operator: **Reflection AI**

SalamandraVLM
`SalamandraVLM`
Operator: **Aithlas**

SBIntuitionsBot
`SBIntuitionsBot`
Operator: **SB Intuitions**

Scrapy
`Scrapy`
Operator: **Scrapy**

Shap-User
`Shap-User`
Operator: **Parallel Web Systems**

TalarionSearchBot
`TalarionSearchBot`
Operator: **Talarion**

tavily-crawler
`tavily-crawler`
Operator: **Tavily**

TavilyBot
`TavilyBot`
Operator: **Tavily**

Timpibot
`Timpibot`
Operator: **Timpi**

TongyiBot
`TongyiBot`
Operator: **Alibaba**

Trae
`Trae`
Operator: **ByteDance**

TurnitinBot
`TurnitinBot`
Operator: **Turnitin**

TwinAgent
`TwinAgent`
Operator: **Twin**

VelenPublicWebCrawler
`VelenPublicWebCrawler`
Operator: **Velen.ai**

webzio-extended
`webzio-extended`
Operator: **Webz.io**

WRTNBot
`WRTNBot`
Operator: **WRTN**

xAI
`xAI`
Operator: **xAI**

xAI-SearchBot
`xAI-SearchBot`
Operator: **xAI**

YepBot
`YepBot`
Operator: **Ahrefs**

YiyanBot
`YiyanBot`
Operator: **Baidu**

YouBot
`YouBot`
Operator: **You.com**

ZanistaBot
`ZanistaBot`
Operator: **Zanista**

**78** Search Engine Bot patterns listed below

360Spider
`360Spider`
Operator: **Qihoo 360**

AdsBot-Google
`AdsBot-Google`
Operator: **Google**

AdsBot-Google-Mobile
`AdsBot-Google-Mobile`
Operator: **Google**

AliyunSecBot
`AliyunSecBot`
Operator: **Alibaba Cloud**

Amazon-Bedrock-AgentCore
`Amazon-Bedrock-AgentCore`
Operator: **Amazon**

Amazonbot
`Amazonbot`
Operator: **Amazon**

Amzn-SearchBot
`Amzn-SearchBot`
Operator: **Amazon**

APIs-Google
`APIs-Google`
Operator: **Google**

Applebot
`Applebot`
Operator: **Apple**

Applebot-Extended
`Applebot-Extended`
Operator: **Apple**

archive.org_bot
`archive.org_bot`
Operator: **Internet Archive**

AspiegelBot
`AspiegelBot`
Operator: **Huawei**

baidu
`baidu`
Operator: **Baidu**

Baiduspider
`Baiduspider`
Operator: **Baidu**

baiduspider-news
`baiduspider-news`
Operator: **Baidu**

Baiduspider-render
`Baiduspider-render`
Operator: **Baidu**

bingbot
`bingbot`
Operator: **Microsoft**

BingPreview
`BingPreview`
Operator: **Microsoft**

Brave Search
`Brave Search`
Operator: **Brave**

Bravebot
`Bravebot`
Operator: **Brave**

Cliqzbot
`Cliqzbot`
Operator: **Cliqz**

coccoc
`coccoc`
Operator: **Coc Coc**

coccocbot
`coccocbot`
Operator: **Coc Coc**

Daum
`Daum`
Operator: **Kakao**

DuckDuckBot
`DuckDuckBot`
Operator: **DuckDuckGo**

DuckDuckGo-Favicons-Bot
`DuckDuckGo-Favicons-Bot`
Operator: **DuckDuckGo**

Exabot
`Exabot`
Operator: **Exalead**

FeedFetcher-Google
`FeedFetcher-Google`
Operator: **Google**

Google Favicon
`Google Favicon`
Operator: **Google**

Google Images
`Google Images`
Operator: **Google**

Google Scholar
`Google Scholar`
Operator: **Google**

Google Videos
`Google Videos`
Operator: **Google**

Google-adstxt
`Google-adstxt`
Operator: **Google**

Google-CloudVertexBot
`Google-CloudVertexBot`
Operator: **Google**

Google-InspectionTool
`Google-InspectionTool`
Operator: **Google**

Google-Read-Aloud
`Google-Read-Aloud`
Operator: **Google**

Google-Site-Verification
`Google-Site-Verification`
Operator: **Google**

Google-Structured-Data-Testing-Tool
`Google-Structured-Data-Testing-Tool`
Operator: **Google**

Googlebot-IA
`Googlebot-IA`
Operator: **Google**

Googlebot-Mobile
`Googlebot-Mobile`
Operator: **Google**

GoogleImageProxy
`GoogleImageProxy`
Operator: **Google**

GoogleProducer
`GoogleProducer`
Operator: **Google**

HaosouSpider
`HaosouSpider`
Operator: **Qihoo 360**

ia_archiver
`ia_archiver`
Operator: **Internet Archive**

JikeSpider
`JikeSpider`
Operator: **Jike**

Mail.RU_Bot
`Mail.RU_Bot`
Operator: **Mail.ru (VK)**

Mediapartners-Google
`Mediapartners-Google`
Operator: **Google**

MegaIndex.ru
`MegaIndex.ru`
Operator: **MegaIndex**

Mojeek
`Mojeek`
Operator: **Mojeek**

MojeekBot
`MojeekBot`
Operator: **Mojeek**

msnbot
`msnbot`
Operator: **Microsoft**

NaverBot
`NaverBot`
Operator: **Naver**

Nutch
`Nutch`
Operator: **Apache Foundation**

PetalBot
`PetalBot`
Operator: **Huawei**

Qwantify
`Qwantify`
Operator: **Qwant**

SeekportBot
`SeekportBot`
Operator: **Seekport**

SeznamBot
`SeznamBot`
Operator: **Seznam**

SeznamHomepageCrawler
`SeznamHomepageCrawler`
Operator: **Seznam**

Slurp
`Slurp`
Operator: **Yahoo**

Sogou
`Sogou`
Operator: **Sogou (Tencent)**

SogouSpider
`SogouSpider`
Operator: **Sogou (Tencent)**

Storebot-Google
`Storebot-Google`
Operator: **Google**

Teoma
`Teoma`
Operator: **Ask.com (IAC)**

Yahoo-Blogs
`Yahoo-Blogs`
Operator: **Yahoo**

Yahoo-FeedSeeker
`Yahoo-FeedSeeker`
Operator: **Yahoo**

Yahoo-MMCrawler
`Yahoo-MMCrawler`
Operator: **Yahoo**

YahooSeeker
`YahooSeeker`
Operator: **Yahoo**

Yandex
`Yandex`
Operator: **Yandex**

YandexAccessibilityBot
`YandexAccessibilityBot`
Operator: **Yandex**

YandexAdditional
`YandexAdditional`
Operator: **Yandex**

YandexAdditionalBot
`YandexAdditionalBot`
Operator: **Yandex**

YandexBot
`YandexBot`
Operator: **Yandex**

YandexImages
`YandexImages`
Operator: **Yandex**

YandexMedia
`YandexMedia`
Operator: **Yandex**

YandexMobileBot
`YandexMobileBot`
Operator: **Yandex**

YandexRenderResourcesBot
`YandexRenderResourcesBot`
Operator: **Yandex**

YandexVideo
`YandexVideo`
Operator: **Yandex**

Yeti
`Yeti`
Operator: **Naver**

**73** Social & SEO Bot patterns listed below

AhrefsBot
`AhrefsBot`
Operator: **Ahrefs**

AhrefsSiteAudit
`AhrefsSiteAudit`
Operator: **Ahrefs**

Algolia
`Algolia`
Operator: **Algolia**

Algolia Crawler
`Algolia Crawler`
Operator: **Algolia**

AudigentAdBot
`AudigentAdBot`
Operator: **Audigent**

AwarioRssBot
`AwarioRssBot`
Operator: **Awario**

AwarioSmartBot
`AwarioSmartBot`
Operator: **Awario**

BacklinkCrawler
`BacklinkCrawler`
Operator: **Ryte**

Barkrowler
`Barkrowler`
Operator: **Babbar**

BLEXBot
`BLEXBot`
Operator: **WebMeUp**

Brandwatch
`Brandwatch`
Operator: **Brandwatch**

Chrome-Lighthouse
`Chrome-Lighthouse`
Operator: **Google**

cognitiveSEO
`cognitiveSEO`
Operator: **cognitiveSEO**

Coveo Bot
`Coveo Bot`
Operator: **Coveo**

Coveobot
`Coveobot`
Operator: **Coveo**

DataForSeoBot
`DataForSeoBot`
Operator: **DataForSEO**

Discordbot
`Discordbot`
Operator: **Discord**

DotBot
`DotBot`
Operator: **Moz**

FacebookBot
`FacebookBot`
Operator: **Meta**

facebookexternalhit
`facebookexternalhit`
Operator: **Meta**

FlipboardProxy
`FlipboardProxy`
Operator: **Flipboard**

Google Page Speed Insights
`Google Page Speed Insights`
Operator: **Google**

GrapeshotCrawler
`GrapeshotCrawler`
Operator: **Oracle (Grapeshot)**

InstagramBot
`InstagramBot`
Operator: **Meta**

Jetslide
`Jetslide`
Operator: **Unknown**

Line
`Line`
Operator: **LINE Corp.**

linkdexbot
`linkdexbot`
Operator: **Linkdex**

LinkedInBot
`LinkedInBot`
Operator: **LinkedIn**

Majestic
`Majestic`
Operator: **Majestic**

MediumBot
`MediumBot`
Operator: **Medium**

Meltwater
`Meltwater`
Operator: **Meltwater**

meta-externalfetcher
`meta-externalfetcher`
Operator: **Meta**

Meta-ExternalFetcher
`Meta-ExternalFetcher`
Operator: **Meta**

MJ12bot
`MJ12bot`
Operator: **Majestic**

NeticleBot
`NeticleBot`
Operator: **Neticle**

Netvibes
`Netvibes`
Operator: **Netvibes**

NinjaCrawler
`NinjaCrawler`
Operator: **Unknown**

OpenLinkProfiler
`OpenLinkProfiler`
Operator: **OpenLinkProfiler**

peer39_crawler
`peer39_crawler`
Operator: **Peer39**

Pinterest
`Pinterest`
Operator: **Pinterest**

Pinterestbot
`Pinterestbot`
Operator: **Pinterest**

proximic
`proximic`
Operator: **Comscore (Proximic)**

Quora-Bot
`Quora-Bot`
Operator: **Quora**

redditbot
`redditbot`
Operator: **Reddit**

rogerbot
`rogerbot`
Operator: **Moz**

Ryte
`Ryte`
Operator: **Ryte**

Screaming Frog SEO Spider
`Screaming Frog SEO Spider`
Operator: **Screaming Frog**

Screaming_Frog_SEO_Spider
`Screaming_Frog_SEO_Spider`
Operator: **Screaming Frog**

SearchmetricsBot
`SearchmetricsBot`
Operator: **Searchmetrics**

SemrushBot
`SemrushBot`
Operator: **Semrush**

SemrushBot-OCOB
`SemrushBot-OCOB`
Operator: **Semrush**

SemrushBotSwa
`SemrushBotSwa`
Operator: **Semrush**

SEOENGWorldBot
`SEOENGWorldBot`
Operator: **Unknown**

SEOkicks
`SEOkicks`
Operator: **SEOkicks**

serpstatbot
`serpstatbot`
Operator: **Serpstat**

Sidetrade indexer bot
`Sidetrade indexer bot`
Operator: **Sidetrade**

Sidetrade Indexer Bot
`Sidetrade Indexer Bot`
Operator: **Sidetrade**

Sistrix
`Sistrix`
Operator: **SISTRIX**

SiteAuditBot
`SiteAuditBot`
Operator: **Semrush**

Slackbot
`Slackbot`
Operator: **Slack**

Snapchat
`Snapchat`
Operator: **Snap Inc.**

spbot
`spbot`
Operator: **SEOprofiler**

SurveyBot
`SurveyBot`
Operator: **Unknown**

TelegramBot
`TelegramBot`
Operator: **Telegram**

TikTokBot
`TikTokBot`
Operator: **ByteDance**

Tumblr
`Tumblr`
Operator: **Automattic**

Twitterbot
`Twitterbot`
Operator: **X (Twitter)**

vkShare
`vkShare`
Operator: **VK**

Wappalyzer
`Wappalyzer`
Operator: **Wappalyzer**

WebMeUp
`WebMeUp`
Operator: **WebMeUp**

WeChat
`WeChat`
Operator: **Tencent**

WhatsApp
`WhatsApp`
Operator: **Meta**

ZoominfoBot
`ZoominfoBot`
Operator: **ZoomInfo**

Research crawlers, archivers, and cloud/security scanners - low SEO/GEO signal either way, so they're grouped as "Other" in your report rather than counted toward AI or Search traffic.

**36** Scraper / Other Bot patterns listed below

Amazon Kendra
`Amazon Kendra`
Operator: **Amazon**

Arquivo-web-crawler
`Arquivo-web-crawler`
Operator: **Unknown**

Bravest
`Bravest`
Operator: **Brave**

Cotoyogi
`Cotoyogi`
Operator: **Unknown**

Crawl4AI
`Crawl4AI`
Operator: **Unknown**

crawler4j
`crawler4j`
Operator: **Unknown**

Crawlspace
`Crawlspace`
Operator: **Unknown**

Datenbank Crawler
`Datenbank Crawler`
Operator: **netEstate**

Echobot Bot
`Echobot Bot`
Operator: **Echobot Media Solutions**

EchoboxBot
`EchoboxBot`
Operator: **Echobox**

Factset_spyderbot
`Factset_spyderbot`
Operator: **FactSet**

Firecrawl
`Firecrawl`
Operator: **Firecrawl**

hada.news
`hada.news`
Operator: **Unknown**

ICC-Crawler
`ICC-Crawler`
Operator: **NICT (Japan)**

ImagesiftBot
`ImagesiftBot`
Operator: **ImageSift**

imediaethics.org
`imediaethics.org`
Operator: **Unknown**

imgproxy
`imgproxy`
Operator: **Unknown**

intelx.io_bot
`intelx.io_bot`
Operator: **Intelligence X**

ISSCyberRiskCrawler
`ISSCyberRiskCrawler`
Operator: **ISS (security)**

JenkersBot
`JenkersBot`
Operator: **Unknown**

Kangaroo Bot
`Kangaroo Bot`
Operator: **Kangaroo LLM**

LivelapBot
`LivelapBot`
Operator: **Unknown**

MauiBot
`MauiBot`
Operator: **Unknown**

MoodleBot
`MoodleBot`
Operator: **Moodle**

netEstate Imprint Crawler
`netEstate Imprint Crawler`
Operator: **netEstate**

news-please
`news-please`
Operator: **Open Source**

NewsNow
`NewsNow`
Operator: **NewsNow**

NovaAct
`NovaAct`
Operator: **Amazon**

Poseidon Research Crawler
`Poseidon Research Crawler`
Operator: **Unknown**

QualifiedBot
`QualifiedBot`
Operator: **Qualified.com**

Seekr
`Seekr`
Operator: **Seekr**

SeekrBot
`SeekrBot`
Operator: **Seekr**

TaraGroup Intelligent Bot
`TaraGroup Intelligent Bot`
Operator: **TaraGroup**

ViennaTinyBot
`ViennaTinyBot`
Operator: **Unknown**

yacy
`yacy`
Operator: **YaCy (Open Source)**

yacybot
`yacybot`
Operator: **YaCy (Open Source)**

## All *Features*

A professional-grade log analyser with 8 analysis tabs, 20+ interactive charts, and PDF export - all running client-side.

AI Bot Detection

- 147 AI bot patterns across 74 AI companies

- Grouped by company & product name

- Per-bot request counts, success rates

- URL-level breakdown by bot

- AI bot share trend over time (daily %)

- Top AI crawler identification

GEO Score™

- Composite score 0–100 (A–F grade)

- 6-component breakdown visualisation

- AI crawl frequency analysis

- Content diversity score

- Crawler diversity score

- AI referral traffic detection

Charts & Visualisations

- Traffic-over-time line chart

- AI bot share trend chart

- Traffic mix bar (AI / Search / Human)

- Response status code doughnut

- Top 10 AI-crawled URLs bar chart

- Crawl heatmap (hour × day of week)

- Weekly trend & AI vs Search comparison

- URL depth analysis & revisit frequency

Search & Other Bots

- Googlebot, Bingbot, Baidu, Yandex etc.

- Search bot activity over time

- SEO tool bots (Ahrefs, Semrush, Moz…)

- Social media scrapers

- Link checkers & archivers

- Browser traffic vs bot separation

Crawl Budget & URL Analysis

- Crawl budget efficiency score

- Error waste ratio (4xx/5xx budget leak)

- Crawl frequency by content type

- URL filter: bot, status, content type

- Avg daily crawl rate per URL

- Redirect chain detection

Reports & Export

- Full PDF report with charts & scores

- Auto-generated recommendations

- robots.txt fix suggestions

- Date range filter (custom from/to)

- 100% browser-side - no data uploaded

- No account or login required

## 20+ *Charts* & Insights

From high-level traffic mix to per-bot URL breakdowns - every chart you need to understand how AI is discovering your content.

📊

Traffic Over Time

Daily request volume split by AI bots, search crawlers, and human visitors - spot crawl spikes instantly.

🥧

Traffic Mix Bar

A proportional bar showing what share of all requests comes from AI, search, other bots and real users.

📈

AI Bot Share Trend

Daily % of traffic from AI bots - rising lines mean more AI platforms are discovering your content.

🔥

Crawl Heatmap

Hour-by-day-of-week heatmap for AI and search bots - see exactly when crawlers hit your server.

🎯

URL Coverage Overlap

Which URLs AI bots and search bots both crawl - and which pages only one side sees.

🔄

Revisit Frequency

How often AI bots return to each URL - "frequent" pages are more likely to feed AI training data.

## Compare: Our Tool *vs* the Paid Alternatives

Screaming Frog costs €99/year and needs a desktop install. JetOctopus starts at $549/month for a cloud plan that ships your logs to their servers. EdgeComet starts at $99/month and requires routing live bot traffic through their DNS/CDN. Ours is **free**, runs entirely in your browser, and needs nothing more than the log file you already have.

| Feature | AI Log Analyzer (This Tool) | Screaming Frog | JetOctopus | EdgeComet |
| --- | --- | --- | --- | --- |
| Price | Free - €0 | €99 / year | From $549 / month | From $99 / month |
| How it works | Drop a log file in your browser | Desktop install (Windows/Mac/Linux) | Upload logs to their cloud platform | Reroute live bot traffic via DNS/CDN |
| Max file size | Up to 1 GB | Depends on RAM | 5–10M log lines/month (paid quota) | N/A - doesn't process log files |
| Max rows | 5 million rows | Limited by licence tier | 5M/mo (Pro) - recurring cap | Billed per Googlebot request instead |
| AI bot detection | 147 AI patterns · 74 AI companies | Basic user-agent matching | ~40 crawler types (per their site) | ~35 bot identities, IP-verified |
| GEO Score™ | Yes - 0–100 with grade | No | Not advertised | Not advertised |
| AI vs Search comparison | Yes - side-by-side bot compare | No | Not advertised | Not advertised |
| Crawl heatmap | Yes - hour × day of week | No | Not advertised | Not advertised |
| Compressed .gz support | Yes - native | Yes | Not specified | N/A - no log upload |
| PDF export | Yes - full report | No (CSV only) | Not advertised | Not advertised |
| Data privacy | 100% browser-side - no upload | Desktop only - no cloud | Logs uploaded to their cloud servers | All bot traffic routed through their infra |
| Recommendations | Yes - with robots.txt fixes | No | General SEO insights, not GEO-specific | Crawl-budget waste flags only |

Competitor pricing and features sourced from their public pricing pages as of Sep 2026 - always double-check current terms directly with each vendor.

## Frequently Asked *Questions*

Everything you need to know before you upload your first log file.

Is this tool really free?

Yes - completely free, with no hidden fees, no account required, and no freemium limits. The entire analysis runs inside your browser using JavaScript. We built this as a free tool to help SEO and GEO professionals who can't justify €99/year for Screaming Frog just for log analysis.

Is my log data private and secure?

Yes. Your log file is never uploaded to any server. All parsing and analysis happens 100% client-side in your browser using JavaScript. When you close the tab, your data is gone. We never see your log data.

What log file formats are supported?

The tool supports **.log**, **.gz** (gzip compressed), **.txt**, **.csv**, and **.tsv** files. It auto-detects Apache Combined Log Format, Nginx access logs, Cloudflare Logpush, and AWS CloudFront formats. For .gz files, decompression happens in the browser using the pako library. You can also select or drop multiple files at once - for example daily or monthly rotated logs - and they'll be merged into a single combined analysis.

How large a log file can I analyse?

The tool is designed to handle files up to **1 GB** and up to **5 million rows** per file. Performance depends on your device's RAM and browser. For very large files (500 MB+), Chrome or Edge on a desktop with 16 GB+ RAM is recommended. Files larger than 1 GB may cause browser memory issues. When uploading multiple files together, the same per-file limits apply to each one.

What is the GEO Score™?

GEO Score™ is a composite 0–100 metric that measures how well your site is being discovered and indexed by AI platforms. It factors in AI crawl volume, crawler diversity (how many different AI bots visit), content diversity (how many different pages AI bots see), error rate for AI bots, revisit frequency, and AI referral traffic. A score above 70 (Grade A) means your site has strong AI visibility.

How many AI bots does the tool track?

We track **147 AI bot user-agent patterns** from **74 AI companies** - including OpenAI (GPTBot, ChatGPT-User, OAI-SearchBot, OAI-AdsBot), Anthropic (ClaudeBot, Claude-Web), Perplexity (PerplexityBot), Google (Google-Extended, Gemini-Deep-Research), xAI (Grok, GrokBot), Meta (Meta-ExternalAgent), Microsoft (BingBot/Copilot), Mistral, DeepSeek, Cohere, ByteDance, HuggingFace, Kagi, Brave, Exa AI, Alibaba (Qwen), Tavily, CommonCrawl, Diffbot, Allen AI and more. In total we track 332 patterns including search engines, social bots and SEO tools. The list is regularly updated as new AI crawlers emerge.

Where do I find my server log file?

It depends on your hosting: **cPanel** - File Manager → logs → access-logs (or /home/username/logs/domain-ssl_log). **Nginx** - /var/log/nginx/access.log. **Apache** - /var/log/apache2/access.log. **Cloudflare** - Analytics → Logs → Logpush (Enterprise). **AWS CloudFront** - Distribution → Logs → S3 Bucket. You can also SSH into your server and run: `gzip -c /var/log/nginx/access.log > access_log.gz`

What is GEO and why does AI crawl analysis matter?

**GEO (Generative Engine Optimisation)** is the practice of optimising your website to appear in AI-generated answers - in ChatGPT, Perplexity, Claude, Gemini and similar tools. Just like Google needs to crawl your site before ranking it, AI platforms need to crawl your content before citing it. Analysing your AI crawler traffic tells you which AI platforms are discovering you, which pages they read, and what you need to fix to improve your AI visibility.

Can I compare AI bots against Google?

Yes - the **Compare Bots** tab lets you select any AI bot and any search bot (Googlebot, Bingbot, etc.) and compare them side-by-side: which URLs each crawls, how much their coverage overlaps, content type breakdown, and the top shared pages. This is useful for diagnosing crawl budget gaps - e.g. Google has crawled a page but GPTBot hasn't.

## Start analysing your *AI traffic* now

Upload your log file above - results in seconds. No account, no install, no cost.

Upload Log File →

100% free · runs in browser · no data uploaded · no account needed

Parsing log file...

[

GEO

Log Analyzer
](#)

GEO

GEO Toolkit 2026
Log Intelligence

[
![Justin Hà](https://justinha.info.vn/wp-content/uploads/2026/08/logo-black.png)
](https://justinha.info.vn/)
/
[GEO Toolkit](#)
/
Log Analyzer

Search crawled URLs...
⌘K

–

Reset

Export Report

New Analysis

GEO Score™

-

Traffic composition

How every request breaks down, out of **-** total requests.

**-**Requests

AI Training-

AI Agent / Search-

Search Engines-

SEO / Social Tools-

Other / Scrapers-

Human Visitors (unmatched)-

Insights

- Top AI Crawler-

- Most Crawled-

- Date Range-

- AI Referrals-

AI Success Rate

2xx/3xx to AI bots

Error Rate

-

Unique URLs

-

Bandwidth

-

Unique IPs

-

-

busiest day

-

-

-

#1 AI visitor on your site

-

-

Traffic over time

Daily visits by type - spikes show unusually high activity worth investigating.

AI bot share trend (%)

What percentage of each day's traffic was AI bots - rising lines mean more AI discovery.

Response status codes

Healthy sites should be mostly green (2xx). Large red or orange slices mean broken pages or server problems.

Top 10 pages crawled by AI

The pages AI bots visit most - these are the parts of your site most likely to appear in AI answers.

AI bot activity over time

By company

AI requests grouped by parent company - short labels, no trend data, so a ranked bar reads faster than a list.

AI crawl heatmap - hour × day of week

When AI bots hit your site the most.

0
Low
Mid
High

💡 Darker cells mean heavier AI bot crawl activity in that hour. Use this to avoid server maintenance during peak hours, or to time new content publishing so AI bots discover it sooner.

AI request outcomes

Successful vs failed requests across all AI bots combined.

**-**Success

Succeeded (2xx/3xx)-

Failed (4xx/5xx)-

Crawl efficiency - requests vs. unique URLs

Top 10 bots by volume. Bots high on the X-axis but low on Y keep re-fetching the same pages - wasted crawl budget. Bots further right on Y discover more of your site per visit.

Content type crawled - by bot

Top 5 bots by volume. Bots skewed toward HTML/Document are reading your actual content; bots skewed toward Script/Image/API are mostly fetching resources, not GEO-relevant pages.

AI bot details

Company

All companies

Search bot / product

Reset

URL analysis

Human Visits lets you spot AI-heavy/human-cold pages (possible over-indexing) or human-favorite pages AI is ignoring (a GEO gap worth closing).

Content type

All
HTML
Image
Script
Document
API
Media

Company

All companies

Filter by bot

All AI bots

Status

All
2xx OK
3xx Redirect
4xx Errors
5xx Errors

Search URL

Reset

## Search Engine *Crawlers*

Search Requests

-

Bot Types

-

Search Share

-

URLs Crawled

-

Search bot activity over time

Top 5 search engines by volume.

By search engine

Short labels, no trend per entry - a ranked bar reads faster than a list.

Search bot crawl heatmap - hour × day of week

0
Low
Mid
High

Search bot breakdown

Search engine

All engines

Search bot name

Reset

Top pages by search bot

Bot

All bots

Content type

All
HTML
Image
Script
Document
API
Media

Status

All
2xx OK
3xx Redirect
4xx Errors
5xx Errors

Search URL

Reset

## Other *Bots* - SEO tools · social media scrapers · link checkers · archivers

Other Bot Requests

-

Bot Types

-

Bot Share

-

URLs Crawled

-

Other bot activity over time

Top 5 categories - SEO tools, social scrapers, link checkers, archivers.

By category

Short labels, no trend per entry - a ranked bar reads faster than a list.

Other bot crawl heatmap - hour × day of week

SEO tools, social scrapers, link checkers and archivers - when they hit your site.

0
Low
Mid
High

Other bot breakdown

Category

All categories

Search bot name

Reset

Top pages by other bots

Bot

All bots

Content type

All
HTML
Image
Script
Document
API
Media

Status

All
2xx OK
3xx Redirect
4xx Errors
5xx Errors

Search URL

Reset

## Browser *Traffic* - requests not matching any known AI/search/other-bot pattern and not an AI-assistant referral. Likely real visitors, but may include unlisted bots.

ℹ️ AI-assistant click-throughs (someone clicking a link inside a ChatGPT/Claude/Perplexity/Gemini answer) are tracked separately as **AI Referrals** in the Overview tab's Insights panel - they are not counted here, since they're attributed to AI discovery rather than organic browser traffic.

Browser Requests

-

Unique IPs

-

Browser Share

-

Top Source

-

Browser traffic over time

Daily real-visitor requests - spikes worth cross-checking against a campaign or referral spike.

Device types

Desktop vs Mobile vs Tablet share of human visits.

**-**Visits

Desktop-

Mobile-

Tablet-

Top 10 pages by browser traffic

The pages real visitors read most.

Traffic sources

Referrer breakdown - where human visits come from, including AI-assistant referrals.

Browser response status codes

Healthy sites should be mostly green (2xx). Large red/orange slices mean broken pages for real visitors.

**-**2xx OK

2xx OK-

3xx Redirect-

4xx Error-

5xx Error-

Browser activity heatmap - hour × day

0
Low
Mid
High

Client / bot breakdown

"Browser Traffic" is a catch-all for anything not matching a known bot signature - this breaks it down by client so unlisted bots hiding behind a script/library User-Agent stand out from real browsers.

Top pages by browser traffic

Content type

All
HTML
Image
Script
Document
API
Media

Status

All
2xx OK
3xx Redirect
4xx Errors
5xx Errors

Search URL

Reset

## AI Bot *vs* Search Bot

Select one bot from each side to compare which URLs they crawl, how much they overlap, and where each focuses its budget. Googlebot and Bingbot appear on the Search side.

AI Bot

Select AI Bot…

vs

Search Bot

Select Search Bot…

URL Coverage Overlap

Each bot's own coverage as its own 100% - a fair 50/50 comparison regardless of which bot crawls more in raw volume.

Content Type Focus

Share of each bot's *own* crawl budget per content type - normalised, so a high-volume bot doesn't drown out the comparison.

Crawl activity over time

Daily requests for the two selected bots - reveals whether one started, stalled, or spiked independently of the other.

Top Shared URLs - Side-by-Side Crawl Count

URLs visited by both bots, ranked by combined activity.

URL coverage comparison

Filter to **Search only** to list pages the search crawler indexes but the AI bot has never fetched - these are your AI blind spots.

Coverage

All URLs
Both bots
AI only
Search only

Content type

All
HTML
Image
Script
Document
API
Media

Search URL

Reset

GEO Score™ component breakdown

The radar shows the score's shape - a lopsided pentagon points straight at which factor is dragging the total down.

**-**/ 100

Weekly traffic trend (AI / Search / Users)

Week-over-week volume - smooths out daily noise to show whether AI discovery is actually growing.

AI crawl by content section (top 10 path segments)

Section-level GEO health

Top 15 sections by AI request volume - success rate and how many distinct AI companies actually crawl each one.

URL coverage - AI vs Search overlap

Site-wide totals across every AI bot and every search bot. For a specific pair (e.g. GPTBot vs Googlebot), use the Compare Bots tab. For content-type mix, see the Crawl Budget tab.

URL depth vs. success rate

Bars = AI request volume by folder depth; line = success rate at that depth - reveals whether deep pages are harder for AI bots to fetch.

AI crawl budget (daily)

Red = above this site's own baseline.

URL revisit frequency - how often AI returns

**Once** = crawled on a single day · **Recurring** = 2–5 distinct days · **Frequent** = 6+ days

Most revisited URLs by AI bots

Pages AI bots return to on many distinct days - usually your highest-priority content, or evergreen pages worth keeping fresh.

Content type

All
HTML
Image
Script
Document
API
Media

Search URL

Reset

## Crawl *Budget* Analysis

Understand how AI crawlers spend their budget on your site - identify waste, protect content pages, and ensure your most important URLs get indexed.

-

Total AI Crawl Requests

-

-

Avg AI Crawls per Day

-

-

HTML Content Rate

-

% of AI requests hitting content pages

-

Wasted Budget

-

errors + non-HTML assets crawled

AI Bandwidth Allocation

-

-
-
-

Query parameter waste

AI budget spent on URLs with a "?" (filters, sorting, tracking params, session IDs) - usually duplicate content that should be blocked in robots.txt.

**-**Parameterized

Clean URLs-

Parameterized URLs-

Top parameters to consider blocking

Content Type Distribution

What content types are consuming the AI crawl budget - HTML and Document carry indexing value; Script/Image/Media/API don't.

**-**HTML

HTML-

Image-

Script-

Document-

API-

Media-

AI Page Health by URL

Each crawled URL's dominant status code - large red/orange slices mean budget spent on broken or inaccessible pages.

**-**2xx OK

2xx OK-

3xx Redirect-

4xx Error-

5xx Error-

Top wasted crawl targets

URLs where AI bots repeatedly hit an error - fix or 410/redirect these, or block them in robots.txt to reclaim budget.

Bandwidth by bot

Top 12 bots by data transferred - a bot with fewer requests but more bytes is fetching heavier pages/assets, not just visiting more often.

Underinvested pages - high human interest, low AI attention

Pages your visitors actually read that AI bots rarely or never crawl - internal-link them more, add them to your sitemap, or ping IndexNow to close the gap.

URL Crawl Budget Breakdown

Bot

All AI Bots

Content Type

All
HTML
Image
Script
Document
API
Media

Status

All
2xx OK
3xx Redirect
4xx Errors
5xx Errors

Search URL

Reset

Full Dashboard Report

Complete analysis across all tabs - download as a multi-page PDF with exact design.

Download PDF Report

Built for [justinha.info.vn](https://justinha.info.vn) · GEO Log Analyzer · AI-powered access log analysis · A [Luminal](https://www.linkedin.com/company/luminal-geo/) tool

ESC

Start typing to search crawled URLs...

Generating PDF Report…

Preparing…