Free SEO Log File Analyzer Tool
Upload your server access log to discover how 147 AI crawlers - GPTBot, ClaudeBot, PerplexityBot, Grok, DeepSeek and more - are indexing your site. Get your GEO Score™ in seconds. 100% free, runs in your browser.
/home/username/logs/domain-ssl_log
Enterprise or via API
s3://your-bucket/cf-logs/
| gzip > access_log.gz
Why use our free tool?
Everything Screaming Frog's Log File Analyser offers - and much more AI intelligence - at zero cost. No install, no account, no data sent to any server.
100% Free Forever
Screaming Frog Log File Analyser costs €99/year. Our tool is completely free - no licence, no subscription, no paywall.
↓ Save €99 vs Screaming FrogUp to 1 GB & 5 Million Rows
Handle large enterprise log files up to 1 GB per file, with up to 5 million rows - all processed locally inside your browser. Upload multiple files at once (e.g. daily or monthly rotated logs) and they're combined into one analysis. No upload limits.
147 AI Bots · 74 Companies
Detect every major AI crawler: OpenAI, Anthropic, Google, Perplexity, xAI, Meta, Mistral, DeepSeek, Cohere, Kagi, Brave, Exa, Qwen and more - by name and product.
Native .log & .gz Support
Drop in raw .log, compressed .gz, .txt, .csv or .tsv - no conversion needed. Your data never leaves your device.
How it works
Analyse millions of log rows in seconds - completely in your browser. No data leaves your device.
Upload Your Log File(s)
Drag and drop or browse for any server access log - Apache, Nginx, Cloudflare, AWS, cPanel. Supports .log, .gz, .txt, .csv up to 1 GB per file. Select several files at once to combine daily or monthly rotated logs into one analysis.
Instant Analysis
Our browser-side engine parses every row, identifies 147 AI bot patterns across 74 AI companies, calculates your GEO Score™, builds 20+ charts and surfaces key recommendations.
Export Your Report
Download a full PDF report with charts, scores and action items - ready to share with clients or your team. No account required.
Why GEO & AI SEO require log file analysis
Log analysis has always been a technical SEO skill. In the GEO era, it becomes indispensable - because AI platforms have no equivalent of Google Search Console. Your server logs are the only source of truth.
AI search has no Search Console.
Google gives you Search Console - impressions, clicks, index coverage, crawl stats. ChatGPT, Perplexity, Claude, Grok and every other AI platform give you nothing. No dashboard. No reports. No API. Your server access log is the only place you can see whether GPTBot, ClaudeBot or PerplexityBot has ever visited your site - and what happened when they did. If you're not reading your logs, you're flying blind in the most important new channel in search.
The AI discovery pipeline
Before an AI platform can cite your content in an answer, it must first crawl it. Log analysis is the only way to verify where your content sits in this pipeline.
AI Bot Crawls Your Page
GPTBot, ClaudeBot or PerplexityBot visits your URL. Visible only in your server logs. If blocked by robots.txt, WAF, or rate limiting - nothing else in the pipeline fires.
Content Gets Indexed / Trained
The AI platform processes your content into its knowledge base or live index. Log frequency, recrawl rate and status codes all influence how much of your content is absorbed.
AI Cites Your Content
Your brand, article or expertise surfaces in ChatGPT, Perplexity or Claude answers. This is the GEO goal - and it starts at Step 01. No crawl = no citation.
What only your log file can tell you
No third-party tool, no rank tracker, no AI visibility monitor can give you this data. It lives only on your server.
Which AI bots are actually crawling you - right now
Not estimates. Not averages. Real bot names, real timestamps, real IPs. You'll know if GPTBot crawled your site yesterday, which 14 pages it read, and whether it got a 200 or a 403.
Whether your robots.txt rules are working for AI bots
You might have Allow: / for GPTBot - but is it actually crawling? Or is a WAF rule, IP block or rate limit silently rejecting every request? Logs expose the truth robots.txt validation tools can't see.
Which pages AI bots ignore - your GEO coverage gap
Google might index 3,000 of your pages. GPTBot might only be reading 12. The pages AI bots never visit are your GEO dead zones - no crawl means no training data means no citations. Logs are the only way to find them.
How AI crawler behaviour is changing over time
Is PerplexityBot crawling you more this month than last? Did ClaudeBot suddenly stop after a server change? Trend data from logs lets you correlate technical events with changes in AI crawl activity - before it hits your citation rate.
The real cost of errors to your AI visibility
Every 404 or 500 that an AI bot hits wastes its crawl budget on your site and teaches it your content is unreliable. Logs quantify exactly how much of your AI crawl budget is being destroyed by errors - and on which pages.
AI referral traffic - proof that GEO is working
When someone clicks a citation in Perplexity or a ChatGPT link and lands on your site, that referral appears in your logs. It's the clearest signal that your GEO efforts are converting into real traffic.
Traditional SEO vs GEO: a new diagnostic skill
Log analysis existed in technical SEO - but GEO makes it a front-line skill, not a specialist one. Here's how the mindset shifts.
- Check Google Search Console for coverage
- Monitor Googlebot crawl stats dashboard
- Fix 404s to protect Google crawl budget
- Optimise for one primary crawler (Googlebot)
- Log files = occasional advanced audit
- Rank tracking tells you if SEO is working
- No Search Console exists - logs are your only data
- Monitor 147 AI bot patterns across 74 AI companies
- Fix errors to protect crawl budget across all AI platforms
- Optimise for 332 total bot patterns simultaneously
- Log files = weekly core GEO health check
- AI referral traffic & citation tracking shows GEO ROI
Which bots are tracked?
332 user-agent patterns across every category - AI crawlers, search engines, social media bots and SEO tools. All detected and classified automatically.
AddSearchBotadidxbotAI2BotAI2Bot-DeepResearchEvalAi2Bot-DolmaaiHitBotAIWebIndexAIWebIndex-Agentamazon-QBusinessAmazonBuyForMeAmzn-UserAndibotAnomuraanthropic-aiApifyBotApifyWebsiteContentCrawleratlassian-botAzureAI-SearchBotbedrockbotBig_Sur_AIBingBotbrave-search-botBrightbotBytespiderCCBotChannel3BotChatGLM-SpiderChatGPT AgentChatGPT OperatorChatGPT-UserClaude-CodeClaude-SearchBotClaude-UserClaude-WebClaudeBotCloudflare-AutoRAGCodecohere-aicohere-training-data-crawlerCohereBotCohereForAICragCrawlerCursorDeepSeekBotDevinDiffbotDigitalOceanGenAICrawlerDoubaobotDuckAssistBotERNIEBotExaBotExaSearchBotFirecrawlAgentFriendlyCrawlerGemini-Deep-ResearchGoogle-AgentGoogle-ExtendedGoogle-FirebaseGoogle-Gemini-CLIGoogle-NotebookLMGoogle-SafetyGoogleAgent-MarinerGoogleAgent-URLContextGooglebotGooglebot-DiscoveryGooglebot-ImageGooglebot-NewsGooglebot-VideoGoogleOtherGoogleOther-ImageGoogleOther-VideoGPTBotGrokGrokBotHenkBothuggingfaceHuggingFaceBotiAskBotiaskspiderimageSpiderimg2datasetkagi-fetcherKagiBotKimi-SearchBotKimi-UserKimiBotKimiCrawlerKlaviyoAIBotlaion-huggingface-processorLINER BotLINER_BotLinerBotLinkupBotMagpie-CrawlerManus-Usermeta-externalagentMeta-ExternalAgentmeta-webindexermicro-crawlmistral-aiMistralAI-IndexMistralAI-UserMistralBotMoonshotBotMoonSpiderMozilla-TabstackMyCentralAIScraperBotNeevabotNotebookLMOAI-AdsBotOAI-SearchBotomgiliomgilibotOpenAIopenai-searchbotopencodeOperatorPanguBotPerplexity-UserPerplexityBotPhindBotPoggio-CitationsQuillBotquillbot.comQwenBotReflectionbotSalamandraVLMSBIntuitionsBotScrapyShap-UserTalarionSearchBottavily-crawlerTavilyBotTimpibotTongyiBotTraeTurnitinBotTwinAgentVelenPublicWebCrawlerwebzio-extendedWRTNBotxAIxAI-SearchBotYepBotYiyanBotYouBotZanistaBot360SpiderAdsBot-GoogleAdsBot-Google-MobileAliyunSecBotAmazon-Bedrock-AgentCoreAmazonbotAmzn-SearchBotAPIs-GoogleApplebotApplebot-Extendedarchive.org_botAspiegelBotbaiduBaiduspiderbaiduspider-newsBaiduspider-renderbingbotBingPreviewBrave SearchBravebotCliqzbotcoccoccoccocbotDaumDuckDuckBotDuckDuckGo-Favicons-BotExabotFeedFetcher-GoogleGoogle FaviconGoogle ImagesGoogle ScholarGoogle VideosGoogle-adstxtGoogle-CloudVertexBotGoogle-InspectionToolGoogle-Read-AloudGoogle-Site-VerificationGoogle-Structured-Data-Testing-ToolGooglebot-IAGooglebot-MobileGoogleImageProxyGoogleProducerHaosouSpideria_archiverJikeSpiderMail.RU_BotMediapartners-GoogleMegaIndex.ruMojeekMojeekBotmsnbotNaverBotNutchPetalBotQwantifySeekportBotSeznamBotSeznamHomepageCrawlerSlurpSogouSogouSpiderStorebot-GoogleTeomaYahoo-BlogsYahoo-FeedSeekerYahoo-MMCrawlerYahooSeekerYandexYandexAccessibilityBotYandexAdditionalYandexAdditionalBotYandexBotYandexImagesYandexMediaYandexMobileBotYandexRenderResourcesBotYandexVideoYetiAhrefsBotAhrefsSiteAuditAlgoliaAlgolia CrawlerAudigentAdBotAwarioRssBotAwarioSmartBotBacklinkCrawlerBarkrowlerBLEXBotBrandwatchChrome-LighthousecognitiveSEOCoveo BotCoveobotDataForSeoBotDiscordbotDotBotFacebookBotfacebookexternalhitFlipboardProxyGoogle Page Speed InsightsGrapeshotCrawlerInstagramBotJetslideLinelinkdexbotLinkedInBotMajesticMediumBotMeltwatermeta-externalfetcherMeta-ExternalFetcherMJ12botNeticleBotNetvibesNinjaCrawlerOpenLinkProfilerpeer39_crawlerPinterestPinterestbotproximicQuora-BotredditbotrogerbotRyteScreaming Frog SEO SpiderScreaming_Frog_SEO_SpiderSearchmetricsBotSemrushBotSemrushBot-OCOBSemrushBotSwaSEOENGWorldBotSEOkicksserpstatbotSidetrade indexer botSidetrade Indexer BotSistrixSiteAuditBotSlackbotSnapchatspbotSurveyBotTelegramBotTikTokBotTumblrTwitterbotvkShareWappalyzerWebMeUpWeChatWhatsAppZoominfoBotAmazon KendraArquivo-web-crawlerBravestCotoyogiCrawl4AIcrawler4jCrawlspaceDatenbank CrawlerEchobot BotEchoboxBotFactset_spyderbotFirecrawlhada.newsICC-CrawlerImagesiftBotimediaethics.orgimgproxyintelx.io_botISSCyberRiskCrawlerJenkersBotKangaroo BotLivelapBotMauiBotMoodleBotnetEstate Imprint Crawlernews-pleaseNewsNowNovaActPoseidon Research CrawlerQualifiedBotSeekrSeekrBotTaraGroup Intelligent BotViennaTinyBotyacyyacybotAll Features
A professional-grade log analyser with 8 analysis tabs, 20+ interactive charts, and PDF export - all running client-side.
- 147 AI bot patterns across 74 AI companies
- Grouped by company & product name
- Per-bot request counts, success rates
- URL-level breakdown by bot
- AI bot share trend over time (daily %)
- Top AI crawler identification
- Composite score 0–100 (A–F grade)
- 6-component breakdown visualisation
- AI crawl frequency analysis
- Content diversity score
- Crawler diversity score
- AI referral traffic detection
- Traffic-over-time line chart
- AI bot share trend chart
- Traffic mix bar (AI / Search / Human)
- Response status code doughnut
- Top 10 AI-crawled URLs bar chart
- Crawl heatmap (hour × day of week)
- Weekly trend & AI vs Search comparison
- URL depth analysis & revisit frequency
- Googlebot, Bingbot, Baidu, Yandex etc.
- Search bot activity over time
- SEO tool bots (Ahrefs, Semrush, Moz…)
- Social media scrapers
- Link checkers & archivers
- Browser traffic vs bot separation
- Crawl budget efficiency score
- Error waste ratio (4xx/5xx budget leak)
- Crawl frequency by content type
- URL filter: bot, status, content type
- Avg daily crawl rate per URL
- Redirect chain detection
- Full PDF report with charts & scores
- Auto-generated recommendations
- robots.txt fix suggestions
- Date range filter (custom from/to)
- 100% browser-side - no data uploaded
- No account or login required
20+ Charts & Insights
From high-level traffic mix to per-bot URL breakdowns - every chart you need to understand how AI is discovering your content.
Compare: Our Tool vs the Paid Alternatives
Screaming Frog costs €99/year and needs a desktop install. JetOctopus starts at $549/month for a cloud plan that ships your logs to their servers. EdgeComet starts at $99/month and requires routing live bot traffic through their DNS/CDN. Ours is free, runs entirely in your browser, and needs nothing more than the log file you already have.
| Feature | AI Log Analyzer (This Tool) | Screaming Frog | JetOctopus | EdgeComet |
|---|---|---|---|---|
| Price | Free - €0 | €99 / year | From $549 / month | From $99 / month |
| How it works | Drop a log file in your browser | Desktop install (Windows/Mac/Linux) | Upload logs to their cloud platform | Reroute live bot traffic via DNS/CDN |
| Max file size | Up to 1 GB | Depends on RAM | 5–10M log lines/month (paid quota) | N/A - doesn't process log files |
| Max rows | 5 million rows | Limited by licence tier | 5M/mo (Pro) - recurring cap | Billed per Googlebot request instead |
| AI bot detection | 147 AI patterns · 74 AI companies | Basic user-agent matching | ~40 crawler types (per their site) | ~35 bot identities, IP-verified |
| GEO Score™ | Yes - 0–100 with grade | No | Not advertised | Not advertised |
| AI vs Search comparison | Yes - side-by-side bot compare | No | Not advertised | Not advertised |
| Crawl heatmap | Yes - hour × day of week | No | Not advertised | Not advertised |
| Compressed .gz support | Yes - native | Yes | Not specified | N/A - no log upload |
| PDF export | Yes - full report | No (CSV only) | Not advertised | Not advertised |
| Data privacy | 100% browser-side - no upload | Desktop only - no cloud | Logs uploaded to their cloud servers | All bot traffic routed through their infra |
| Recommendations | Yes - with robots.txt fixes | No | General SEO insights, not GEO-specific | Crawl-budget waste flags only |
Competitor pricing and features sourced from their public pricing pages as of Sep 2026 - always double-check current terms directly with each vendor.
Frequently Asked Questions
Everything you need to know before you upload your first log file.
Yes - completely free, with no hidden fees, no account required, and no freemium limits. The entire analysis runs inside your browser using JavaScript. We built this as a free tool to help SEO and GEO professionals who can't justify €99/year for Screaming Frog just for log analysis.
Yes. Your log file is never uploaded to any server. All parsing and analysis happens 100% client-side in your browser using JavaScript. When you close the tab, your data is gone. We never see your log data.
The tool supports .log, .gz (gzip compressed), .txt, .csv, and .tsv files. It auto-detects Apache Combined Log Format, Nginx access logs, Cloudflare Logpush, and AWS CloudFront formats. For .gz files, decompression happens in the browser using the pako library. You can also select or drop multiple files at once - for example daily or monthly rotated logs - and they'll be merged into a single combined analysis.
The tool is designed to handle files up to 1 GB and up to 5 million rows per file. Performance depends on your device's RAM and browser. For very large files (500 MB+), Chrome or Edge on a desktop with 16 GB+ RAM is recommended. Files larger than 1 GB may cause browser memory issues. When uploading multiple files together, the same per-file limits apply to each one.
GEO Score™ is a composite 0–100 metric that measures how well your site is being discovered and indexed by AI platforms. It factors in AI crawl volume, crawler diversity (how many different AI bots visit), content diversity (how many different pages AI bots see), error rate for AI bots, revisit frequency, and AI referral traffic. A score above 70 (Grade A) means your site has strong AI visibility.
We track 147 AI bot user-agent patterns from 74 AI companies - including OpenAI (GPTBot, ChatGPT-User, OAI-SearchBot, OAI-AdsBot), Anthropic (ClaudeBot, Claude-Web), Perplexity (PerplexityBot), Google (Google-Extended, Gemini-Deep-Research), xAI (Grok, GrokBot), Meta (Meta-ExternalAgent), Microsoft (BingBot/Copilot), Mistral, DeepSeek, Cohere, ByteDance, HuggingFace, Kagi, Brave, Exa AI, Alibaba (Qwen), Tavily, CommonCrawl, Diffbot, Allen AI and more. In total we track 332 patterns including search engines, social bots and SEO tools. The list is regularly updated as new AI crawlers emerge.
It depends on your hosting: cPanel - File Manager → logs → access-logs (or /home/username/logs/domain-ssl_log). Nginx - /var/log/nginx/access.log. Apache - /var/log/apache2/access.log. Cloudflare - Analytics → Logs → Logpush (Enterprise). AWS CloudFront - Distribution → Logs → S3 Bucket. You can also SSH into your server and run: gzip -c /var/log/nginx/access.log > access_log.gz
GEO (Generative Engine Optimisation) is the practice of optimising your website to appear in AI-generated answers - in ChatGPT, Perplexity, Claude, Gemini and similar tools. Just like Google needs to crawl your site before ranking it, AI platforms need to crawl your content before citing it. Analysing your AI crawler traffic tells you which AI platforms are discovering you, which pages they read, and what you need to fix to improve your AI visibility.
Yes - the Compare Bots tab lets you select any AI bot and any search bot (Googlebot, Bingbot, etc.) and compare them side-by-side: which URLs each crawls, how much their coverage overlaps, content type breakdown, and the top shared pages. This is useful for diagnosing crawl budget gaps - e.g. Google has crawled a page but GPTBot hasn't.
Start analysing your AI traffic now
Upload your log file above - results in seconds. No account, no install, no cost.
- Top AI Crawler-
- Most Crawled-
- Date Range-
- AI Referrals-
| Bot | Company | Product | Requests | Share | URLs | Revisit Ratio | 14-day Trend | Last Seen | Success Rate |
|---|
| URL | Type | AI Requests | Human Visits | Avg/Day | Status |
|---|
Search Engine Crawlers
| Bot | Search Engine | Requests | Share | URLs | Revisit Ratio | 14-day Trend | Last Seen | Success Rate |
|---|
| URL | Type | Search Requests | Human Visits | Avg/Day | Status |
|---|
Other Bots - SEO tools · social media scrapers · link checkers · archivers
| Bot | Category | Requests | Share | URLs | Revisit Ratio | 14-day Trend | Last Seen | Success Rate |
|---|
| URL | Type | Bot Requests | Human Visits | Avg/Day | Status |
|---|
Browser Traffic - requests not matching any known AI/search/other-bot pattern and not an AI-assistant referral. Likely real visitors, but may include unlisted bots.
ℹ️ AI-assistant click-throughs (someone clicking a link inside a ChatGPT/Claude/Perplexity/Gemini answer) are tracked separately as AI Referrals in the Overview tab's Insights panel - they are not counted here, since they're attributed to AI discovery rather than organic browser traffic.
| Client | Requests | Share | Type |
|---|
| URL | Type | Browser Requests | Avg/Day | Unique IPs | Status |
|---|
AI Bot vs Search Bot
Select one bot from each side to compare which URLs they crawl, how much they overlap, and where each focuses its budget. Googlebot and Bingbot appear on the Search side.
| URL | Type | AI Bot | Search Bot | Coverage |
|---|
| Section | AI Requests | Success Rate | AI Companies |
|---|
| URL | Days Crawled | Type |
|---|
Crawl Budget Analysis
Understand how AI crawlers spend their budget on your site - identify waste, protect content pages, and ensure your most important URLs get indexed.
| URL | Type | Wasted Requests | Error Rate | Total Requests |
|---|
| URL | Type | Human Visits | AI Requests | Gap |
|---|
| URL | Type | Requests | % of Budget | Avg/Day | Crawled By | Status |
|---|
