I have built Scrawly, a Free, Open-Source Screaming Frog Alternative

Agency clients often ask a question legacy crawlers cannot answer: "We rank on Google, so why does ChatGPT never mention us?"

Often, the blocker is invisible: robots.txt allows AI crawlers, but a CDN or firewall silently blocks them. Traditional crawlers only inspect the file directives; they do not test real-time AI user-agent responses.

I built Scrawly to bridge that gap: a free, open-source (MIT License, v1.0.0) desktop crawler for macOS, Windows, and Linux. It audits the technical signals governing Google rankings alongside visibility in AI engines like ChatGPT, Claude, Perplexity, and Google AI Overviews. There are no subscriptions, account walls, or URL caps—all audit data remains stored in a local SQLite database on your machine.

Scrawly is a local-first, open-source technical SEO crawler and audit platform engineered without artificial URL limits or licensing paywalls. Built from the ground up as a robust free screaming frog alternative software, Scrawly provides search professionals and technical architects with enterprise-grade crawl diagnostics, JavaScript rendering evaluation, and site architecture mapping right on their local machine. Beyond replacing traditional desktop crawlers, it functions as a modern, free Sitebulb alternative, pairing comprehensive technical issue categorization with AI-readiness auditing and native Model Context Protocol (MCP) integration. For teams seeking a transparent, community-driven screaming frog alternative, Scrawly delivers full crawl data sovereignty without the enterprise subscription tag.

250 Technical Checks Across 23 Categories

Scrawly evaluates 250 distinct parameters using Playwright Chromium rendering:

Crawlability & Indexability (15): Robots blocks, noindex tags/headers, orphan pages, click depth > 4.
Status Codes & Redirects (14): 4xx/5xx responses, soft 404s, broken links, redirect loops/chains, 302 vs 301.
Canonicals (11): Missing, relative, conflicting, or dynamic post-render canonical shifts.
Titles, Meta & Headings (21): Pixel-width truncation, thin descriptions, duplicate H1s, Open Graph.
Content Quality (12): Exact/near duplicates, keyword cannibalization, placeholder copy, missing dates.
Internal Architecture (12): Link equity distribution, single-inlink pages, generic anchor text.
External Links (6): Broken outbound links, insecure HTTP, missing rel="sponsored" or ugc.
Images & Media (11): Missing alt text, files > 100 KB, missing dimensions, WebP/AVIF adoption.
Hreflang (8): Missing return tags, invalid language codes, x-default, header/sitemap conflicts.
Structured Data (15): JSON-LD validation, @id conflicts, Organization, Article, Product, and FAQ schema.
XML Sitemaps (10): Format errors, non-200 URLs listed, file size limits, stale lastmod.
Robots Directives (7): Site-wide disallows, blocked render assets, wildcard errors, high crawl delays.
Security (10): Mixed content, HSTS configuration, expiring SSL certs, server version leaks.
Core Web Vitals & Speed (14): LCP (>2.5s), INP (>200ms), CLS (>0.1), TTFB, DOM node size.
Mobile (7): Viewport setup, tap targets, horizontal scrolling, desktop/mobile parity.
JavaScript Rendering (6): DOM drift, client-only links, post-render metadata alterations.
Accessibility (9): WCAG elements impacting SEO and AI agents (ARIA, landmarks, color contrast).
AI Search & GEO (13): Silent bot blocking, client-side content barriers, entity markup, answer structure.
Agentic Web & llms.txt (13): /llms.txt quality (<800 links; 20–50 ideal), agent navigation discovery.
URL Hygiene (9): Uppercase characters, unsafe strings, tracking parameters, URLs > 115 chars.
Pagination (5): Page 2+ noindex/canonical traps, infinite scroll URL accessibility.
WordPress & HTML (17): Thin tag archives, staging leaks, dual Yoast/Rank Math conflicts, DOCTYPE.
Analytics & Config (5): GA4/GTM tagging, Search Console verification, favicon, 404 templates.

GEO & Agentic Audits

AI Crawler Matrix: Tests 10 bots (Googlebot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, CCBot, Google-Extended, bingbot) via live requests to uncover firewall blocks. Differentiates AI training crawlers (flagged as Info) from AI search bots (flagged as High).
Agentic Readiness Probes: Runs 21 targeted root probes (~25 total requests) scoring sites across Levels 0–5 on Markdown negotiation, OAuth discovery, MCP cards, A2A agent cards, and agentic commerce (x402, ACP).

Workflow, Prioritization & MCP

Intelligent Scoring: Prioritizes issues (Critical to Info) using coverage-aware rules (escalates when issues affect 20%+ of pages; prevents double-counting). Includes a PageRank-style Internal Authority score (0–100).
Integrations: Connects directly with Google Search Console, GA4, PageSpeed Insights, and CrUX.
Reversible WordPress Fixes: A lightweight (6 KB) plugin safely fixes titles, descriptions, canonicals, and 301 redirects with automated rollback snapshots.
Native MCP Server: Exposes 11 tools (list_crawls, get_issues, get_page_detail, propose_fix, diff_crawls, check_ai_access, run_lighthouse, generate_report, apply_fix, create_redirect, revert) so Claude Desktop, Claude Code, and Cursor can inspect audits and patch code directly.

Installation

Requires Python 3.11+ and Git (installed automatically if missing):

macOS / Linux:
curl -fsSL [https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.sh](https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.sh) | bash
Windows (PowerShell):
irm [https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.ps1](https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.ps1) | iex
(Ubuntu/Debian UI libs: sudo apt install libxkbcommon-x11-0 libxcb-cursor0 libnss3 libgbm1)

Open-Source Alternative to Screaming Frog & Sitebulb

I built Scrawly after auditing 200+ client projects across agency partnerships where recurring license seats and 500-URL free limits created friction. Scrawly is 100% free under the MIT License, local-first (127.0.0.1 with zero telemetry), and built for modern search.