Release History

Reliable crawling of WAF-protected sites — smarter block detection and a one-click User-Agent switch

• New

  User-Agent block detection with a one-click fix. Many site firewalls (e.g. SiteGround) block Crab's Chrome user-agent outright. Crab now detects this after a crawl and shows a banner offering to switch to Safari and re-crawl in one click.

• Fix

  Sites on SiteGround and similar WAFs that returned 0 pages now crawl correctly — block detection extended to the X-Proxy-Cache-Info and Host-Header response headers, plus captcha / "Robot Challenge" interstitials that were previously stored as empty pages.

• Performance

  Crawls no longer trigger their own IP rate-limits. Crab stopped running its full WAF-bypass retry on every static asset (fonts, images, CSS, JS), which had been hammering protected origins into blocking the whole crawl.

• Fix

  JavaScript, CSS, font and image files were being listed as HTML pages — classification now trusts the URL extension over a mislabeled Content-Type.

• Fix

  Pages whose content fingerprint had its high bit set (roughly half of all values) were silently dropped by SQLite and never appeared in results.

• Fix

  Country-code domains like example.co.uk were scoped to co.uk, so every page looked external and the crawl stopped after one page. Now resolved via public-suffix eTLD+1 (.co.uk, .com.au, .co.nz, …).

• Fix

  Crawls could finish early — the completion check could close the queue before all sitemap URLs were enqueued, ending with fewer pages than expected.

• New

  429 Too Many Requests responses are now retried — Crab respects Retry-After (or waits 10s) and re-queues up to 3× instead of dropping the page.

• Fix

  Scheduling rejected standard cron expressions using ranges (1-5) or lists (8,12,18); the full cron syntax is now supported.

• UI

  User-Agent presets updated to current browser versions (Chrome 131, Firefox 128, Safari 17.6).

Compare Crawls, big-site reliability, and the version label finally tells the truth

• New

  New Compare tab — diff any two crawls of the same site and see what's fixed and what's new since last time. Site Health, total issues, errors/warnings/notices and per-issue deltas with Fixed/New pills.

• New

  Cross-project comparison — if you've crawled the same domain multiple times across different projects, all of those snapshots show up in the Compare dropdowns, sorted by date.

• Fix

  Big sites no longer fail Issues with "Failed to load issues report" — the 15s endpoint timeout was killing the audit on 10k+ page sites. Bumped to 120s, results are now cached per crawl, and tab-switches don't re-run the audit.

• Performance

  Near-duplicate detection rewritten for big sites — the O(N²) self-join was scanning tens of millions of page pairs on whydonate-sized sites. Added a partial simhash index and a GROUP BY approximation above 5,000 pages.

• Fix

  App footer version now reads the actual build version instead of the v0.1.0 placeholder that had been there since launch.

Re-enable redirect prompt buttons

• Fix

  The redirect prompt modal was leaving its buttons disabled because isStarting got stuck true after the first click.

Preflight on the home-page New Crawl flow

• Fix

  The seed-URL preflight check (host-changing redirects, scheme upgrades) now runs from the home-page New Crawl entry point, not just from inside an existing project.

Detect host-changing redirects on the seed URL

• New

  When the seed URL 301s to a different host (e.g. http://crab.so → https://www.crab.so), Crab now surfaces the chain and asks before crawling under the wrong domain.

Eliminate "Loading projects forever" on Windows 11

• Fix

  Port walking — when 8080 is held by McAfee / IIS / Hyper-V the backend now walks to the next free port, and the frontend asks for the actual port via Wails instead of hard-assuming 8080.

• Fix

  Pinned API host to 127.0.0.1 to dodge the IPv4/IPv6 localhost resolution mismatch that some Win11 boxes were hitting.

Robust CSV export with friendly errors

• Fix

  CSV writes are now atomic (temp file + rename) so a half-written file is never left behind, and lock / permission failures get a readable error instead of a stack trace.

Honour <base href> for relative URL resolution

• Fix

  Pages with a <base href> tag now resolve relative links correctly per HTML spec §4.2.3 instead of treating the request URL as the base.

Comprehensive SEO audit engine — 62 issue checks

• New

  A full Issues dashboard with 62 issue types across HTML Tags, Content, Links, Images, Indexability, Structured Data, Performance, Headings, Security, and URL Structure — each with an actionable fix guide and impact explanation.

• New

  Weighted Site Health score that deducts based on severity × priority × percentage of pages affected, instead of binary pass/fail.

PostHog analytics, table sorting fixes, external-links restructure

• New

  PostHog product analytics wired into the desktop app.

• Fix

  Fixed several table sorting edge cases (numeric columns, nulls, locale).

• UI

  External links restructured into a dedicated tab with source-page drill-in.

Cloudflare bypass with uTLS fingerprinting + adaptive rate limiting

• New

  uTLS fingerprinting so Crab presents a real Chrome TLS handshake to Cloudflare-protected sites; falls back to bundled Chrome when uTLS isn't enough; adaptive rate limiting backs off on 429s and gradually re-ramps.

Screaming Frog-style Images tab + External URLs tab + schema-type fixes

• New

  Dedicated Images tab with size, alt-text, and missing-alt filters.

• New

  External URLs tab grouped by target with source-page drill-in.

• Fix

  Several JSON-LD schema parsing bugs that were marking valid schema as invalid.

JavaScript rendering with bundled Chrome

• New

  JS rendering powered by a bundled Chrome headless shell — modes: off, auto (render only when needed), forced (always render). No need to install Chrome separately.

Sentry error tracking

• New

  Sentry wired into both the Go backend and React frontend with session replay on errors, so we actually see crashes when they happen.

Native-style pagination and layouts

• UI

  Pagination, table rows, and toolbar layouts re-styled to feel native on each platform instead of webby.

Fix "failed to fetch" on Windows

• Fix

  CORS preflight was being rejected on Windows because the Wails asset server set a different Origin than the API expected.

Crab 1.0 — initial release

• New

  Crab launches as a free, local-only SEO crawler for macOS, Windows and Linux. No URL limits, no subscriptions, no data leaving your machine.