The short version
- ((screamingfrog-pricing|Screaming Frog)) is the default for technical work — most control, transparent pricing, free under 500 URLs.
- Consider ((sitebulb-pricing|Sitebulb)) when the output has to be read by someone who is not you.
- Bundled suite audits are fine for monitoring and inadequate for investigation.
- A free crawl of a small site is available to everyone and is the single most informative free thing in SEO.
A crawler is the one tool category where the free option is genuinely competitive for small sites, and where the paid options differentiate on something other than data volume. Everything below is judged on the two things that actually matter in practice: how much you can control the crawl, and how usable the output is.
The options
Suite-bundled audits (Semrush, Ahrefs) are included in those subscriptions rather than priced separately — see the comparisons linked below for their limits.
Screaming Frog — the default choice
€245 per licence per year with volume discounts from five licences, and a free tier that crawls 500 URLs. Runs locally, so crawling is unmetered and you can iterate freely. Custom extraction with XPath or CSS, granular crawl configuration, JavaScript rendering control, and integration with Search Console data.
The trade-off is that its output is a spreadsheet. It tells you what it found and leaves interpretation entirely to you, which is ideal if you know what you are looking at and unhelpful if you do not.
Sitebulb — when the report is the deliverable
Desktop Lite crawls 10,000 URLs per audit; Pro handles 500,000, raisable to two million. Cloud starts from £95/mo with unlimited crawling and scheduled recurring audits. The differentiator is its hints, which explain why an issue matters and what to do about it, and its audit comparisons, which make progress visible over time.
Bundled suite audits — monitoring, not investigation
Both Semrush and Ahrefs include cloud site audits. They schedule themselves, track a health score and require no configuration, which makes them good at noticing regressions. They lack custom extraction and fine crawl control, which makes them poor at answering specific questions. Ahrefs meters its audits in crawl credits.
The free route
For a site under 500 URLs, the Screaming Frog free tier plus Ahrefs Webmaster Tools plus Search Console gives you a complete technical picture for nothing. Most small business sites never exceed that limit.
Pick by situation
The limit shared by all of them
Every crawler reads rendered output. None of them sees the template that produced it. So a single bad line in one component becomes 400 separate rows in a crawl report, and fixing them "page by page" is a task that never completes.
That is why crawl reports so often look identical quarter after quarter. The list is accurate, the volume is demoralising, and the underlying cause is a handful of template-level decisions nobody traced back. Knowing that changes how you read a crawl: group the findings by template before you start fixing anything.