Wellington Crawler

Screaming Frog is the standard, and I've used it for years. But the shape of it always bothered me. The crawl is the easy part. The gating is the product.

So I built the crawler and stopped there. It crawls a site, as far as I point it, and the data stays on my machine. That's the whole scope.

Why bother

Every crawler I pay for wants to be a suite. Dashboards, scores, a number out of 100 that means nothing. I don't want a score. I want the URLs, the status codes, the redirect chains and the canonicals, and then I'll decide what's wrong myself.

Also, the data is client data. Sending a client's full URL structure through someone else's cloud to get a crawl back is a trade I'd rather not make. Local means I don't have to think about it.

What it does

It pulls about eighty fields per URL. Status and indexability with the reason, canonicals from both the HTML and the HTTP header, hreflang from both, redirect chains, titles and meta descriptions measured in pixels as well as characters, headings, word counts, readability, response times, inlinks and outlinks.

On top of that: link scoring, near-duplicate detection, orphan pages checked against the sitemap, offline spelling in five languages, structured data, n-grams. It exports to CSV, Excel and XML sitemaps.

Two parts I'm actually pleased with, because they go past the tool I was copying.

It watches how a site responds and backs off on its own. If a server starts returning 429s or throwing challenges, the crawler slows down, then recovers when the site does. No sitting there tuning thread counts by hand.

And when a WAF blocks a fetch outright, it quietly refetches that URL through headless Chromium instead of recording a dead row.

The limits are all off by default. That's deliberate. The paid tools ship with a cap switched on.

How I know it works

There's a harness that runs the same fixture site through my crawler and through Screaming Frog, then diffs the output column by column. Where they disagree, that difference is written down rather than quietly smoothed over.

The test suite is about the same size as the crawler itself.

What it doesn't

It isn't a clone of the whole suite. Crawler only. No rank tracking, no backlinks, no reporting layer. If I need those I have Ahrefs, and I'd rather run one honest tool than a bad copy of five.

It doesn't crawl PDFs, and it doesn't hunt for video or audio files. Those are gaps, not decisions.

Stack: Python and PySide6, so it's a real Qt desktop app rather than a browser in a costume. httpx over HTTP/2 for fetching, selectolax and lxml for parsing, Playwright when a page needs rendering, SQLite for storage. A crawl saves as a single file you can open with any SQLite client.