Wellington Crawler

A crawler of my own.

It walks a website page by page. It follows the links it finds and writes down what it sees on each one. Titles, redirects, what's blocked, what's broken. That list is what an SEO audit runs on.

Screaming Frog already does this and I've paid for it. It's licence gated and capped. The crawl is the only part of it my work touches. So I built the crawl.

I didn't write the spec from memory. The vendor's own site got read first, page by page, and every claim I wrote down had to point back at the page it came from. Then the same treatment for the interface: their guide against the version running on my screen. Memory would've got me a crawler that matched what I remembered.

That proves nothing about my code though. How do I know the copy is a copy? I don't, unless I build something that catches me out.

There's one small test site. I crawl it twice with the same settings. Once with the licensed Screaming Frog, which is the only thing on this machine that can tell me the right answer. Once with mine. Then the two sets of results get lined up column by column and everything that disagrees gets printed: the column, how often, one example.

The result I care about most is the dull one. 38 pages against 38. The two lists matched. Line for line.

Both crawlers found the same pages. Same links followed. Same redirects, and the same pages skipped for the same reasons. Matching one column is easy next to agreeing on which pages exist at all.

In one place I copied a bug. Screaming Frog writes a fingerprint for every page and sometimes drops a character off the front of it (a quirk of the language it's built in). Mine drops the same character in the same place. Two of those short fingerprints sit in my tests as the expected answer.

One gap I won't paper over. Screaming Frog counts the sentences on a page its own way and won't show me how. It finds 74 where mine finds 42. I could tune my counter until it said 74 and have a number that's right on that one page and invented everywhere else.

The test asserts my 42, and asserts 42 is lower than 74. Direction only. A green tick there wouldn't have meant anything.

The copy has habits of its own. A page that blocks the crawler gets fetched again in a real browser, and the record comes from what the browser saw. It also watches how hard a site's pushing back and slows itself down before the whole crawl gets blocked.

Where it stands: it runs on my own machine, uncapped, with no licence to check, and nothing leaves the room unless I hook it up to an outside service myself. What comes out lands in the shape my technical SEO audits already read.

Ten columns still disagree. All ten are named, and printed on every run, so a column I've stopped looking at keeps turning up until I fix it.