Footprint

I needed to understand a large, messy company in a hurry. Manulife, before an interview. Insurance in Canada, John Hancock in the US, separate operating companies across most of Asia, and nothing saying plainly what belongs to whom.

I started with a crawler, because that's what I had. It found the website. It didn't find the company.

That's the flaw in crawling as a map. A crawler only sees what someone linked to. api., sso. and vpn. are never in a nav bar. Build the map out of hyperlinks and you've mapped a marketing site.

So Footprint reads the public record first and crawls last. Certificate Transparency leads, because every TLS certificate issued for a subdomain is logged publicly whether anyone links to it or not. Then async DNS, plus a brute force over a 159-word list. Then a probe of everything that resolved: status, the full redirect chain, title, a light tech fingerprint. Then the crawl, then the graph. Each stage writes its output to disk, so no step is a black box.

My last recorded run was eleven Manulife seeds. 451 hostnames found, 353 resolving, 188 alive, 83 returning an actual 200, across 52 registrable domains.

Reading before knocking

CT logs, DNS, and ASN records via RIPEstat need no account. That was deliberate. The moment a tool needs a paid key to do the basic thing, I stop reaching for it.

Keyless also means flaky, so I built for it. crt.sh 502s often, and on that run it returned zero rows. I only noticed in the log, because CT is two providers unioned, and certspotter carried all 217 hostnames alone.

Ahrefs is optional enrichment on top. My own access is OAuth through a connector, not a REST token, so a second path ingests a pre-fetched linked-domains export. Same edges, no credentials. In the one run I've archived the crawl was off, so all 50 edges came from that export.

It doesn't have its own crawler

Stage four imports Wellington Crawler's live source tree and drives its headless crawler. One engine, two tools. If Wellington isn't on disk it degrades to a minimal homepage link extractor and says so loudly.

The reuse cost me two bugs, both caught by running it, not reading it. Wellington's default scope is one subdomain, so manulife.com redirecting to www.manulife.com left nothing in scope. The crawl finished in about a second, with one edge. The other was the per-host rate cap set on the wrong config section, so the politeness limit I'd documented didn't exist.

What a review pass turned up

A review pass over the codebase produced 18 confirmed fixes. One still bothers me. The function that normalises a seed didn't lowercase it, so a seed typed as Manulife.com never matched the lowercased hosts coming back from the sources. Zero results, no error, no clue. A silent zero is the worst thing a tool can hand you.

Another was a security hole. Hosts resolving to private or reserved IPs were being probed, so a seed typed into the dashboard could have pointed it at 169.254.169.254. Screened at the resolver now.

Where it's honestly thin

No active scanning. No port sweeps, nothing intrusive past a liveness request. That's a decision, not a gap.

The entity labelling is a gap. Six brand rules, thirteen country suffixes, fourteen business keywords, all written for Manulife and John Hancock. A TOML file layers your own rules on top, but the brand default is unknown, so out of the box it labels one company and shrugs at every other. Coverage too. A brand behind aggressive CDN wildcarding gets under-mapped and I don't have an answer yet.

Version 0.1.0. Twenty-five tests over the real code paths. Every finished run auto-archives, so I can reopen an old scan exactly as it ran. My history folder holds one so far. Local repo, no remote, no release, no account needed.

Stack: Python 3.12 with httpx and dnspython, because the work is thousands of independent lookups and asyncio is the honest shape for that. networkx and pyvis build the graph and inline the viewer, so the output is one HTML file that opens offline. Streamlit for the dashboard, which trades polish for speed. The pipeline under it is headless, so a better front end can arrive later without touching the core.