Wellington Log Analyzer

I went looking for the AI bots visiting my site. I found familiar names. I also found a lot more I couldn't identify.

I'm building a site and I want to understand how those bots use it. Which pages. How often. I wanted an early signal of where the site might be appearing in AI answers.

That's why I built Wellington Log Analyzer. It reads server logs, the records of requests made to a website, and lets me follow a bot from its name to the pages it requested. Each visit gets recorded separately.

There's also the cost. Screaming Frog's Log File Analyser allows 1,000 log events in one project for free. I wanted to analyze more requests and keep multiple projects without a paid licence. I also wanted the files on my own machine. Wellington runs locally on my Mac.

The bots I couldn't name

I recognized the common AI crawler names. There were still a lot more unfamiliar visitors than ones I could put a name to. A lot more. I'd gone looking for a few known bots. Now I wanted to understand the rest of the traffic too.

I start with the user agent, the name a browser or bot sends with a request. Wellington checks it against a list of patterns. It groups the matches. It keeps the original text available if I want to inspect a request myself.

A request might match a named crawler. Or it might match a broad pattern like "bot" or "crawler". I can inspect those broad matches under Other Bot. If none of the rules match, the request lands under Unknown. I don't know from that label whether it's a person, a bot or an AI service.

I wanted to keep that unfamiliar traffic visible. Wellington imports all user agents by default. Looking for the names I know doesn't require throwing away the rest. I can always clear the filter.

I can't verify a name alone. For supported bots, I can turn on checks against published network addresses or confirmed domain records. Otherwise it's unverified.

A visit is an early clue

OpenAI has separate agents for search, possible training use and visits prompted by a user. They're all associated with the same company. Adding their requests together wouldn't tell me which of those things was happening.

I want those visits recorded. A page and a time to investigate. The request doesn't tell me which answer might mention the site. A place in ChatGPT search isn't guaranteed.

I start there. To say the site appeared in an answer, I'd need to find the answer too.

Putting the visits next to the pages

Wellington Crawler gives me another way to look at that activity. I can import its saved crawl into the analyzer. I can compare the pages it found with the requests in my logs.

I can see which pages appear in both sets. Which requested pages are missing from the crawl. Which crawl pages have no recorded requests. All within the selected period. I can narrow that comparison to one bot.

If a page has no recorded visit, I need to check the dates and files I've imported. There could be older visits or gaps in the logs. I can't assume the bot hasn't seen it.

I can open a group and inspect individual requests. Or I can export every matching row to a spreadsheet. All of them. I'm interested in following the unfamiliar visitors from there. The pages they request. The names they supply. Whether they return.