Competitor Analysis

A comparison table is mostly boxes, and the hard ones are the boxes that stay empty. The competitor's site says nothing about the feature I'm checking. Does that mean they don't have it, or that I didn't look hard enough? My engine answered the first way. Every time.

I built Competitor Analysis to fill in those tables. It crawls a company and its rivals, reads the marketing pages and the help docs, then fills a grid with a row for every feature and a column for every product. The client gets a spreadsheet.

Before July my grid could say Yes and it could say No. So every hole in a crawl came out the far end reading as a fact about somebody's product. I shipped nine of those spreadsheets and I went back through all nine, and between them they say No 215 times and don't once say I don't know.

The word it was missing

There's a third answer now. Unknown. I don't let the engine say No until it has earned the No.

I hunt an empty box in stages. Pages I already hold. Then a search inside the company's own site. Then plain Google results. Then everybody else's pages, the reviews and the roundups. That last stage can't earn a No, because a review that skips a feature isn't evidence the feature is missing, it's a review that skipped a feature. It can still prove a Yes, so it stays in.

Most blanks stay blank. Of every hundred I chase, roughly one earns the right to be a No. The twelve spreadsheets I've shipped since July say Unknown 1,658 times and No seventeen.

I keep a log of the engine correcting itself and it holds 743 entries. Every one runs the same direction: something a competitor did have and my scan had missed. All 743. Not one of them runs the other way, catching the engine claiming something a competitor didn't have. So when it's wrong, it's wrong by not seeing what was sitting on the page.

What honesty costs in minutes

Honesty is slow. Every stage is another search a box has to fail through. My fastest full scan took 87 minutes and my longest took 153, and the longest is the most recent.

One scan in August chased 80 blanks and closed 29, earned not one No and left 51 open. My crawler had gotten three pages off one company's site and a single help document, while the rival handed it 15 pages and 132 help docs. That took 153 minutes. The stages behaved. My answer was still worth nothing.

The perfect scores that prove nothing

I have one scan hand-labelled to grade the engine against, 73 boxes of it. It answered Unknown on all 73. It grades perfect twice over. It never stated a wrong answer, because it stated none. And it scores full marks for finding what was there, because everything it failed to find got written off before the scoring started. Both numbers are true. Neither one tells me anything.

That's where it sits. The wrong-No problem is bought and paid for, in 1,658 Unknowns a client gets in place of an answer. The rest is open. I don't know how often my engine shrugs at something that was there to find, and the only way I know to measure it is hand-labelling a few thousand boxes.

What it doesn't read yet

Keyword Clustering already counts what people search for. Those numbers are built and I can reach them from here, and nothing in this engine reads them yet.

If you know how to count what my engine misses without hand-labelling every box, please let me know.