Lab – 02

Lumberjack.

I built Lumberjack because of a question I kept running into on AI search work. Everyone wants to know what ChatGPT or Claude says about their brand. Far fewer people check whether the assistant can actually get to the page in the first place. The answer sits in the server logs, and in 19 years I have rarely seen anyone read them for it.

So I wrote a tool that does. I give it a site's access logs, from the CDN or the server, and it shows me what every AI crawler and search engine asked for, what it received, and whether it waited for it. I verify every bot against the operator's published IP ranges, because a user agent is just a name anyone can type, and I sort them by what they are doing: training a model, building a search index, or fetching a page because someone has just asked a question.

That last group is the one I care about most. When ChatGPT or Claude fetches a page live, a person is waiting for the answer. If the server is slow, the assistant hangs up and the page is not cited. Lumberjack finds those moments, and the pattern behind them, so I can hand engineering something specific to fix.

I called it Lumberjack because it spends all day with logs. Like everything in the Lab, it is my own internal tool rather than a product. It runs on my machine, it builds on a method published by Zach Chahalis at iPullRank, and it is still a prototype: it will grow with each set of client logs I put through it.

See it in action

What it does

Six jobs, in the order I run them. The report below shows each.

  1. Read the logs: CDN or server, any common format
  2. Sort every request by bot and tier
  3. Verify bots against published IP ranges
  4. Find the pages bots never reach
  5. Check what they receive: errors, shells, redirects
  6. Trace the timeouts, and what to fix first

Inside the tool

A walk through one of my reports. I built it from a synthetic two-week log with known problems planted in it, so you can see what it catches.

What to fix, first I open every report with the fixes, in priority order, each with its evidence: live fetches hanging up, a firewall turning verified bots away, and pages no AI bot has ever fetched.
…and the rest Lower down: a page that only sends bots a shell, pages that are gone but still requested, redirects, and the impostors.
Who is knocking I sort every request into the four tiers (training, indexing, live fetching, acting) and verify it against each operator's published IP ranges.
Impostors and people sent by AI Anyone can call themselves GPTBot, so I count requests from outside the published ranges separately. People arriving from ChatGPT or Perplexity I keep apart from the bots.
Are they getting in? I check the client's priority pages one by one: did any AI bot actually get the page, or only an error?
Pages they never reach Site pages no AI bot fetched successfully, and old URLs that only the bots still know about.
What do they get? Status codes per bot, colour-coded: firewall 403s, gone pages, timeouts and redirects stand out at a glance.
403s and shells How many 403s were refused in under 50 milliseconds, the signature of a firewall rule, and 200s that carried a fraction of the page.
Do they wait for it? Response times by section and by cache status. A search crawler may come back later; an assistant answering someone's question may not wait.
The 499s When an assistant hangs up, the server logs a 499. I group them by how long it waited and by cache status, so I have a pattern to hand to engineering, not a single anecdote.
01 / 10
Ask me about a log audit →