Going further

Bots & AI crawlers

Automated traffic now makes up a serious share of hits on content-heavy sites. NumberHill classifies it at the ingest layer — before anything is written to your analytics — so it never inflates visitor counts or dilutes revenue per visitor. It is sorted into two reports, because the two kinds mean opposite things.

• AI visibility — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest. A crawl here is how your pages become citable inside ChatGPT, Claude, Perplexity or AI Overviews, so it is reported as a marketing signal with per-engine trend and the pages each one read.
• Bot traffic — scrapers, SEO tools, uptime monitors, security scanners and headless automation. Noise, tallied so you can see it and move on.

How it works

Every request to /api/collect has its user agent matched against a signature list. A hit records to a separate bot table and returns { ok: true, filtered: "bot" }; it never lands in events. That keeps the bot report accurate while leaving your real metrics clean.

Server-side capture (required for AI visibility)

Add this to middleware.ts in a Next.js app — it reports one hit per HTML request and never blocks or delays the response. Any server or edge function can do the same thing with a POST.

import { NextResponse, type NextRequest } from "next/server";

const NUMBERHILL_SITE = "YOUR_SITE_ID";
const NUMBERHILL_URL = "https://numberhill.com/api/collect/server";

export function middleware(req: NextRequest) {
  const res = NextResponse.next();

  // Fire and forget: analytics must never delay a page.
  fetch(NUMBERHILL_URL, {
    method: "POST",
    headers: { "Content-Type": "text/plain" },
    body: JSON.stringify({
      site: NUMBERHILL_SITE,
      ua: req.headers.get("user-agent"),
      ip: req.headers.get("x-forwarded-for")?.split(",")[0]?.trim(),
      url: req.nextUrl.pathname,
    }),
  }).catch(() => {});

  return res;
}

// HTML pages only — assets and API routes are noise.
export const config = {
  matcher: ["/((?!_next|api|favicon.ico|.*\\.[a-z0-9]+$).*)"],
};

The endpoint records only hits it classifies as bots, so calling it on every request cannot double-count a human visitor — those still come from the browser tracker. It responds with the verdict, so you can act on it:

{ "ok": true, "bot": "GPTBot", "verification": "verified" }

Telling real crawlers from impersonators

A user-agent is self-asserted. Sending ClaudeBot from a laptop takes one curl flag, and scrapers do exactly that because sites treat a recognised crawler more gently. The only unforgeable signal is the source IP, so NumberHill checks every hit against the operator's own published ranges.

Whose ranges are published

OpenAI (a separate list per agent), Perplexity, Google, Bing and Apple all publish JSON IP lists, and NumberHill caches them for six hours. Anthropic and ByteDance publish nothing, so ClaudeBot and Bytespider can only ever be unverified — that verdict is deliberately not the same as spoofed. Marking a real crawler as an impersonator would push its traffic into a block rule and cost you the AI reach you are trying to measure, so verification always fails open.

What gets detected

Reading AI visibility

Answer engines are the discovery channel that classic analytics cannot see. A visitor who asks ChatGPT a question and clicks through arrives as a referral, or as Direct — but the crawl that made the citation possible happened days earlier. That crawl is what AI visibility shows:

• Which engines read you, named by the product they feed — ChatGPT, Claude, Perplexity, Gemini / AI Overviews — not by the raw agent string.
• Which pages they read. Coverage gaps here are the AEO equivalent of pages missing from an index.
• Whether it is growing. The trend compares the recent half of the window against the earlier half; crawl traffic is too bursty for a day-over-day delta to mean anything.

Why this matters for revenue

Revenue per visitor is a ratio. Inflate the denominator with crawler traffic and every channel looks worse than it is — unevenly, since crawlers hit blog posts far harder than pricing pages. Filtering at ingest is what keeps the comparison between channels honest.