Crawler Watch

# Build Crawler Watch

You are a senior full-stack engineer pairing with me on a side project. Build a working first version of the product described below. Read the whole brief before writing any code, then follow "How to work" at the end.

## The idea

**Crawler Watch.** Upload your server logs and see which AI crawlers and search bots visit, which pages they read, and how much of your traffic is really people.

**Why now:** GEOInflow, a self-hosted AI crawler analytics tool, drew hundreds of GitHub stars after launching on Oct 26. A Dev.to post on Oct 30 also found one site counting 14 times too many visitors because of a bot. Site owners want to know who is really reading.

**Who it is for:** Site owners curious about AI search

**What makes it fun:** Finding out which AI has read your whole site.

## Scope

- **Build time:** Weekend. Size every decision to ship a working first version inside this window.
- **Technical ambition:** Level 2 of 5, "Basic logic / CRUD". A few screens over a simple data model with create, read, update, and delete. One lightweight persistence layer, no heavy infrastructure.
- **First version only.** Ship the core features below end to end. Stretch goals are listed so you can design around them, not build them.

## Core features (each must work end to end in the first version)

1. Upload server or CDN logs
2. Known AI crawlers and search bots identified
3. Most-read pages by bot
4. Humans against bots over time

## AI features

- Summarises what changed this week
- Labels unknown bots with a confidence note

How to build them:
- Put every model call behind one server-side module with a typed interface, so the provider can change without touching the UI.
- Read API keys from environment variables only, and document each one in `.env.example`.
- Write prompts as versioned constants with clear instructions and an example of the expected output; parse and validate responses before using them.
- Handle timeouts, rate limits, and malformed output with a graceful fallback the user can understand. Stream responses where a wait would exceed a couple of seconds.
- Never let a feature depend on the model being perfect. Show the result, let the user edit or retry.

## Stretch goals (later, not in the first version)

- Live log stream
- Alerts on unusual crawling
- Check your robots.txt rules

Leave clean seams for these (a module boundary, a nullable column, a feature flag), but do not implement them unless I ask.

## Tech stack

Use exactly: Next.js, DuckDB, Recharts.

Stay on current stable versions, follow each tool's official conventions, and add a library only when the standard library or the chosen stack cannot do the job cleanly. If part of this stack is a poor fit for a specific requirement, say so and propose the smallest change.

## Owner's notes

Written by me, the project owner. Treat these as requirements and preferences that override the generated material above when they conflict.

The first version should be exactly this, and no bigger: Upload one log file and see visits grouped by crawler and page.

## Build plan (with saved progress)

0 of 4 steps complete. Resume at step 1: Read logs. Before you start it, check what is already implemented so you do not repeat finished work.

1. **Read logs** [not started]
   Parse path, time, and user agent.

2. **Name the bots** [not started]
   Match against a list of known crawlers.

3. **Show pages** [not started]
   What each crawler reads the most.

4. **Compare** [not started]
   People against bots over time.

## Definition of done for the first version

- [ ] Upload server or CDN logs works end to end, with real data, not a placeholder.
- [ ] Known AI crawlers and search bots identified works end to end, with real data, not a placeholder.
- [ ] Most-read pages by bot works end to end, with real data, not a placeholder.
- [ ] Humans against bots over time works end to end, with real data, not a placeholder.
- [ ] Every screen handles loading, empty, and error states.
- [ ] Typecheck, lint, and any tests pass, and the main user flow was exercised by hand.
- [ ] A README explains setup, environment variables, and how to run it locally.

## Quality bar

- Responsive from a phone to a desktop, keyboard-accessible, semantic HTML, visible focus states, labelled inputs, and sensible contrast.
- Loading, empty, and error states for every screen that reads or writes data. Errors say what happened and what to do next.
- Validate on the server, never trust the client, and keep secrets in environment variables with a committed `.env.example`.
- Type-safe end to end. No `any` without a comment explaining why.
- Small, readable modules named after what they do. No premature abstraction, no dead code, no TODOs left for later without a note.
- Sensible defaults over settings. If a choice is not in this brief, pick the simplest option that keeps the product coherent and say what you chose.

## How to work

1. **Read, then restate.** Summarize the product in three sentences and list your assumptions. Ask only the questions that block implementation. For everything else, decide and note the decision.
2. **Plan before code.** List the files and modules you will create, in build order, mapped to the core features. If a build plan is included above, use it instead: resume at the first step not marked done and check what already exists before touching it.
3. **Set up.** If a project already exists, inspect its code, instructions, and conventions first and build on them. Otherwise scaffold the stack above with linting, formatting, a typecheck script, `.env.example`, and a README stub.
4. **Model the data first.** Define the types, schema, and any migrations before the UI, so the shape of the product is settled early.
5. **Build in vertical slices.** Finish one core feature end to end (data, logic, UI) before starting the next. Keep the app runnable after every slice and commit each slice with a message that says what changed and why.
6. **Then the interface.** Apply the quality bar above to each screen as it lands, not as a final pass.
7. **Verify.** Run typecheck, lint, and tests. Exercise the main user flow by hand. Check each item in the definition of done and each acceptance criterion.
8. **Report.** Tell me what works, what you tested and how, what is not done, every setup step I must complete myself, and the three most useful next steps. Never claim a service, test, or flow was verified when it was not.

If you can edit and run the project, start implementing right after the short plan. If you cannot, give me concrete code and the exact commands to run for the next step.

Rules: keep the product name and the scope as written, do not invent features, prefer boring and proven choices, and stop to ask me when a decision would be expensive to reverse.