docs: per-package reference + showcase site (GitHub Pages) (#107)

* docs: per-package reference under docs/ (agent, index, ingest, eval, app, scripts)

Adds one reference doc per code package under docs/<package>/README.md,
plus a docs/README.md index that ties them to the existing cross-cutting
design docs. Each package doc follows the same shape — boundary, flow,
design decisions & rationale (with rejected alternatives), tool choices,
and a file-by-file map — grounded in the code as written, not the plan.

Also points the root README at docs/ for discoverability.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LX1gm8fuzK2ZeEWi3Zar1b

* site: showcase landing page + GitHub Pages deploy

A self-contained, responsive landing page (site/index.html) — dark/light
themes, the PyTorch flame palette, and a hero that plays the product's live
reasoning trace and resolves it into a grounded, cited answer. Describes
what the agent is, how the pipeline works, and how to use it (live Space or
self-host), linking into the per-package docs.

Publishes via GitHub Pages using the Actions deploy (actions/upload-pages-
artifact + deploy-pages), serving site/ as-is so it never touches docs/.
Requires a one-time repo setting: Pages → Source: GitHub Actions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LX1gm8fuzK2ZeEWi3Zar1b

---------

Co-authored-by: Claude <noreply@anthropic.com>

.github/workflows/pages.yml

@@ -0,0 +1,44 @@
+name: Deploy showcase site to GitHub Pages
+
+# Publishes the static landing page in site/ to GitHub Pages via the modern
+# Actions deploy (no gh-pages branch, no Jekyll). It serves site/ as-is, so it
+# never touches the markdown under docs/.
+#
+# ONE-TIME MANUAL STEP: in the repo, Settings → Pages → Build and deployment →
+# Source: "GitHub Actions". Until that is set, this workflow's deploy step has
+# nowhere to publish to. (This environment cannot flip that setting for you.)
+on:
+  push:
+    branches: [main]
+    paths:
+      - "site/**"
+      - ".github/workflows/pages.yml"
+  workflow_dispatch:
+
+# least privilege + exactly what Pages deploy needs (OIDC token to publish)
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+# one live deploy at a time; don't cancel a publish mid-flight
+concurrency:
+  group: pages
+  cancel-in-progress: false
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/configure-pages@v5
+      - name: Upload the site/ folder as the Pages artifact
+        uses: actions/upload-pages-artifact@v3
+        with:
+          path: site
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4

README.md

@@ -65,7 +65,10 @@ If the primary provider is unreachable or a free model is rate-limited, the app
 - When a question goes beyond the docs, say so honestly and point to where to look (source links, GitHub search) instead of guessing.
 - Stay easy to run locally with minimal setup.
 
-See [PLAN.md](PLAN.md) for the current roadmap and TODO list.
+See [PLAN.md](PLAN.md) for the current roadmap and TODO list, and
+[docs/](docs/README.md) for the design rationale and a per-package reference
+(one doc per code package: `agent/`, `index/`, `ingest/`, `eval/`, `app/`,
+`scripts/`).
 
 ## Building the index
 

docs/README.md

@@ -0,0 +1,83 @@
+---
+title: "TorchDocsAgent — documentation index"
+kind: index
+---
+
+# TorchDocsAgent — documentation
+
+This folder is the reasoning behind the code: **why** each part exists, the
+logic it follows, and the tool choices it rests on. It has two layers.
+
+- **Cross-cutting design docs** explain the system as a whole — the product
+  shape, the content pipeline, the agent contract, deployment.
+- **Per-package references** (`docs/<package>/README.md`) mirror the code tree
+  one-to-one: each documents a single top-level package — its boundary, its
+  flow, the decisions and their rejected alternatives, and a file-by-file map.
+
+Start with the design docs for the shape of the whole; drop into a package
+reference when you're about to touch that package.
+
+## The system in one picture
+
+```
+                         the docs.pytorch.org site
+                                   │
+   ingest/   discover → crawl → chunk           builds the _corpus/ snapshot
+                                   │
+   index/    embed → Neon (pgvector + tsvector)  storage + hybrid retrieval
+                                   │  pointers + hydrated sections
+   agent/    guard → route → grounded | loop → llm → Answer   the answering brain
+                                   │
+   app/      Gradio Space: stream the reasoning, render the answer + citations
+                                   │
+                              the user
+
+   scripts/  the CLI + CI surface that builds, evaluates, and operates all of the above
+   eval/     the measurement layer that keeps every layer honest
+```
+
+Data flows down the middle; `scripts/` and `eval/` wrap the column as tooling.
+The knowledge boundary is firm: the docs **site** is the corpus, source code is
+never indexed (it is referred out), and every answer is grounded in retrieved
+sections with clickable citations.
+
+## Per-package references
+
+| Package | What it is | Reference |
+|---|---|---|
+| **`ingest/`** | The crawl → snapshot → chunk pipeline that turns the live docs site into a heading-chunked corpus. Touches no DB. | [docs/ingest/](ingest/README.md) |
+| **`index/`** | The Neon/pgvector storage + retrieval layer: embeddings, hybrid search, DB-served hydration, and the self-healing freshness pass. | [docs/index/](index/README.md) |
+| **`agent/`** | The answer-generation brain: guard → route → grounded/loop → LLM dispatch → validated `Answer`. | [docs/agent/](agent/README.md) |
+| **`app/`** | The Gradio web app (the Hugging Face Space): serving, concurrency, and the live grey reasoning trace. | [docs/app/](app/README.md) |
+| **`eval/`** | The measurement layer: static answer checks, retrieval metrics, and LLM-as-judge — wired into CI and the live answer path. | [docs/eval/](eval/README.md) |
+| **`scripts/`** | The command-line + CI surface: build the index, generate synthetic data, evaluate, calibrate, and smoke-test. | [docs/scripts/](scripts/README.md) |
+
+## Cross-cutting design docs
+
+| Document | What it covers |
+|---|---|
+| [design-content-and-agent-flow.md](design-content-and-agent-flow.md) | The architecture bible: corpus scope, the ingestion pipeline, the three agent tools, the session flow, LangChain-vs-LangGraph, and how stored pointers map to live links. Start here. |
+| [retrieval-gaps-and-improvements.md](retrieval-gaps-and-improvements.md) | Known retrieval weaknesses and the improvement backlog (reranking, multilingual embedder, judge model, …). |
+| [loop-vs-langgraph.md](loop-vs-langgraph.md) | The measured comparison of the manual tool loop (`agent/loop.py`) against its LangGraph twin (`agent/graph.py`). |
+| [deploy-hf-spaces.md](deploy-hf-spaces.md) | The Hugging Face Spaces deployment walkthrough — secrets, the sync workflow, and the post-deploy smoke test. |
+
+See also [PLAN.md](../PLAN.md) for the milestone roadmap and the binding
+decisions that constrain execution.
+
+## Suggested reading order
+
+1. [design-content-and-agent-flow.md](design-content-and-agent-flow.md) — the whole shape in one read.
+2. [docs/ingest/](ingest/README.md) → [docs/index/](index/README.md) — how the corpus is built and retrieved (data flows up from here).
+3. [docs/agent/](agent/README.md) — how a question becomes a grounded answer.
+4. [docs/app/](app/README.md) — how it is served and streamed to the user.
+5. [docs/eval/](eval/README.md) and [docs/scripts/](scripts/README.md) — how it is measured and operated.
+
+## Conventions
+
+- Each package reference has YAML frontmatter (`title`, `kind: reference`,
+  `package`), then: boundary → flow → design decisions & rationale → tool
+  choices → file-by-file → related docs.
+- Docs describe the **code as written**, not an idealized plan. Where a plan
+  item is unbuilt or a decision was superseded, the doc says so.
+- These are OKF-style knowledge documents (markdown + frontmatter), the same
+  convention the corpus chunks and the other `docs/*.md` reports use.

docs/agent/README.md

@@ -0,0 +1,98 @@
+---
+title: "The agent/ package — answer generation"
+kind: reference
+package: agent
+---
+
+# The `agent/` package
+
+The answer-generation brain: it takes one vetted PyTorch question and returns a single docs-grounded `Answer` — with validated citations, honest referrals, and a bounded worst-case cost.
+
+## Why this package exists / its boundary
+
+`agent/` owns everything from "a question has arrived" to "a structured `Answer` is ready to render". It owns the **trust boundary check** (`guard.py`), the **cost decision** (`route.py`), the two answering strategies (`grounded.py` single-shot and `loop.py` / `graph.py` multi-tool), the **provider dispatch and structured-output contract** (`llm.py`), the **output schema** (`schemas.py`), and the **tool surface** the agent drives (`tools.py`, `tools_exec.py`).
+
+It does **not** own retrieval or content. The three tools are thin: `search_docs` calls `index.retrieve` + `index.hydrate`, `read_page` calls `index.hydrate`, and the guard's topicality check calls `index.retrieve.top_distance`. All embedding, hybrid search (pgvector + tsvector RRF), snapshot hydration, and the index itself live in `index/`. `agent/` treats retrieval as a pointer-returning oracle and works entirely with the hydrated section dicts it hands back. The static answer checks (parses/imports/symbols) live in `eval/checks.py`; `grounded.py` merely wires them into the live path. So the boundary is: **`agent/` decides how to think about a question and how to phrase the answer; `index/` decides what the docs say.**
+
+## The flow
+
+A question travels through the package like this:
+
+```
+question
+  │
+  ├─ guard.py         one check at the trust boundary (length → language → topicality)
+  │                   fail → refusal string, never reaches the LLM
+  │
+  ├─ route.py         zero-LLM regex heuristic: multi-source shape?
+  │      ├─ yes ─────────────────────────────▶ loop.py  (answer_agentic)
+  │      └─ no  ─────▶ grounded.py (answer_grounded, one pass)
+  │                        └─ produced NO citations? ─▶ escalate to loop.py
+  │
+  ├─ loop.py / graph.py   planner ⇄ tools within budgets, accumulating sections
+  │                       (search_docs ≤6, read_page ≤2, ask_source ≤1)
+  │
+  ├─ grounded.answer_from_sections   the shared terminus for BOTH paths
+  │                       build context → llm.answer_question → static-check
+  │                       repair (≤1) → validate_citations
+  │
+  └─ llm.py + schemas.py  provider dispatch → structured JSON → validated Answer
+```
+
+**The routing heuristic** (`route.needs_loop`) is a single compiled regex over the raw English question, matching the shapes where one retrieval pass is provably not enough: catalog (`what/which … exist / are available`, `list all …`), compare (`difference between`, `X vs Y`, `should I use X or Y`), recipe (`build/train … a model/network/classifier`, `end-to-end`, `from scratch`), and internals (`how is … implemented`, `under the hood`). Everything else goes to the single-shot grounded path. The heuristic is deliberately coarse because it decides **cost, not correctness** — it is allowed to be wrong.
+
+**Escalation on no citations** is the safety net that makes coarseness acceptable. When `answer_routed` sends a question to `answer_grounded` and the returned `Answer` has an empty `citations` list, that means one fixed retrieval pass found nothing groundable. Rather than ship an unsourced reply, `route.py` escalates the same question to `answer_agentic`, which can reformulate and re-search. The heuristic thus fails open in both directions: a misrouted multi-source question still gets a (slower) correct loop answer, and a misrouted simple question that comes back empty gets a second, more thorough attempt.
+
+Both the loop and the single-shot path converge on `grounded.answer_from_sections`, so the grounding contract (context-only generation, one static-check repair round, citation validation) is applied **exactly once and identically** no matter how the sections were gathered.
+
+## Design decisions & rationale
+
+**The grounding contract.** `GROUNDED_SYSTEM` in `grounded.py` instructs the model to answer *only* from the numbered context sections, be concise, cite every section it used verbatim, and add a `referral` instead of guessing when the context falls short. That instruction is enforced, not trusted:
+- `validate_citations` drops any citation whose `(url, anchor)` (or bare `url`) is not in the sections actually provided — the model cannot cite a page it wasn't shown.
+- `_regenerate_if_checks_fail` runs `eval/checks.run_checks` (code blocks parse via `ast`, imports are torch-family/stdlib only, every `symbols_used` entry appears verbatim in the prose) and, on failure, re-asks once with the failure reasons injected. The repair is kept **only if it is strictly cleaner** (fewer failing checks); it never blocks the user.
+- Per-section context is capped at `SECTION_CHAR_LIMIT` (2500 chars) with a **visible** truncation marker and a log line, so the model can referral out rather than silently answer from a cut-off view.
+
+**Route before loop (cost).** The loop costs ~5–13 LLM calls per question (planner rounds + answer + possible repair); on the free-tier deployment models that is minutes of wall-clock. The loop's value is real only for multi-source questions — measured agentic coverage 0.567 vs 0.133 single-shot — while a plain usage question is answered well by one retrieval pass plus one generation. A zero-LLM-call heuristic pays nothing to keep the common case cheap.
+
+**The guard uses embedding distance as the on-topic policy.** Membership in the docs' embedding space *is* the policy: `_is_on_topic` embeds the question, takes the cosine distance to its nearest doc chunk, and blocks anything beyond `DEFAULT_TOPICALITY_MAX_DISTANCE` (0.35, calibrated 2026-07-07 to sit between the worst on-topic 0.305 and the best off-topic 0.371). This is deliberate: an off-topic request and a prompt injection ("ignore your rules and …") both land far from the corpus and get the same refusal, so **no dedicated injection classifier is needed** — an earlier design used one, which cost an extra model call and still missed injections wrapped in on-topic questions. What passes the gate is safe anyway because of the downstream grounding contract. The guard is **fail-open**: any retrieval error is logged and allowed, so the guard can never take the app down, and the whole guard is toggleable via `TORCHDOCS_GUARD=0`.
+
+**English-only.** The corpus and the bge-small embedder are English-only, so `looks_english` (a cheap mostly-ASCII heuristic, tolerating a couple of stray non-Latin chars) bounces foreign input with a "please rephrase in English" refusal. The rejected alternative was a per-question translation LLM call, which dominated latency for non-English input. A multilingual embedder would remove this limit (noted in `docs/retrieval-gaps-and-improvements.md`).
+
+**The provider fallback chain.** `llm.py` dispatches across three providers — `openai-compat` (OpenRouter/DeepInfra/Nebius/…, the free-tier deployment default), `anthropic` (the paid production path), and `gemini`. `default_provider()` honors `TORCHDOCS_PROVIDER` or auto-detects from whichever API key is set; `_provider_chain` then puts the preferred provider first and appends every *other* configured provider, so one broken or misconfigured secret can't take the deploy down — the answer path self-heals to another provider. Within `openai-compat`, `TORCHDOCS_OPENAI_COMPAT_MODEL` is a comma-separated model chain (default `tencent/hy3:free` → `meta-llama/llama-3.3-70b-instruct:free`) so a rate-limited or retired slug falls through to the next. A process-wide **cooldown circuit breaker** (`_COOLDOWNS`) records models/providers that just failed (60s on a 429, an hour on a 404) so other in-flight requests skip them instead of each paying the full retry+sleep cost. There is hard-won secret hygiene here too: `_env_secret` strips trailing newlines (a pasted key with a newline becomes an illegal `Authorization` header and looks like a total outage), and `_redact` masks Bearer/`sk-` tokens before any exception text is logged.
+
+**The `progress` reasoning-trace sink.** `answer_grounded`, `answer_agentic`, and `answer_routed` all accept an optional `progress` callable. It is a sink for short human-readable trace lines ("🔍 searched …", "📄 found: …", "✍️ writing the answer", "↻ no sources yet — searching more thoroughly") that the web UI streams in grey while the answer assembles. `loop._humanize` turns the terse transcript records into these lines, falling back to the raw line on any shape it doesn't recognise so a format change degrades gracefully. It defaults to `None` (no-op), so scripts and tests are unaffected.
+
+**The LangGraph twin.** `graph.py` reimplements the manual loop as a LangGraph state machine (`planner → tools → planner` cycle, `planner/tools → generate → END`). It is intentionally the *second* implementation: it shares the exact same tools, budgets, `_plan` planner, and forced seed search, and its tool step is literally `tools_exec.execute_tool` — so the two drivers **cannot drift**. Only the control flow differs (an explicit graph vs a Python `while`-loop). The point is a controlled comparison of lines-of-code, debuggability, and latency, recorded in `docs/loop-vs-langgraph.md`, and a ready path to features that genuinely need a graph runtime (checkpointed multi-turn, human-in-the-loop, parallel tool fan-out).
+
+## Tool & library choices
+
+| Library | Used for | Why (and what was rejected) |
+|---|---|---|
+| **pydantic** | `schemas.py` `Answer`/`Citation`/`Referral`; every provider validates into `Answer` | One schema drives structured output across all three SDKs (`Answer.model_json_schema()` becomes a Gemini `response_schema`, an OpenAI `json_object` prompt, and an Anthropic forced-tool `input_schema`) and the validation error text feeds the single repair round. |
+| **anthropic** SDK | `_answer_anthropic` / anthropic raw path | Forced tool call (`tool_choice` → `submit_answer`) is the most reliable structured-output mechanism on the paid production model. |
+| **google-genai** SDK | `_answer_gemini` | Native `response_schema` + `application/json` mime type; free tier for cheap runs. |
+| **openai** SDK | `_answer_openai_compat` + shared `_compat_client`/`_compat_complete` | Talks to any OpenAI-compatible host (OpenRouter default). One pooled client per `(base_url, key, timeout)` keeps httpx keep-alive across the ~13 calls a question makes. |
+| **No LangChain** | — | Provider abstraction is `llm.py`'s own dispatch layer; the three tools are ours (SQL retrieval + snapshot reads + referral links); nothing is left for LangChain to abstract, so it is not a dependency. Older design docs mention **LiteLLM and Langfuse — neither is in the real code**; `agent/llm.py` is the hand-rolled dispatch, and observability is `print`-to-logs. |
+| **langgraph** | `graph.py` only | The orchestration runtime for the twin loop; imported lazily inside `build_graph()` so the manual path has no hard dependency on it. |
+
+A dispatch layer instead of a framework is the recurring theme: the structured-output contract is uniform (schema-valid `Answer` or `GenerationError`, one repair round) but each provider reaches it its own way, and keeping that in ~640 lines of `llm.py` bought the exact cooldown/fallback/secret-hygiene behavior the free-tier deployment needed.
+
+## File by file
+
+- **`__init__.py`** — empty package marker; imports are done lazily at call sites throughout the package (to keep optional SDKs like `langgraph`/`google-genai` from being hard import-time dependencies).
+- **`guard.py`** — the single input guardrail, run once on the raw user question at the trust boundary. Three cheapest-first checks (length → English → topicality). *Key decision:* embedding-distance membership is the entire on-topic policy — off-topic and prompt injection get the same refusal, no separate classifier — and it fails open on any error.
+- **`route.py`** — the cost router. A zero-LLM regex sends multi-source shapes to the loop and everything else to the single-shot path. *Key decision:* a grounded answer with no citations escalates to the loop, which lets the heuristic be coarse without shipping unsourced replies.
+- **`grounded.py`** — the single-shot path *and* the shared answer terminus (`answer_from_sections`). Builds the numbered context, generates, runs the static-check repair, and validates citations. *Key decision:* citations are validated against the provided sections, so the model can never cite a page it wasn't shown.
+- **`loop.py`** — the manual agent loop. A planner LLM returns a JSON action each step; tools run within budgets (`search_docs:6, read_page:2, ask_source:1`) until the model answers or a budget trips. *Key decision:* a forced seed `search_docs` on the raw question runs *before* the first planner call, so a rate-limited planner never blocks the obvious first retrieval.
+- **`graph.py`** — the LangGraph twin of `loop.py`, same tools/budgets/planner. *Key decision:* the tool node delegates to the shared `tools_exec.execute_tool`, guaranteeing the two drivers can't diverge behaviorally.
+- **`tools_exec.py`** — one tool-execution step (`do_search`, `execute_tool`) shared by both drivers. Mutates the section/referral/seen/transcript accumulators in place with consistent dedup and observation strings. *Key decision:* extracting the dispatch + dedup here is exactly what keeps loop and graph identical; budget accounting stays with each driver.
+- **`tools.py`** — the three tool functions themselves. `search_docs` (hybrid retrieval, optional `kind`/`library` filter, unknown `kind` degrades to unrestricted), `read_page` (whole-page hydrate; rejects a non-URL with a self-correcting error message), `ask_source` (referral-only). *Key decision:* `ask_source` never returns claims about the code — only DeepWiki + GitHub-code-search referral links — so it works with no network access and keeps source knowledge strictly out of docs-cited text.
+- **`llm.py`** — the provider dispatch layer: one question in, one validated `Answer` out, across gemini/anthropic/openai-compat with a self-healing provider+model fallback chain, cooldown circuit breaker, retry/backoff, and secret redaction/stripping. *Key decision:* a hand-rolled dispatch (not LiteLLM/LangChain) because the free-tier reality — dead slugs, per-minute and per-day rate limits, newline-corrupted secrets — needed bespoke resilience.
+- **`schemas.py`** — the pydantic output contract: `Answer` (`answer_md`, `symbols_used`, `torch_version`, `citations`, `referrals`) plus `Citation` and `Referral`. *Key decision:* the distinction between a `Citation` ("the answer came from here") and a `Referral` ("this is beyond the docs — look here") is encoded in the type system, mirroring the product's knowledge-boundary rule.
+
+## Related docs
+
+- [`../design-content-and-agent-flow.md`](../design-content-and-agent-flow.md) — the design rationale for the tools, session flow, and live-link mechanism this package implements.
+- [`../loop-vs-langgraph.md`](../loop-vs-langgraph.md) — the measured comparison of the manual `loop.py` against the `graph.py` twin.
+- [`../index/README.md`](../index/README.md) — the sibling package that owns retrieval, hydration, and the docs index this package delegates to.
+- [`../eval/README.md`](../eval/README.md) — the evaluation harness and `eval/checks.py`, the static answer checks wired into `grounded.py`.

docs/app/README.md

@@ -0,0 +1,156 @@
+---
+title: The app/ package
+kind: reference
+package: app
+---
+
+# `app/` — the Gradio web app
+
+The long-lived Gradio server that fronts the agent: the Hugging Face Space
+entrypoint that loads the embedding model once, streams a live reasoning trace
+while it works, and delegates every answer to `agent/`.
+
+## Why this package exists / its boundary
+
+`app/` owns **UI and serving concerns only**. It paints the chat surface, keeps
+the server up, manages concurrency and rate limiting, streams progress, and
+formats the final markdown. It does **not** answer questions — the moment a
+question needs answering, `_pipeline()` hands off to `agent/` (`agent.guard`,
+`agent.route.answer_routed`) and `index/` (`index.embed`, `index.freshness`).
+
+The split is deliberate: the batch GitHub Actions runs and the eval harness call
+the same `agent/` code with no Gradio in sight, and the app stays a thin,
+replaceable shell. If you are looking for retrieval, tool-calling, or JSON
+validation logic, it is not here — it is in `agent/`. What lives here is
+everything that turns a validated `Answer` object into a responsive web page.
+
+## The request lifecycle in the UI
+
+`respond()` (`app/main.py`) is a **generator**. Gradio streams every value it
+yields to the same output `Markdown` component, so the screen updates in place
+with no page refresh. The sequence per question:
+
+1. **Thinking note** — yields the static `THINKING_NOTE` ("🔎 Searching the
+   PyTorch docs…") immediately, so the first paint happens the instant Enter is
+   pressed.
+2. **Live grey trace + spinner** — the real work (`_pipeline`) runs on a daemon
+   **worker thread**. While it is alive, the generator loops every
+   `THINKING_TICK` (0.6 s), reads the trace lines the pipeline has appended, and
+   yields them via `_render_trace()`: each step in the theme's subdued grey with
+   a turning Braille wheel (`THINKING_SPINNER`) on a trailing line. A multi-
+   second wait reads as visible work rather than a freeze.
+3. **Black answer** — when the worker finishes, the generator yields the final
+   markdown (`render(answer)`), replacing the whole grey trace with the answer in
+   normal text.
+4. **Freshness spinner** — if the answer has citations and `index.freshness` is
+   enabled, a stale-while-revalidate pass starts. The answer stays fully on
+   screen while a **bare** wheel (just the spinner, no words) turns underneath it
+   via `run_below()` — the user reads while the app verifies the cited pages.
+5. **(Maybe) regenerated answer** — a clean check just drops the spinner. If the
+   cited docs **drifted**, `freshness.refresh_pages()` self-heals the stored
+   copies, `answer_routed` is re-run against the fresh content, and the new
+   answer swaps in with the `↻ FRESHNESS_NOTE`. A failed check or failed
+   regeneration silently keeps the answer already shown.
+
+### Why stream the reasoning trace and not answer tokens
+
+The obvious move — stream answer tokens like a chatbot — does not fit this
+pipeline. The answer is **not free prose**; it is a validated JSON `Answer`
+object assembled over several tool calls (search → read → generate → static
+check). There are no partial tokens to emit until the whole thing is built and
+validated. So instead the app streams the **reasoning**: the pipeline emits a
+short trace line per step (which docs it searched, what it found, when it starts
+writing) through the `progress` sink, and `respond()` renders those live. The
+grey trace is honest visible progress for a request whose payload can only arrive
+all at once.
+
+## Serving & robustness
+
+| Concern | Mechanism (in `app/main.py`) | Why |
+| --- | --- | --- |
+| **Warmup** | `_warm_up()` calls `index.embed.embed_query("warmup")` once at startup | Loads bge-small (~130 MB) before the first request, so no user eats the cold-start; also warms the guard's topicality embed. Best-effort — a warmup failure is logged, not fatal. |
+| **Concurrency** | `demo.queue(default_concurrency_limit=CONCURRENCY)` | Gradio defaults every event to serial `concurrency_limit=1`; opening the queue lets many I/O-bound requests (LLM + Neon) overlap so nobody waits in line. `max_threads` is lifted in step (`max(40, CONCURRENCY*2)`) so the thread pool — which also holds threads parked in 429 back-off — never becomes the hidden ceiling. |
+| **Backpressure** | `max_size=QUEUE_SIZE` | Bounds how many requests may wait behind the workers. Under a flood, extra callers get "queue full" instead of an unbounded, forever-growing line. |
+| **Per-client rate limit** | `_rate_limited()` — sliding window keyed on client IP | At most `RATE_LIMIT` questions per `RATE_WINDOW` seconds per IP, so one over-eager caller can't occupy every worker slot and burn the shared free-tier LLM quota. Set `RATE_LIMIT=0` to disable. The bucket table self-prunes past 4096 entries. |
+| **Fail-open errors** | `try/except` in `_pipeline` and in the worker `work()` | The UI must never crash or hang. Any exception is logged with type + message and the user gets the generic `ERROR_NOTE`; the real error never reaches the browser, since an exception string can leak hosts, model slugs, and config internals. |
+| **Smoke-test contract** | `ERROR_NOTE` contains the literal phrase **"went wrong"** | The post-deploy smoke test (`scripts/smoke_space.py`) greps for that marker to detect a broken Space. Keep the phrase. The `respond` event is also registered with `api_name="respond"` so the smoke test has a stable `client.predict(..., api_name="/respond")` endpoint; because `gradio_client.predict` returns the *last* yielded value, the generator's final yield is always a real answer. |
+
+## UX decisions & rationale
+
+- **Send-on-Enter requires `lines=1`.** The question `Textbox` is `lines=1` (not
+  2), and this is load-bearing: Gradio only fires `Textbox.submit` on a **bare
+  Enter for a single-line box**. A multi-line box (`lines>1`) treats Enter as a
+  newline and submits on Shift+Enter instead. `max_lines=6` still lets a long
+  question grow visually — the submit rule keys off the `lines` prop, not the
+  rendered height — so Enter keeps sending. Do not bump `lines` back to 2.
+- **Theme-aware subdued grey that survives the sanitiser.** Trace lines use the
+  inline style `color:var(--body-text-color-subdued)` (`TRACE_STYLE`), a Gradio
+  CSS variable that adapts to light/dark theme. The inline style survives
+  Gradio's markdown sanitiser (verified on gradio 6.20), which would strip a
+  `<style>` block or class. Trace text is `html.escape`d because a step can echo
+  the user's untrusted query.
+- **English-only copy.** All UI strings (`INTRO`, `EXAMPLES`, notes) are English,
+  matching the agent's English-only answering contract — no localisation layer.
+- **In-place freshness swap.** The freshness pass never blanks the screen or
+  refreshes the page: it streams updates to the same output component, keeping
+  the answer visible with only a spinner line changing underneath. Verification
+  is invisible unless it actually changes the answer.
+
+## Tool & library choices
+
+- **gradio** — provides the whole web surface (`Blocks`, `Textbox`, `Markdown`,
+  `Examples`, `Button`) and, crucially, the **queue** model that makes the app
+  concurrent and gives it backpressure. It is also the native Hugging Face Spaces
+  SDK, so the same file is the local dev server and the deployed entrypoint. Its
+  generator-streaming support is what lets `respond()` push a live trace.
+- **threading** (stdlib) — the answer pipeline runs on a daemon worker thread so
+  `respond()` can stay responsive and stream the spinner/trace instead of
+  blocking. A `threading.Lock` guards the shared `trace` list (worker appends,
+  generator reads) and the rate-limit buckets (`_RATE_LOCK`). `run_below()`
+  reuses the same pattern for the freshness and regeneration passes.
+- **python-dotenv** — `load_dotenv()` reads local `.env` so `python -m app.main`
+  works with `NEON_URL` + OpenRouter config without exporting vars by hand; on
+  the Space the real environment already carries them, so this is a no-op there.
+
+## File by file
+
+- **`app/main.py`** — the entire app: config constants and the `TORCHDOCS_*`
+  env-var reads; the rate limiter (`_rate_limited`); warmup (`_warm_up`); the
+  pure answer pipeline (`_pipeline`, no UI concerns) and markdown assembly
+  (`render`, `_render_trace`); the streaming UI generator (`respond`) with its
+  freshness pass; and the wiring (`build_ui`, `serve`, `main`). Running it as
+  `python -m app.main` starts a local server.
+- **`app/__init__.py`** — empty package marker (present but zero-length); the
+  module is imported as `app.main`.
+- **`app.py`** (repo root) — the Hugging Face Spaces entrypoint shim. It imports
+  `_warm_up`, `build_ui`, and `serve` from `app.main`, warms the model, and
+  builds the `demo` at import time (Spaces looks for a module-level `demo`), then
+  calls `serve(demo)` under `__main__` for local runs. It exists so the Space has
+  a top-level `app.py` while the real code stays in the `app/` package — the two
+  share `serve()` so bind settings never drift between entrypoints.
+
+## Configuration
+
+All read once at import in `app/main.py`; override per deploy in the Space's
+Variables and secrets.
+
+| Env var | Default | What it controls |
+| --- | --- | --- |
+| `TORCHDOCS_CONCURRENCY` | `16` | How many questions are answered at once (Gradio `default_concurrency_limit`, and the floor for `max_threads`). Generous because requests are almost all I/O wait. |
+| `TORCHDOCS_QUEUE_SIZE` | `64` | Queue `max_size`: how many requests may wait behind the workers before new ones are rejected with "queue full". |
+| `TORCHDOCS_RATE_LIMIT` | `8` | Max questions per client IP per window. `0` disables the throttle. |
+| `TORCHDOCS_RATE_WINDOW_SECONDS` | `60` | Length of the sliding rate-limit window, in seconds. |
+| `PORT` | `7860` | Server bind port (`server_name` is fixed to `0.0.0.0`). |
+
+Answering-side config (`NEON_URL`, the OpenRouter/LLM provider vars,
+`TORCHDOCS_LIVE_HYDRATE`) is consumed by `agent/` and `index/`, not by this
+package — see the deploy runbook.
+
+## Related docs
+
+- [`../deploy-hf-spaces.md`](../deploy-hf-spaces.md) — how the Space is created,
+  configured, and smoke-tested.
+- [`../design-content-and-agent-flow.md`](../design-content-and-agent-flow.md) —
+  the design rationale; §5.4 covers what the user sees (citations, referrals).
+- [`../agent/README.md`](../agent/README.md) — the pipeline this package
+  delegates to (guard, routing, tool loop, `Answer` schema).

docs/eval/README.md

@@ -0,0 +1,100 @@
+---
+title: "eval — the evaluation harness"
+kind: reference
+package: eval
+---
+
+# `eval/` — the measurement layer
+
+The `eval/` package is how we know whether a change to the agent made things better or worse. It holds the static checks that run on every answer, the offline benchmark suites (retrieval / judge / agentic / v0), a retrieval microscope for debugging misses, and the labeled question sets and result files that make "before vs after" a number instead of an opinion.
+
+## Why this package exists / its boundary
+
+The plan committed to **eval from day one** (`PLAN.md` §1.3, before any grounding was built). That order is deliberate: this project's whole reason to exist is that an ungrounded LLM confidently invents PyTorch APIs, and the only honest way to justify the grounding work — retrieval, glosses, reranking, citation-scoping, static-check regeneration — is to *measure* the hallucination it removes. The `eval/hallucinations.md` log is the origin point: 15 v0 questions run with no retrieval, 5 documented invented-API / wrong-signature findings, each one a measurable target that grounding then has to erase.
+
+So the boundary of this package is **measurement, not machinery**. Nothing here answers a user's question in production; it exercises the code that does (`agent/`, `index/`) and scores the output. The one deliberate exception is `checks.py`, which is imported by the live answer path — see [The static checks](#the-static-checks). Everything else reads from the same database and LLM providers the app uses, runs offline (locally or in the `Eval` GitHub Action), and writes JSONL to `eval/results/` for diffing.
+
+## The suites
+
+Each runner is a standalone `python -m eval.<runner>` entry point. They measure different layers so a regression can be localized — retrieval, generated prose, and the full agent loop are scored separately rather than as one opaque end-to-end number.
+
+| Runner | Layer measured | Metric emitted | Results file | In `Eval` workflow |
+|---|---|---|---|---|
+| `run_v0.py` | Generation + static checks on the manual v0 set | pass/fail table across parses/imports/symbols; `--grounded` adds `grounded_api_rate` + avg citations | `results/v0.jsonl`, `results/v0-grounded.jsonl` | no (local baseline) |
+| `run_retrieval.py` | Retrieval only — no LLM | mean `recall@k`, mean `MRR` | `results/retrieval_<set>.jsonl` | `suite=retrieval`, `retrieval+judge`, `all` |
+| `run_judge.py` | Generated answer prose (grounded single-shot) | LLM-as-judge faithfulness / answer-relevance / citation-correctness (1–5 → [0,1]) + overall; latency p50/p95/max/mean | `results/judge_<set>.jsonl` | `suite=judge`, `retrieval+judge`, `all` |
+| `run_agentic.py` | The full agent loop vs one-shot | citation `coverage` and the **agentic − single-shot delta** | `results/agentic_v1.jsonl` | `suite=agentic`, `all` |
+| `diagnose_retrieval.py` | (debugging, not a benchmark) | per-miss rank/distance triage, printed | none (stdout) | runs after `suite=retrieval` |
+
+Notes on the workflow (`.github/workflows/eval.yml`): the default one-click suite is `retrieval+judge` — recall/MRR plus the answer-quality anchor — chosen deliberately because the GitHub mobile app can't set workflow inputs, so the default has to be the measurement worth having. `agentic` stays opt-in (it is slow: many LLM calls per question for both paths, hence the 120-minute job timeout). Every job commits its `results/*.jsonl` back to `main` with `[skip ci]` so before/after diffs live in git history, not in ephemeral CI logs.
+
+A recurring guard shared by the LLM suites: a partial run (`TORCHDOCS_JUDGE_LIMIT`, `TORCHDOCS_AGENTIC_LIMIT`, non-8 `k`) suffixes its results filename (`_first10`, `_k4`) so a bounded CI run can **never masquerade as the full-set result** it would otherwise overwrite.
+
+## The static checks
+
+`checks.py` runs three checks on every `Answer`, and it runs no code — ever. This is a docs assistant, not a sandbox; the checks are static-analysis only:
+
+| Check | What it enforces |
+|---|---|
+| `parses` | Every ` ```python ` block in `answer_md` passes `ast.parse` (untagged fences, and `pycon` console sessions with `>>>` prompts, are intentionally skipped) |
+| `imports` | Every import in those blocks resolves to a torch-family root (`torch`/`torchvision`/`torchaudio`) or the stdlib (`sys.stdlib_module_names`); relative imports are always rejected |
+| `symbols` | Every entry in the answer's `symbols_used` actually appears in the prose, tolerant of conventional spelling (`nn.Linear` for `torch.nn.Linear`, `F.relu`, `.add_`) with word-boundary matching so `torch.relu` isn't "found" inside `prelu` |
+
+**The important fact: these checks run in the *live* answer path, not just offline.** `agent/grounded.py` imports `run_checks` and calls it inside `_regenerate_if_checks_fail`: when a generated answer fails a check (unparseable snippet, off-family import, a symbol listed but missing from the prose), it re-asks the model *once* with the specific failure reasons injected, and keeps the repair only if it is strictly cleaner — never blocking the user on a failed check. So the same code that scores answers in the benchmark is a real-time quality gate on production answers. That dual use is why `checks.py` is pure functions over an `Answer` with no I/O: it has to be cheap and side-effect-free enough to sit in the request path.
+
+## LLM-as-judge
+
+`run_judge.py` closes the gap the other suites leave open: retrieval eval scores whether the right *pages* were found and the static checks score whether *code* parses, but neither scores the prose the user actually reads. The judge does, on three dimensions:
+
+- **faithfulness** — is every claim supported by the provided context, or invented beyond it? (an honest "not in the docs" referral counts as faithful, not a failure). This is the hallucination axis the grounding contract exists to hold.
+- **answer_relevance** — does the answer address the question actually asked?
+- **citation_correctness** — do the cited sections genuinely support the claims, and is every load-bearing claim cited?
+
+Each is scored 1–5 and normalized to `[0,1]` so it shares a scale with the retrieval metrics; the mean is the `overall` before/after number. The judge sees the **same numbered context the answer saw** (`build_context`), so faithfulness is checked against the real inputs rather than a re-retrieval.
+
+Two caveats are baked into the docstring, honestly:
+
+1. **Same-family bias.** On free-tier keys the judge may be the *same model* that wrote the answer, which biases toward leniency — so the score is a **relative gauge for regressions, not an absolute grade**. Pointing `TORCHDOCS_*` at a stronger, independent judge model is an open `PLAN.md` M4 item ("Pick a dedicated judge model"); until it lands, read the numbers as a trend, not a grade.
+2. **Trust boundary.** The judge reads model-written answers and doc text, so its system prompt is hardened to *score* those as data, never to *follow* embedded instructions — but a suspiciously perfect run still deserves skepticism.
+
+The judge run also captures **latency**, and deliberately times only what the user waits on — retrieval + answer generation, with the eval-only judge call excluded — then reports p50/p95/max/mean. That is the core UX number: question in → answer out.
+
+## Design decisions & rationale
+
+- **Static before semantic.** The checks in `checks.py` (deterministic, free, in-path) run before any LLM judgment. A parse error or a fabricated import is a hard defect that never needs a model to adjudicate, and catching it deterministically keeps the expensive, noisier judge focused on prose quality. The regeneration loop reflects the same ordering: cheap static repair first, LLM judgment reserved for offline scoring.
+- **Before/after comparability via `index_version`.** The corpus is always-latest (a PyTorch release just shows up as a hash-diff on the next recrawl), so "recall went up" is only meaningful against a fixed index. `index_version` is an internal crawl-build id kept precisely for eval comparability (and cache invalidation), decoupled from PyTorch version numbers — it's what lets a retrieval delta be attributed to a retrieval change rather than a corpus change underneath it.
+- **A labeled retrieval set.** `run_retrieval.py` measures recall/MRR against questions carrying *expected* source groups (each group a list of alternative URL/title substrings, any alternative counting as a hit). This costs authoring effort — 100 questions written against the verified docs inventory — but it's what makes retrieval measurable without an LLM in the loop, which is what makes it fast, deterministic, and cheap enough to run on every retrieval-affecting change. The glosses×reranker recall jump from 0.430 → 0.840 (documented in `docs/retrieval-gaps-and-improvements.md`) is only a claim because this labeled set exists.
+- **Coverage delta, not absolute, for the agent loop.** `run_agentic.py`'s headline is `agentic_coverage − single_shot_coverage`: catalog/compare/recipe answers are spread across pages, so the honest question isn't "is the agentic answer good" but "did the loop assemble *more* of the answer than one search would." A negative delta is reported as an honest negative result, not hidden.
+
+## Tool & library choices
+
+- **`ast` for the static checks.** Parsing candidate code with the standard-library `ast` module (not regex, not execution) is both safe — no code runs — and precise: `ast.walk` finds every `Import`/`ImportFrom` node including relative imports, which a regex would miss. `textwrap.dedent` first, because models routinely indent whole blocks inside markdown lists.
+- **The app's own LLM dispatch, reused for judging.** `run_judge.py` calls `agent.llm._raw_completion` with the shared provider/fallback chain rather than a bespoke client. Same dispatch, same fallback behavior, one place to configure providers — and the judge reply is validated through Pydantic models (`JudgeScores`), with `_extract_json` tolerating the fences and stray prose models emit despite an "only JSON" instruction.
+- **JSONL result files.** Every runner writes one JSON object per line to `eval/results/`. It's append-friendly, diffs cleanly in git (which is where before/after lives), and is trivial to load back for aggregation. Runners that spend scarce free-tier quota (`run_v0.py`, and the LLM suites) flush after every question and, in `run_v0.py`, resume from prior results unless `--fresh` — so a crash or rate-limit mid-run never discards answers already paid for.
+
+## File by file
+
+- **`__init__.py`** — empty; marks `eval/` as a package so the runners are importable as `eval.<name>`.
+- **`checks.py`** — the three static checks (`check_code_parses`, `check_imports_allowed`, `check_symbols_present`), the `CHECKS` registry, `run_checks`, and `format_table` for the pass/fail grid. Pure, no I/O; imported by both the runners and the live `agent/grounded.py` regeneration path.
+- **`run_v0.py`** — runs the 15-question manual v0 set through `answer_question` (ungrounded) or, with `--grounded`, through `answer_grounded`, applies `run_checks`, and prints the pass table. `--grounded` additionally computes `grounded_api_rate` (share of `symbols_used` that actually exist in the docs index, via a tsvector probe) and average citations. Resumable; flushes per question.
+- **`run_retrieval.py`** — the retrieval-only benchmark. Loads the selected set (`v1` inline expectations, or `v0` with a `retrieval_v0.jsonl` sidecar), runs `index.retrieve.retrieve` at `k`, and computes per-question `recall@k` and `MRR`, then aggregates. No LLM. The reference before/after suite for any retrieval change.
+- **`run_judge.py`** — the LLM-as-judge suite (above): generates a grounded single-shot answer, scores it on the three dimensions with a hardened judge prompt, normalizes to `[0,1]`, aggregates, and reports UX latency percentiles. Pure parts (`_normalize`, `_extract_json`, `parse_judge_reply`, `aggregate`) are unit-tested.
+- **`run_agentic.py`** — the agent-loop benchmark for multi-page (catalog/compare/recipe) questions in `agentic_v1.jsonl`. Runs each through both `answer_agentic` and `answer_grounded`, scores citation `coverage` against `expected_any` (objective URL/anchor/title substring match — no judge), and reports the loop-vs-single-shot delta.
+- **`diagnose_retrieval.py`** — a debugging microscope, not a benchmark. For a handful of known descriptive misses it prints, per kind-pool, the nearest candidates with cosine distances and locates the *expected* page in the raw dense/keyword candidates — triaging whether a miss was never a candidate, out-ranked within its pool, or dropped by the relevance-gap filter, and whether the expected page's own nearest chunk is a *crowding* problem (deeper pool helps) or an *embedding* problem (only doc-side enrichment helps). Runs in Actions after the retrieval suite; intended to be deleted once the fix it points at lands.
+
+### Data files under `eval/`
+
+- **`questions_v0.jsonl`** (15) — the manual day-one set spanning the five question types (usage / catalog / recipe / source / edge); driven by `run_v0.py`.
+- **`questions_v1.jsonl`** (100) — the main labeled set, authored against the verified external docs inventory with expected sources inline; driven by `run_retrieval.py` and `run_judge.py`.
+- **`retrieval_v0.jsonl`** — expected-source sidecar for the v0 set (v1 keeps expectations inline).
+- **`agentic_v1.jsonl`** (20) — the multi-page catalog/compare/recipe set with `expected_any` source groups; driven by `run_agentic.py`.
+- **`invalid_v1.jsonl`** (100) — out-of-scope questions (e.g. React/`useState`) for refusal / negative-case calibration, consumed by `scripts/` (e.g. `calibrate_guard.py`), not the core runners.
+- **`docs_inventory.jsonl`** / **`index_manifest.jsonl`** — the verified external docs inventory (symbol → URL) and the live index manifest; produced by `scripts/dump_*` and used to author expectations and diff coverage (`scripts/coverage_diff.py`).
+- **`hallucinations.md`** — the v0 ungrounded-baseline hallucination log (OKF-style, per-finding frontmatter): the measurable target grounding had to erase.
+- **`results/`** — committed JSONL outputs of the suites (`retrieval_*`, `judge_*`, `agentic_*`, `v0*`), the substrate for before/after diffs.
+
+## Related docs
+
+- [`../design-content-and-agent-flow.md`](../design-content-and-agent-flow.md) — the product boundary (site + `main` only, no sandbox, tool-calling loop) the suites are written to measure.
+- [`../retrieval-gaps-and-improvements.md`](../retrieval-gaps-and-improvements.md) — the RAG maturity review whose recall/MRR numbers come straight out of `run_retrieval.py` and `diagnose_retrieval.py`.
+- [`../agent/README.md`](../agent/README.md) — the sibling package these suites exercise (`answer_grounded`, `answer_agentic`, and the `checks.py` regeneration hook).

docs/index/README.md

@@ -0,0 +1,277 @@
+---
+title: "The index/ package — retrieval + storage layer"
+kind: reference
+package: index
+---
+
+# The `index/` package
+
+The Neon/pgvector storage and retrieval layer: it owns the `chunks` table, embeds
+corpus chunks with a local bge-small model, serves hybrid (dense + keyword) search
+as ranked pointers, hydrates section/page content, and self-heals drifted chunks
+against the live docs after an answer goes out.
+
+## Why this package exists / its boundary
+
+Three packages split the RAG core along a clean seam, and `index/` is the middle one:
+
+| Package | Owns | Hands off |
+|---|---|---|
+| `ingest/` | Crawls the docs site, builds the on-disk `_corpus/` snapshot, chunks pages by heading | Chunk *units* (`{url, anchor, heading_path, content, content_hash, kind, library, …}`) |
+| **`index/`** | **Neon schema, embeddings, the `chunks` table, hybrid retrieval, hydration, freshness** | **Ranked *pointers* with content already attached** |
+| `agent/` | The tool-calling loop, grounding, generation, guards | Answers with citations |
+
+`ingest/` decides *what the corpus is*; `index/` decides *how it is stored and found*;
+`agent/` decides *what to say*. The boundary is deliberate: `index/` never crawls
+(that is `ingest/`'s job — see `_live_units` and `hydrate` importing `ingest.crawl`
+rather than re-implementing it) and never reasons about answers (it returns pointers,
+not prose). The one place this package reaches back into `ingest/` is to re-use the
+*exact* chunking and HTML→markdown code, so a chunk healed at answer time is
+byte-identical to one the batch build would have written — see the freshness section.
+
+The snapshot under `_corpus/` remains the crawl-time source of truth; the `content`
+the DB now stores is a *served copy* of it (see the data-model note below).
+
+## The data model
+
+Everything lives in one wide table, `chunks` (defined in `index/db.py`'s `SCHEMA`).
+One row = one heading-granular section (or one size-split *part* of an oversized
+section).
+
+| Column | Purpose |
+|---|---|
+| `id` | `bigserial` surrogate key |
+| `chunk_key` | `sha256(url#anchor#heading_path[#partN])`, **unique** — the identity used for upsert, skip-unchanged, purge, and heal |
+| `url`, `anchor`, `page_title`, `heading_path`, `source_link` | The pointer — enough to render a live citation and a `[source]` referral without a lookup |
+| `library` | `core` / `vision` / `audio` / … — lets retrieval filter or route per question |
+| `kind` | `api` / `tutorial` / `guide` — the per-kind retrieval pools key off this |
+| `content_hash` | Page/section content hash; drives the incremental skip and the freshness compare |
+| `index_version` | Which build wrote the row (answers are stamped with it) |
+| `part` | Ordinal within a size-split section; `0` keeps the legacy key format so older rows stay valid |
+| `embedding` | `vector(384)` — the bge-small dense vector (cosine) |
+| `tsv` | `tsvector` — Postgres full-text index of the same `indexed_text` |
+| `content` | Raw section text, **served at answer time** so hydration needs no live fetch |
+
+A second tiny table, `index_meta` (key/value), stores the live `embed_recipe` so the
+build can detect a recipe change and force a full re-embed.
+
+**Indexes.** `chunks_embedding_idx` is HNSW with `vector_cosine_ops` (approximate
+nearest-neighbour for the dense channel); `chunks_tsv_idx` is a GIN index over the
+`tsvector` (the keyword channel); `chunks_url_idx` supports the freshness/hydrate
+lookups that fetch all rows for one page. Uniqueness is on `chunk_key` — the whole
+upsert/skip/heal machinery hangs off it.
+
+**Width can't drift.** `EMBED_DIMS` is derived from the configured model
+(`_MODEL_DIMS` table in `db.py`), reading the *same* env var (`TORCHDOCS_EMBED_MODEL`)
+that `embed.py` reads, so the column width and the vectors written into it can never
+disagree. An unknown model with no `TORCHDOCS_EMBED_DIMS` override is a loud config
+error, not a silently wrong-width table. If the dimension *does* change (a model swap),
+`ensure_schema` reads `atttypmod` off `pg_attribute`, sees the mismatch, and drops +
+recreates the table — the index is treated as a rebuildable cache, not precious state.
+
+**Pool + runtime-migration-at-pool-open.** There are two ways into Neon, for two
+workloads:
+
+- `connect()` — one dedicated connection for the batch build (a single long-running
+  writer that commits in checkpoints).
+- `get_pool()` — a process-wide `ConnectionPool` (cached with `functools.cache`) for
+  the web app, where many concurrent questions each borrow a connection for a couple
+  of quick reads. Reconnecting per read costs a TLS handshake (~100–300 ms) that would
+  dominate answer time under load and risk exhausting Neon's free-tier connection cap.
+  `max_size` (`TORCHDOCS_DB_POOL`, default 8) is kept at or under the plan limit;
+  `check=check_connection` validates a connection on checkout so a Neon-side idle
+  timeout surfaces as a fresh connection rather than a mid-request query error.
+
+The subtle bit: **`RUNTIME_MIGRATIONS` are applied when the pool opens, not only at
+build time.** `create table if not exists` won't touch an existing table, so a column
+added after the table first shipped (`part`, `content`) needs an idempotent
+`alter table … add column if not exists`. The app SELECTs those columns, and a fresh
+deploy can go live *before* the next index build runs `ensure_schema` — so `get_pool()`
+runs the same migrations itself, or every search would 500 until the next build. Both
+writers (`ensure_schema`) and the reader (`get_pool`) apply the identical idempotent
+list.
+
+## The retrieval flow
+
+`retrieve()` in `index/retrieve.py` is the engine behind the agent's `search_docs`
+tool. For each content **kind** (`api`, `tutorial`, `guide`) it runs two channels and
+fuses them *within the kind*:
+
+- **Dense** (`DENSE_SQL`): `embedding <=> query_vector` cosine order over the HNSW
+  index, `pool` candidates.
+- **Keyword** (`KEYWORD_SQL`): `tsv @@ plainto_tsquery(...)` ranked by `ts_rank` — this
+  rescues exact symbol names (`scaled_dot_product_attention`) that dense similarity
+  misses.
+
+The two rankings are merged with **Reciprocal Rank Fusion** (`rrf_merge`,
+`score = Σ 1/(k+rank)`). A per-pool relevance gate (`RELEVANCE_GAP = 0.15`) drops
+candidates far from *that kind's own* best hit — gated per pool, never against a global
+best, so a close tutorial can't set a threshold that filters the entire `api` pool out.
+Pools are then interleaved round-robin, strongest pool first (`_interleave_pools`), and
+the first `k` win. When the query contains a dotted/underscored identifier a third
+**symbol channel** (`SYMBOL_SQL`) runs, matching the token in url/title/heading and
+preferring `kind='api'`; it leads the interleave, and an *exact* API-reference hit is
+pinned to position 0 — the docs-search behaviour users expect.
+
+Two hard-won details live here as comments:
+
+- **`SET hnsw.ef_search = 150`** on every pool query. HNSW is approximate *and*
+  pgvector applies the `WHERE kind=…` filter *after* the index scan; at the default
+  `ef_search=40` a `kind='api'` query first collects the ~40 globally-nearest chunks
+  (mostly tutorials for a descriptive question) and only then filters — the api page
+  can be discarded before the filter sees it. Widening the candidate scan (~2 ms extra
+  on a 7K-chunk index) rescues pages whose true in-pool rank is good.
+- **The cross-encoder reranker is gone.** A rerank stage sat between fusion and the
+  final order until the 2026-07-10 ablation on real content measured it at
+  +0.02 recall / −0.005 MRR — no earned keep for a ~90 MB model and per-query cost.
+  The fused RRF order is the final order. (Global-RRF, a reference channel, and reserved
+  seats were also tried and rejected; see the module docstring.)
+
+**What a "pointer" is.** A pointer is a dict of `POINTER_COLUMNS` — `chunk_key, url,
+anchor, page_title, heading_path, library, kind, source_link, part` — **plus `content`**.
+That last column is the latency story: `content` rides along in the *same* retrieval
+query, so the answer path hydrates each section straight from it with zero extra
+round-trips. `hydrate_section` in `index/hydrate.py` takes a fast path when
+`pointer["content"]` is present (just returns it); only an empty/un-backfilled row falls
+through to a snapshot read or live fetch. Previously each of *k* sections was a separate
+live page fetch — the dominant answer latency (measured p50 ≈ 12 s, one outlier 69 s).
+`hydrate_sections` also runs the remaining fetch-path work concurrently in a thread pool
+(preserving retrieval order, dropping gone sections), collapsing *k* round-trips into
+roughly one. `hydrate_page` serves a whole page for `read_page`, returning the heading
+outline instead when a page exceeds `PAGE_CHAR_LIMIT` (30 k chars).
+
+`top_distance()` is a small sibling of retrieve: it returns the single smallest cosine
+distance for a query (a topicality signal the agent's input guard uses to tell on-topic
+from off-topic), using the same pooled access.
+
+## Freshness (stale-while-revalidate)
+
+`index/freshness.py` implements the standard stale-while-revalidate pattern for the
+docs index: **serve the stored copy instantly, revalidate right after.** The design
+model that drove it: the product promises answers grounded in the docs *as the site
+serves them today*, but hydrating from the DB `content` column means an answer can be
+served from a chunk that drifted since the last weekly crawl. So *after* an answer goes
+out, `refresh_pages(cited_urls)` re-checks exactly the pages that answer cited and heals
+any drift, telling the caller which urls changed so it can regenerate the just-shown
+answer.
+
+The pass, step by step:
+
+1. **TTL gate** (`_due`, `TORCHDOCS_FRESHNESS_TTL_SECONDS`, default 3600 s). A
+   thread-safe check-and-set so one hot page isn't re-fetched on every question; the
+   URL table is bounded (evicted past 4096 entries) for a long-lived process. Docs move
+   slowly and the weekly crawl is the backstop — an hour bounds both fetch traffic and
+   the staleness window.
+2. **Live compare.** `_live_units` fetches the page *right now* and chunks it with the
+   exact `ingest` code path (`fetch_html` → `extract_main_html` → `to_markdown` →
+   `chunk_page`), computing the page-level `content_hash` the same way the crawl does.
+   Each live chunk is matched by `chunk_key` to its stored row and compared by `content`.
+3. **In-place heal** (`_HEAL`). A drifted row is fixed *completely*: `content`,
+   `content_hash`, a freshly computed `embedding`, and the `tsv` — all in one UPDATE. The
+   embedding model is already hot in this process (it embeds every incoming query), so
+   re-embedding a handful of chunks costs milliseconds and leaves the row exactly as a
+   fresh build would have written it. Because the stored hash now matches the re-embedded
+   text, the next weekly crawl correctly *skips* the healed rows instead of redoing them.
+4. **Scope limit.** The re-embed re-uses the page's *existing* gloss and hypothetical
+   questions (via `indexed_text`) — they describe what the symbol is *for*, which small
+   edits don't change. Sections that appeared or vanished on the live page are skipped;
+   structural change (new/deleted pages, restructured sections, drift drastic enough to
+   need fresh enrichment) stays the job of the periodic Build Index crawl.
+5. **Fail-open + kill switch.** Every failure — a dead page, a bad fetch, a DB error — is
+   logged and skipped; the outer `try/except` guarantees a freshness error can never
+   break an already-shown answer. `TORCHDOCS_FRESHNESS=0` disables the pass entirely.
+
+## Design decisions & rationale
+
+- **Local bge-small on CPU (no API, no quota, no cost).** Gemini's free embedding quota
+  (~100 items/day) would take *weeks* for the corpus; a 130 MB open model has no key, no
+  quota, and no per-item cost — the whole corpus embeds in minutes on a CI runner's CPU,
+  and the *same* model embeds queries at answer time (which is what makes millisecond
+  in-process healing possible). The tradeoff is accepted and bounded: bge-small is
+  English-only, and a controlled A/B against bge-base (768d, full re-embed + benchmark,
+  2026-07-08) showed **identical** recall (0.846) and marginally worse MRR at 4× the
+  model size and 2× the build time — no measurable gain, so the cheaper model stays. The
+  `_MODEL_DIMS` table remains so a future swap is a one-line change plus an automatic
+  rebuild.
+- **`EMBED_RECIPE` versioning.** `indexed_text()` embeds far more than raw body: symbol
+  + synopsis + Contextual-Retrieval gloss + QuOTE-style hypothetical questions + heading
+  + content (see `retrieval-gaps-and-improvements.md` for *why* — descriptive questions
+  live in a different region of embedding space than terse reference pages). When that
+  *shape* changes, the row-skip check would otherwise keep stale vectors, so the recipe
+  string (`v7-<model>-g<gloss_stamp>-q<questions_stamp>`) is stored in `index_meta`; any
+  change forces a one-time full re-embed. The gloss/question **content stamps** are
+  hashes of the committed enrichment files, so editing an enrichment file (or its first
+  arrival) forces the re-embed by itself — no manual version bump to forget. Folding the
+  model tag in means even a same-dims model swap re-embeds and `index_meta` stays honest
+  about which model's vectors are live.
+- **`content_hash` skip-unchanged.** A chunk whose `(chunk_key, content_hash)` already
+  matches the DB is skipped; every batch commits, so a build is resumable and CI-safe
+  (kill it anytime, re-run continues). This is the chat-langchain record-manager lesson.
+  Introducing size-capped `part` rows embedded *only* the new parts because `part 0`
+  keeps the legacy key format.
+- **Why `content` moved into the DB.** Originally the DB stored *no* text — content was
+  re-read from the `_corpus/` snapshot (or fetched live) at answer time. On a deployed
+  Space with no bundled snapshot, that per-section live fetch was the dominant answer
+  latency. Storing `content` lets retrieval hand back the section text in the row it
+  already returns; the migration is safe because empty `content` falls through to the old
+  fetch path, and `_backfill_content` fills pre-existing rows once with a cheap
+  metadata-only UPDATE (no re-embed).
+- **pgvector + tsvector over a dedicated vector DB.** Postgres already holds the pointers
+  and content; putting the vectors (HNSW) and the keyword index (GIN `tsvector`) in the
+  *same* table means a hybrid query is two SELECTs on one connection, an upsert is
+  atomic, and freshness heals content + vector + keyword index in one statement. A
+  separate vector store would add an operational component, a second consistency problem,
+  and cross-store fan-out — for a lean index where "the agent loop, not index
+  sophistication, carries retrieval quality" (see the design doc), it isn't worth it.
+
+## Tool & library choices
+
+| Tool | Role | Why |
+|---|---|---|
+| `psycopg[binary,pool]` (v3) | Neon client + `ConnectionPool` | Modern psycopg with server-side params and a first-party pool; `open=False` + explicit `open()` avoids the deprecated constructor-time open |
+| `pgvector` (Postgres extension) | `vector` column + HNSW index + `<=>` cosine | Keeps dense search inside Postgres, next to pointers and content |
+| Postgres `tsvector` / GIN | Keyword channel | Free lexical search co-located with the vectors; rescues exact symbol matches dense misses |
+| `sentence-transformers` + `BAAI/bge-small-en-v1.5` | Local 384d embeddings (docs + queries) | No API/quota/cost, runs on CPU in CI, hot in-process for query embedding and healing |
+| Neon (serverless Postgres) | Managed store | Free tier fits the corpus; connection cap is the reason for the shared pool |
+
+The model is loaded exactly once via a double-checked lock (`_model()`), not
+`functools.cache`, so two concurrent first queries can't both load the 130 MB model;
+`encode()` itself is safe to call concurrently on the shared instance.
+
+## File by file
+
+- **`__init__.py`** — empty package marker.
+- **`db.py`** — Neon connectivity and schema ownership: `EMBED_DIMS` derived from the
+  model, the `chunks` + `index_meta` `SCHEMA`, `connect()` (single writer) vs `get_pool()`
+  (shared pool with runtime migrations applied at open), `RUNTIME_MIGRATIONS`,
+  `ensure_schema()` (idempotent create + migrate + rebuild-on-dims-change), and the
+  `get_meta`/`set_meta` recipe helpers.
+- **`embed.py`** — the batch index build. `indexed_text()` assembles what actually gets
+  embedded and tsvector'd (symbol + synopsis + gloss + questions + heading + body);
+  `EMBED_RECIPE` and the enrichment stamps drive full-re-embed decisions; `build_index()`
+  walks the snapshot, skips unchanged chunks, upserts new/changed ones in committing
+  batches, backfills `content`, purges stale rows, and stamps the recipe. Local model
+  loading and `embed_texts`/`embed_query` live here too.
+- **`retrieve.py`** — hybrid retrieval: per-kind dense + keyword pools, RRF fusion, the
+  symbol channel, per-pool relevance gating and round-robin interleave, `ef_search`
+  widening, exact-API pinning. Returns pointers with `content` attached. Also
+  `top_distance()` for the input guard.
+- **`hydrate.py`** — turns pointers into content: `hydrate_section` (fast path off the
+  stored `content`, else snapshot-then-live fallback), `hydrate_sections` (concurrent),
+  and `hydrate_page` (whole page, outline-first when oversized). `TORCHDOCS_LIVE_HYDRATE=0`
+  requires the snapshot.
+- **`freshness.py`** — the post-answer stale-while-revalidate pass: TTL gate, live fetch +
+  chunk-by-chunk compare, full in-place heal (content + hash + embedding + tsv), fail-open,
+  and the `TORCHDOCS_FRESHNESS=0` kill switch.
+
+## Related docs
+
+- [`../design-content-and-agent-flow.md`](../design-content-and-agent-flow.md) — the
+  system design this package implements (corpus, tools, session flow, live links).
+- [`../retrieval-gaps-and-improvements.md`](../retrieval-gaps-and-improvements.md) — the
+  measured retrieval-quality analysis behind glosses, hypothetical questions, the
+  `ef_search` fix, and the reranker decision.
+- [`../ingest/README.md`](../ingest/README.md) — the sibling package that builds the
+  snapshot this one indexes.
+- [`../agent/README.md`](../agent/README.md) — the consumer of the pointers this package
+  returns.

docs/ingest/README.md

@@ -0,0 +1,89 @@
+---
+title: "ingest/ — the crawl → snapshot → chunk pipeline"
+kind: reference
+package: ingest
+---
+
+# `ingest/` — the crawl → snapshot → chunk pipeline
+
+Turns the live PyTorch documentation *site* into an on-disk knowledge snapshot: enumerate every page, fetch and clean it to markdown, and split it into heading-granular OKF units that the index step embeds.
+
+## Why this package exists / its boundary
+
+`ingest/` produces the **source of truth** for retrieval: the on-disk `_corpus/` snapshot (one markdown file per page) and, from it, the OKF chunk units. That is the whole job — **it never touches the database.** Embedding, tsvector computation, and the upsert into Neon all live downstream in `index/embed.py`, which *consumes* what this package writes.
+
+The split is deliberate and load-bearing for the design's pointer-based storage (see `../design-content-and-agent-flow.md` §1.2): the DB stores only vectors, tsvectors, and pointers — **no page text**. At query time content is re-read ("hydrated") from the snapshot. So the snapshot isn't a scratch cache you can delete after indexing; it's a runtime dependency that ships with the deploy (PLAN.md M5). Keeping ingest DB-free means the crawl can run anywhere (a laptop, a CI runner) with no credentials, and the snapshot it emits is a plain, greppable, diff-able directory of markdown — the artifact a human can inspect to see exactly what the agent will ever be able to say.
+
+## The pipeline
+
+Three stages, each a separate module, each handing a concrete artifact to the next:
+
+```
+discover.py   →   crawl.py            →   chunk_docs.py        →  (index/embed.py)
+enumerate URLs    fetch + clean + snap    split by heading         embed + upsert
+                                                                   [downstream, not us]
+
+{library:            _corpus/<path>.md         OKF units
+ {url, url, …}}      ---                        ---
+                     url, title, library,       url, anchor, page_title,
+                     content_hash, crawled_at   heading_path, library, kind,
+                     ---                         source_link, content_hash,
+                     <clean markdown body>       part, synopsis
+                                                 ---
+                                                 <section body, code attached>
+```
+
+1. **discover** (`discover.py`) — for each seed in the tiered `SEEDS` map, fetch `objects.inv` (the Sphinx inventory: every documented symbol → exact page + anchor) and `sitemap.xml` (tutorials/guides no inventory covers). Returns `{library: set(page_url)}`. Only inventory entries ending in `.html`, and only sitemap URLs under the seed base, are kept.
+2. **crawl** (`crawl.py`) — for each discovered URL: `fetch_html` (following the meta-refresh redirect stubs `docs/stable/` serves), strip nav/chrome to the main content container, convert HTML → markdown, hash the body, and write `_corpus/<url-path>.md` with YAML frontmatter. If the `content_hash` matches what's already on disk, the page is skipped — nothing downstream re-runs.
+3. **chunk** (`chunk_docs.py`) — read each snapshot page, split it at every heading into `Section`s (code fences stay attached to their section), cut oversized sections at natural seams, and emit OKF unit dicts / files carrying the pointer metadata + a per-page synopsis.
+
+## Design decisions & rationale
+
+**Heading-granular chunks, never a token window.** `split_by_heading` cuts a page at its markdown headings; a chunk is one coherent section with its full `heading_path` breadcrumb. This is the kapa.ai lesson (design doc §Research grounding): a fixed-size token window slices a catalog list ("What LR schedulers exist?") or a worked example in half, and half a list retrieves as noise. Sections are the unit an author already made coherent. Oversized sections are *still* split — but by `split_oversized`, which greedily packs whole paragraphs and whole code fences up to `CHUNK_TARGET_CHARS` (2000, aligned with `index/embed.MAX_EMBED_CHARS`), and only ever cuts *inside* an atom that alone exceeds the budget, at line boundaries. Each resulting part inherits the section's `heading_path` and URL+anchor, so citations stay exact and every part is prefixed with the same symbol/heading synopsis.
+
+**Code blocks are atoms.** `_atoms` walks the fenced-code regex so that the blank lines *inside* a `​```…```​` block never become split points. Half a code block is noise to embed; a fence is indivisible.
+
+**The snapshot is the source of truth.** Covered above under boundary — the DB stores no page text, so the crawl output is not disposable. `save_page` writes human-readable markdown + frontmatter precisely so the snapshot doubles as an auditable knowledge artifact, not an opaque blob.
+
+**`content_hash` idempotency / incrementality.** `save_page` sha256's the *rendered markdown body* and short-circuits if the on-disk file already carries that hash (`return False`, "unchanged"). This is the chat-langchain record-manager lesson (design doc §Research grounding): a weekly recrawl over thousands of mostly-unchanged pages must be cheap. Because chunk identity is `(url, anchor)` and the hash rides through into every unit's frontmatter, an unchanged page re-chunks and re-embeds to nothing downstream. Hashing the *body* (not the raw HTML) is the right choice — it ignores chrome/timestamp churn that doesn't change meaning.
+
+**The tiered seed list.** `SEEDS` is the v1-core tier (core, tutorials, vision, audio) from design doc §1.1. The whole point, restated in the code comment: adding a doc set (ExecuTorch, torchao, …) is *one line here and nothing else*, because every PyTorch domain library is a Sphinx site with its own `objects.inv` that the same discover→crawl→chunk path already handles. The `library` field is stamped on every page and chunk so retrieval can filter/route per question.
+
+**Why the docs SITE, not the source code.** The corpus is what `docs.pytorch.org` serves, never the PyTorch source tree. This supersedes an earlier five-source-modules plan (design doc scope-history note) and is externally validated: LangChain tried indexing their own source code for retrieval and dropped it — raw code chunks retrieved worse than prose docs. Source questions are handled by *referral* instead: `chunk_docs` captures each API page's `[source]` GitHub link (`GITHUB_SOURCE_RE`) as `source_link` metadata, and the agent's `ask_source` tool points at DeepWiki. The docs are the knowledge boundary; code is a link, not a claim.
+
+**OKF units.** Chunks are emitted as Open Knowledge Format files — YAML frontmatter over a markdown body — not raw DB rows. This makes each chunk a human/agent-readable knowledge snapshot you can open and read, consistent with the repo-wide OKF convention (PLAN.md) for hand-authored/generated knowledge documents. (Note the deliberate limit: OKF is *not* used for the DB schema itself, which is pointer-based with typed columns — the units are the on-disk representation the embed step loads.)
+
+## Tool & library choices
+
+| Tool | Where | Why this one |
+|---|---|---|
+| **requests** | `discover.fetch` | Streaming GET with a hard `MAX_PAGE_BYTES` cap enforced mid-download, retry/backoff on 5xx/429 (permanent 4xx raises immediately), and a `torchdocs-agent` UA. Streaming lets a runaway body be abandoned early instead of after a full download. |
+| **beautifulsoup4** | `crawl.extract_main_html`, `discover.redirect_target` | Select the Sphinx content container (`article.pytorch-article` → `div[role=main]` → … → `body`) and `decompose()` the chrome tags (`nav/header/footer/script/style/aside/form/iframe`); also to read the `<meta http-equiv=refresh>` / `<link rel=canonical>` redirect target. |
+| **markdownify** | `crawl.to_markdown` | HTML fragment → markdown with ATX headings (`#`), `*` bullets. ATX headings are what `chunk_docs.HEADING_RE` splits on, so the two modules are coupled by this choice. |
+| **PyYAML** | `crawl` + `chunk_docs` | `safe_dump`/`safe_load` the frontmatter on both snapshot pages and OKF units (`sort_keys=True` for stable diffs). |
+| **Sphinx `objects.inv` parsing** | `discover.parse_objects_inv` | Hand-rolled on stdlib `zlib` + Sphinx's *own* inventory-line regex. The format is `"<name> <domain:role> <priority> <uri> <dispname>"` and **names may contain spaces** (`std:label "PyTorch Contribution Guide"`), so a naive split corrupts the URI — hence the exact upstream regex. Handles the `$` shorthand (uri ending in `$` expands to the entry name). |
+| stdlib `xml.etree` | `discover.parse_sitemap` | Namespace-agnostic sitemap parsing (`_localname` strips ElementTree's `{ns}` prefix), following one level of `<sitemapindex>`, taking only the `<loc>` directly under each `<url>`/`<sitemap>` so nested `<image:loc>` is ignored. |
+
+Two hard-won specifics worth calling out, both documented in code comments:
+
+- **Redirect stubs.** `docs.pytorch.org/docs/stable/<…>` serves a "Redirecting…" page whose real content is behind a `<meta refresh>` — which `requests` does *not* follow. A naive fetch captured empty stubs for 3,435 of 4,517 pages (the entire core API reference). `fetch_html` detects the refresh meta and follows it, hop-bounded and loop-protected, while the caller keeps the *original* URL as the citation key so stable URLs stay stable.
+- **The size cap is generous on purpose.** `MAX_PAGE_BYTES` is 20 MB, not 5 — a 5 MB cap silently dropped legitimate image-heavy tutorials (dcgan_faces, hybrid_demucs) from the index on the 2026-07-08 build.
+
+## File by file
+
+All four modules are **implemented** (with a full test suite under `tests/ingest/` — `test_discover.py`, `test_crawl.py`, `test_chunk_docs.py`). Note PLAN.md's M2.1 checkboxes for these files were still unticked at the time of writing; the code is present and substantive, so treat the boxes as lagging the work, not the reverse.
+
+- **`__init__.py`** — empty (0 bytes). Marks `ingest` as a package; there is no package-level API surface, callers import the modules directly (e.g. `from ingest.discover import fetch_html`).
+
+- **`discover.py`** — page enumeration. Pure parsers (`parse_objects_inv`, `parse_sitemap`, `is_sitemap_index`, `redirect_target`) take bytes/text and return entries, so they test offline; only `fetch`/`fetch_html`/`discover` touch the network. `discover()` walks `SEEDS`, tries inventory then sitemap per seed, and *tolerates only network errors* — a parse error (PyTorch changing the inventory format) propagates and fails the run loudly rather than silently shrinking the index. Also home to the resilient `fetch` (retry/backoff/size-cap) and the meta-refresh-following `fetch_html`.
+
+- **`crawl.py`** — fetch → clean → snapshot. `extract_main_html` selects the content container and strips chrome; `to_markdown` converts; `page_path` maps a URL to a stable on-disk path (host dropped, path preserved, `.html`→`.md`); `save_page` hashes the body and writes-or-skips; `crawl` drives the whole set with a politeness delay (`TORCHDOCS_CRAWL_DELAY`, default 0.2s) and per-page exception isolation (one bad page must not kill the crawl). Returns per-library change counts.
+
+- **`chunk_docs.py`** — snapshot page → OKF units. `split_by_heading` builds `Section`s with a breadcrumb stack; `clean_heading` recovers the *true* Sphinx anchor from the surviving headerlink (`[¶](#sgd …)`) rather than re-slugifying; `split_oversized`/`_atoms`/`_hard_split` handle over-budget sections without cutting code fences; `page_kind` classifies url → `api`/`tutorial`/`guide`; `page_synopsis` deterministically extracts the page's first real description sentence (the docstring summary buried after Sphinx's `:` definition-line marker) — API pages only, since terse reference pages otherwise embed far from descriptive questions. `chunk_page` assembles the unit dicts; `write_units` serializes them, always suffixing the filename with the enumerate index so two same-named sections (e.g. two "Parameters" headings) can't clobber each other on disk.
+
+- **`watch.py`** — **not present / not yet built.** The design doc (§1.3) and PLAN.md M5 specify a release watcher that polls the `pytorch/pytorch` GitHub Releases API and kicks an immediate recrawl on a new stable tag. It does not exist in `ingest/` yet; the scheduled/triggered recrawl orchestration is M5 work.
+
+## Related docs
+
+- `../design-content-and-agent-flow.md` — the design rationale this package implements, especially §1 (content extraction), §1.1 (the tiered corpus/seed list), and §1.2–1.3 (pipeline properties and recrawl cadence).
+- `../index/README.md` — the sibling `index/` package (`embed.py`, `hydrate.py`) that consumes the snapshot: embeds the OKF units into Neon and hydrates content back at query time.

docs/scripts/README.md

@@ -0,0 +1,248 @@
+---
+title: "scripts/ — operator CLI entrypoints and CI jobs"
+kind: reference
+package: scripts
+---
+
+# `scripts/` — the command-line surface
+
+The operator/CLI layer: every way a human (or a GitHub Actions job) drives the
+system by hand — build and refresh the index, generate synthetic index-side
+content, evaluate, calibrate the guard, smoke-test, and dump inventories for
+diagnostics.
+
+## Why this package exists / its boundary
+
+These scripts are **thin wrappers**. They own no business logic of their own —
+they parse argv, `load_dotenv()`, fail fast on a missing env var, and then call
+into `ingest/`, `index/`, and `agent/`. The real work lives there:
+
+- `build_index.py` calls `ingest.discover.discover`, `ingest.crawl.crawl`, and
+  `index.embed.build_index` — it just sequences them and prints timing.
+- `ask.py` calls `agent.guard.guard` then `agent.loop.answer_agentic`.
+- `search.py` calls `index.retrieve.retrieve` + `index.hydrate.hydrate_section`.
+- `calibrate_guard.py` calls `index.retrieve.top_distance`.
+
+The one piece of genuine logic that *does* live here is the batch-job
+scaffolding in `generate_glosses.py` (`git_checkpoint`, batching, resume) —
+because it is an operational concern (surviving a cancelled CI run), not a
+system concern. `generate_questions.py` imports that scaffolding rather than
+duplicating it.
+
+Heavy imports are deliberately done **inside `main()`**, after the env-var
+check, so a missing `NEON_URL` prints one clean line instead of a stack trace
+from importing a DB module at file scope.
+
+`__init__.py` is empty — it exists only to make `scripts` a package so
+`generate_questions.py` can `from scripts.generate_glosses import ...`.
+
+## Script catalog
+
+| Script | What it does | Run it | Workflow(s) that invoke it |
+|---|---|---|---|
+| `build_index.py` | End-to-end index build: discover → crawl → chunk+embed → Neon. Every stage resumable; every embed batch commits. | `python scripts/build_index.py [--skip-crawl] [--skip-embed] [--libraries core,vision]` | **build-index.yml** (`--skip-embed` then `--skip-crawl`), **backfill-content.yml** (`--skip-crawl`) |
+| `ask.py` | Answer one question end-to-end from the CLI: guard, then the full agent tool loop; prints answer + citations + referrals. | `python scripts/ask.py "how do I build a CNN?"` | — (local only) |
+| `search.py` | Retrieval-only acceptance test: run hybrid retrieve, print the ranked pointers + a snippet of the top hit. No LLM. | `python scripts/search.py "scaled_dot_product_attention" [-k 8] [--library vision]` | — (local only) |
+| `calibrate_guard.py` | Recompute the guard's topicality cutoff against the **live** index: distances for 100 on-topic + 100 off-topic + borderline probes, plus a suggested threshold. Prints only. | `python scripts/calibrate_guard.py` | **calibrate-guard.yml** |
+| `generate_glosses.py` | Batch LLM job: a 1-sentence Contextual-Retrieval gloss per api page → `index/glosses.jsonl`. Batched, resumable, per-batch flush, optional `--push` checkpoint. | `python scripts/generate_glosses.py [--limit N] [--batch N] [--push]` | **build-index.yml**, **generate-glosses.yml** |
+| `generate_questions.py` | Batch LLM job: a few QuOTE-style hypothetical questions per api page → `index/questions.jsonl`. Same scaffolding as glosses. | `python scripts/generate_questions.py [--limit N] [--batch N] [--push]` | **build-index.yml**, **generate-glosses.yml** |
+| `smoke.py` | Preflight: one Neon write/read, one Gemini call, one local bge-small embedding, one optional Anthropic call. Exits non-zero on any required failure. | `python scripts/smoke.py` | — (local preflight) |
+| `smoke_space.py` | Post-deploy health check against the live HF Space: poll runtime until RUNNING, ask a real question via the Gradio API, fail on an LLM/transport error marker. | `python scripts/smoke_space.py` | **sync-to-hf.yml** |
+| `dump_docs_inventory.py` | Dump external ground truth: every documented symbol/page the docs SITE publishes (`objects.inv` + sitemap) → `eval/docs_inventory.jsonl`. Needs network to docs.pytorch.org. | `python scripts/dump_docs_inventory.py` | **build-index.yml** (inventory job) |
+| `dump_index_manifest.py` | Dump internal ground truth: every distinct page actually in the `chunks` table → `eval/index_manifest.jsonl`. Needs `NEON_URL`. | `python scripts/dump_index_manifest.py` | **build-index.yml** (inventory job) |
+| `coverage_diff.py` | Diff the two dumps: pages the site publishes but our index is missing. A non-empty gap is a pipeline bug. | `python scripts/coverage_diff.py` | **build-index.yml** (inventory job) |
+| `__init__.py` | Empty package marker (enables `from scripts.generate_glosses import ...`). | — | — |
+
+Note: the **eval** jobs (`eval.yml`) run `python -m eval.run_retrieval`,
+`eval.diagnose_retrieval`, `eval.run_agentic`, `eval.run_judge` — those live in
+the `eval/` package, not here. `ci.yml` runs `ruff` + `pytest`, and
+`security.yml` runs Trivy; neither invokes a script in this package.
+
+## Operational flows
+
+### (a) Build / refresh the index
+
+`build_index.py` is the whole pipeline. Locally you run it plain; in CI the
+**build-index.yml** workflow splits it so glossing can happen *between* crawl
+and embed:
+
+1. `build_index.py --skip-embed` — crawl only, refresh the on-disk `_corpus`
+   snapshot, never touch Neon (so the crawl-only path doesn't even require
+   `NEON_URL`).
+2. `generate_glosses.py --limit 0 --push` then `generate_questions.py --limit 0
+   --push` — enrich only the pages that don't have a gloss/question set yet.
+3. `build_index.py --skip-crawl` — embed the snapshot into Neon. A changed
+   `glosses.jsonl`/`questions.jsonl` bumps the embed recipe, so only the touched
+   pages re-embed.
+
+Resumability is the design point: crawling skips unchanged pages, embedding
+skips chunks whose hash is already in the DB, every embed batch commits. Kill it
+and re-run — it continues. Both **build-index.yml** and **backfill-content.yml**
+share the `concurrency: build-index` lock so two writers never race the DB.
+`--libraries core,vision` restricts the run to a subset of the seed list.
+
+### (b) Generate synthetic index-side content (glosses / questions)
+
+`generate_glosses.py` and `generate_questions.py` are the two batch LLM jobs.
+Both walk every `api`-kind page in the snapshot, batch them into LLM calls, and
+append JSON lines to `index/{glosses,questions}.jsonl`, flushing after each
+batch. They are **resumable**: URLs already present in the output file are
+skipped, so a rate-limited death just means "run it again." Core-torch pages are
+glossed first, so a partial run still covers the pages that matter most. After 5
+failed batches they stop, assuming the provider is down. `generate_questions.py`
+imports `api_pages`, `existing_urls_of`, and `git_checkpoint` directly from
+`generate_glosses.py` — one pipeline shape, not two.
+
+The `--push` flag turns on `git_checkpoint`: commit **and push** the jsonl after
+every batch. See rationale below.
+
+### (c) Evaluate
+
+Retrieval acceptance from the CLI is `search.py` (pointers only, no LLM) and
+end-to-end answering is `ask.py`. The scored benchmarks
+(recall/MRR, agentic, judge) are the `eval/` package run via `eval.yml`, not
+scripts here. `coverage_diff.py` is the eval-adjacent check that the corpus the
+index holds actually matches the corpus the docs site publishes.
+
+### (d) Calibrate the guard
+
+`calibrate_guard.py` re-derives the topicality distance cutoff against the live
+index. It runs three groups — 100 on-topic (must all pass), 100 off-topic
+(should all block), and a handful of borderline/injection probes to eyeball —
+through the guard's `top_distance` path and prints every distance plus a
+suggested `TORCHDOCS_TOPICALITY_MAX_DISTANCE` (the midpoint between the worst
+on-topic and best off-topic distance). It **only prints**; a threshold change is
+a policy decision, so a human reads the log and edits the constant in
+`agent/guard.py` by hand. Run it (via **calibrate-guard.yml**) after any corpus
+change big enough to shift the distance distribution — a re-embed or a new doc
+set.
+
+### (e) Smoke-test — locally and post-deploy
+
+Two different tests for two different moments:
+
+- `smoke.py` — **before building anything**, verify each external connection
+  works: Neon write/read, Gemini, local embedding, optional Anthropic. A missing
+  key skips (Anthropic) or fails with a clear message rather than a traceback.
+- `smoke_space.py` — **after deploy**, verify the live Space actually answers.
+  It runs in **sync-to-hf.yml** right after the push (GitHub Actions can reach
+  both the Space and the LLM provider; the dev sandbox can't). It polls the HF
+  runtime API until RUNNING, asks a real question through the Gradio
+  `/respond` endpoint, and fails the job if the answer contains an
+  LLM/transport error marker — so a broken deploy is a red check, not a Space
+  that silently serves errors. An empty-index answer warns but doesn't fail
+  (that's a separate subsystem).
+
+### (f) Inventory / diagnostics
+
+Three scripts build and compare two ground-truth files, all in the inventory job
+of **build-index.yml**:
+
+- `dump_docs_inventory.py` → `eval/docs_inventory.jsonl` — what the site
+  publishes (external truth; must run where docs.pytorch.org is reachable).
+- `dump_index_manifest.py` → `eval/index_manifest.jsonl` — what our index
+  actually holds (internal truth; needs `NEON_URL`).
+- `coverage_diff.py` — pages in the first but not the second: a page the docs
+  document that our system can never retrieve. Both dumps are committed so the
+  diff can run offline.
+
+## Design decisions & rationale
+
+**Batch git-checkpoint (`git_checkpoint` + `--push`).** A gloss/question pass
+over the ~3.6K-page corpus is a multi-hour LLM job. The batches are flushed to
+disk, but on a GitHub runner that file only reaches the repo via the workflow's
+final commit step — so a cancel or a job timeout part-way through throws away
+everything generated in the run. `--push` commits and pushes the jsonl every few
+batches instead, so a killed run keeps its progress. Every git failure (unset
+identity, a push race with a concurrent enrichment run, a rebase conflict) is
+logged and swallowed — a missed checkpoint just defers to the final commit; it
+must **never** kill the long run. The committer identity is injected per-command
+(`git -c user.name=...`) so no global config or extra workflow step is needed,
+and the commit message carries `[skip ci]` so a checkpoint push doesn't kick off
+a CI run. It is opt-in (`--push`) so local runs never commit.
+
+**`--push` off by default.** Local runs of the batch jobs should produce a
+jsonl and nothing else; only CI, which needs to persist progress across a
+possible cancellation, turns pushing on.
+
+**Smoke tests exist to fail fast on deploy.** `smoke.py` stops you before an
+hour of crawling if a credential is wrong; `smoke_space.py` turns a broken
+deploy into a red check on the very run that produced it (one workflow does both
+push and verify). Both do exactly one real round-trip per subsystem — enough to
+prove the wire works, cheap enough to run every time.
+
+**Guard recalibration is manual after corpus changes.** The topicality
+threshold is a distance in embedding space; re-embedding or adding a doc set
+shifts the whole distribution. `calibrate_guard.py` measures the new
+distribution and *suggests* a cutoff, but a human commits the constant — where
+to draw the on-topic/off-topic line is a policy call, and the script prints the
+overlap so you can see when the groups aren't cleanly separable and refuse to
+split the difference blindly.
+
+## Tool & library choices
+
+- **`argparse` + `main() -> int` + `sys.exit(main())`** everywhere. Exit codes
+  are load-bearing: they make each script a CI gate. `smoke*.py`,
+  `coverage_diff.py`, and the batch jobs return non-zero on failure so a
+  workflow step goes red. The batch jobs treat partial success as success
+  (resumable) and total failure as loud (`return 0 if written else 1`).
+- **`python-dotenv`** — every script that touches a credential calls
+  `load_dotenv()` first, so a local `.env` and CI secrets are configured the
+  same way.
+- **Reuse of the real modules, not reimplementation.** The scripts import the
+  exact code the app uses: `agent.guard`/`agent.loop` (ask), `index.retrieve`
+  (search + calibrate), `index.embed`/`ingest.*` (build), `agent.llm._raw_completion`
+  for the batch LLM calls. The batch jobs therefore ride the same provider
+  dispatch and fallback chain (OpenRouter/hy3 → Gemini) the workflows configure
+  via env — nothing is stubbed, so a CLI green means the production path is green.
+- **`gradio_client`** in `smoke_space.py` to hit the Space through its real
+  public API, tolerating the `hf_token`→`token` kwarg rename across versions.
+
+## File by file
+
+- **`__init__.py`** — empty; makes `scripts` an importable package so the two
+  batch jobs can share code.
+- **`ask.py`** — one question end-to-end from the CLI. Guards the input first
+  (bails with the guard's reason if it fails), then runs `answer_agentic` and
+  pretty-prints the answer, citations (title › anchor + URL), and referrals.
+- **`build_index.py`** — the overnight-safe full pipeline. Fails fast if
+  `NEON_URL` is missing (unless `--skip-embed`, which never touches Neon).
+  Stamps an `index_version` from the crawl timestamp. `--skip-crawl` re-embeds
+  the existing snapshot; `--skip-embed` refreshes the snapshot without embedding.
+- **`calibrate_guard.py`** — reads the 100/100 eval question files plus inline
+  borderline probes, measures `top_distance` per question, prints sorted
+  distances + per-group stats, and suggests a threshold (or flags an overlap).
+  Print-only by design.
+- **`coverage_diff.py`** — set-difference of two committed jsonl dumps; reports
+  site pages missing from the index, bucketed by library. Returns 1 if either
+  dump is absent. No network, no DB.
+- **`dump_docs_inventory.py`** — reads each seed's Sphinx `objects.inv` (kept
+  roles: classes/functions/methods/attributes/data + `std:doc`) and sitemap,
+  de-dups, and writes the external ground-truth inventory. Must run where
+  docs.pytorch.org is reachable.
+- **`dump_index_manifest.py`** — one SQL pass over the `chunks` table, rolled up
+  per page (title/kind/library + up to 40 headings + chunk count), written as
+  the internal manifest. Needs `NEON_URL`; run from Actions.
+- **`generate_glosses.py`** — the batch-job home base: `api_pages` (snapshot →
+  api pages, core first), batched LLM calls, `parse_glosses` (tolerant JSON
+  extraction), `existing_urls_of`/resume, and `git_checkpoint`/`--push`. Writes
+  `index/glosses.jsonl`.
+- **`generate_questions.py`** — the QuOTE-style twin. Imports the scaffolding
+  from `generate_glosses.py` and only differs in prompt, parser
+  (`parse_questions`), batch size, and output (`index/questions.jsonl`).
+- **`search.py`** — retrieval acceptance test. `retrieve(..., debug=True)`, print
+  ranked pointers, hydrate and snippet the top hit. Returns 1 if the index is
+  empty. No LLM.
+- **`smoke.py`** — four preflight checks (Neon, Gemini, local embedding, optional
+  Anthropic), each catching its own exception so one broken connection reports
+  instead of crashing the run. Exits non-zero if any required check fails.
+- **`smoke_space.py`** — post-deploy health check: poll the HF runtime API to
+  RUNNING (or a failure stage), call the Gradio `/respond` endpoint, and
+  fail on error markers in the answer. Empty-index → warn, not fail.
+
+## Related docs
+
+- [`../design-content-and-agent-flow.md`](../design-content-and-agent-flow.md) — the system these scripts drive (pipeline, agent tools, session flow).
+- [`../deploy-hf-spaces.md`](../deploy-hf-spaces.md) — the deploy that `smoke_space.py` verifies.
+- [`../index/README.md`](../index/README.md) — `embed`/`retrieve`/`hydrate`, called by `build_index.py`, `search.py`, `calibrate_guard.py`.
+- [`../agent/README.md`](../agent/README.md) — `guard`/`loop`/`llm`, called by `ask.py` and the batch jobs.

site/index.html

@@ -0,0 +1,530 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>TorchDocs Agent — grounded PyTorch answers</title>
+<style>
+  :root {
+    --sans: system-ui, -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
+    --mono: ui-monospace, "SF Mono", "JetBrains Mono", "Cascadia Code", Menlo, Consolas, monospace;
+
+    /* warm, ember-biased neutrals — the greys are literally the reasoning-trace colour */
+    --bg: #17120E;
+    --bg-raised: #211a14;
+    --bg-inset: #100c09;
+    --text: #F3EDE6;
+    --muted: #A99C8D;      /* the grey a trace line is drawn in */
+    --faint: #6E6255;
+    --border: rgba(243, 237, 230, 0.11);
+    --border-strong: rgba(243, 237, 230, 0.2);
+    --ember: #FF5A36;
+    --ember-2: #FF8A5B;
+    --verified: #64C296;   /* semantic: grounded/cited — NOT the accent */
+    --shadow: 0 24px 60px -30px rgba(0,0,0,0.75);
+  }
+  @media (prefers-color-scheme: light) {
+    :root {
+      --bg: #FAF8F5;
+      --bg-raised: #FFFFFF;
+      --bg-inset: #F1EBE4;
+      --text: #211B16;
+      --muted: #6E6357;
+      --faint: #9B9184;
+      --border: rgba(34, 27, 22, 0.12);
+      --border-strong: rgba(34, 27, 22, 0.22);
+      --ember: #E5431F;
+      --ember-2: #FF6A3D;
+      --verified: #2E9367;
+      --shadow: 0 24px 60px -34px rgba(60, 30, 12, 0.28);
+    }
+  }
+  :root[data-theme="dark"] {
+    --bg: #17120E; --bg-raised: #211a14; --bg-inset: #100c09;
+    --text: #F3EDE6; --muted: #A99C8D; --faint: #6E6255;
+    --border: rgba(243,237,230,0.11); --border-strong: rgba(243,237,230,0.2);
+    --ember: #FF5A36; --ember-2: #FF8A5B; --verified: #64C296;
+    --shadow: 0 24px 60px -30px rgba(0,0,0,0.75);
+  }
+  :root[data-theme="light"] {
+    --bg: #FAF8F5; --bg-raised: #FFFFFF; --bg-inset: #F1EBE4;
+    --text: #211B16; --muted: #6E6357; --faint: #9B9184;
+    --border: rgba(34,27,22,0.12); --border-strong: rgba(34,27,22,0.22);
+    --ember: #E5431F; --ember-2: #FF6A3D; --verified: #2E9367;
+    --shadow: 0 24px 60px -34px rgba(60,30,12,0.28);
+  }
+
+  * { box-sizing: border-box; }
+  body {
+    margin: 0; background: var(--bg); color: var(--text);
+    font-family: var(--sans); line-height: 1.6;
+    -webkit-font-smoothing: antialiased; text-rendering: optimizeLegibility;
+  }
+  .wrap { width: 100%; max-width: 1120px; margin: 0 auto; padding: 0 24px; }
+  a { color: inherit; }
+  h1, h2, h3 { text-wrap: balance; margin: 0; }
+
+  .eyebrow {
+    font-family: var(--mono); font-size: 0.72rem; letter-spacing: 0.16em;
+    text-transform: uppercase; color: var(--ember); font-weight: 600;
+  }
+  .sub-eyebrow { color: var(--faint); }
+
+  /* ---- top bar ---- */
+  header.bar {
+    position: sticky; top: 0; z-index: 20;
+    backdrop-filter: blur(12px);
+    background: color-mix(in srgb, var(--bg) 82%, transparent);
+    border-bottom: 1px solid var(--border);
+  }
+  .bar-inner { display: flex; align-items: center; gap: 24px; height: 62px; }
+  .brand { display: flex; align-items: center; gap: 10px; font-weight: 700; letter-spacing: -0.01em; }
+  .flame {
+    width: 22px; height: 22px; flex: none;
+    filter: drop-shadow(0 0 10px color-mix(in srgb, var(--ember) 55%, transparent));
+  }
+  nav.links { display: flex; gap: 22px; margin-left: auto; align-items: center; }
+  nav.links a {
+    text-decoration: none; color: var(--muted); font-size: 0.92rem;
+    font-weight: 500; transition: color .15s;
+  }
+  nav.links a:hover { color: var(--text); }
+  .nav-hide { }
+  @media (max-width: 760px) { .nav-hide { display: none; } nav.links { gap: 14px; } }
+
+  .btn {
+    display: inline-flex; align-items: center; gap: 8px;
+    font-family: var(--sans); font-weight: 600; font-size: 0.92rem;
+    text-decoration: none; border-radius: 8px; padding: 9px 15px;
+    border: 1px solid transparent; cursor: pointer; transition: transform .12s, box-shadow .2s, background .2s;
+  }
+  .btn-primary {
+    color: #fff; background: linear-gradient(135deg, var(--ember), var(--ember-2));
+    box-shadow: 0 8px 22px -10px color-mix(in srgb, var(--ember) 70%, transparent);
+  }
+  .btn-primary:hover { transform: translateY(-1px); box-shadow: 0 12px 26px -10px color-mix(in srgb, var(--ember) 75%, transparent); }
+  .btn-ghost { color: var(--text); border-color: var(--border-strong); background: transparent; }
+  .btn-ghost:hover { border-color: var(--ember); color: var(--ember); }
+
+  .toggle {
+    width: 34px; height: 34px; border-radius: 8px; border: 1px solid var(--border-strong);
+    background: transparent; color: var(--muted); cursor: pointer; display: grid; place-items: center;
+    flex: none; transition: color .15s, border-color .15s;
+  }
+  .toggle:hover { color: var(--ember); border-color: var(--ember); }
+
+  /* ---- hero ---- */
+  .hero { padding: 74px 0 40px; }
+  .hero-grid { display: grid; grid-template-columns: 1.05fr 0.95fr; gap: 54px; align-items: center; }
+  @media (max-width: 900px) { .hero-grid { grid-template-columns: 1fr; gap: 40px; } .hero { padding: 46px 0 20px; } }
+  h1.head {
+    font-size: clamp(2.3rem, 5.2vw, 3.9rem); line-height: 1.05; letter-spacing: -0.03em;
+    font-weight: 800; margin: 18px 0 0;
+  }
+  h1.head .flame-word { color: var(--ember); }
+  .lede { color: var(--muted); font-size: clamp(1.02rem, 1.6vw, 1.2rem); margin: 20px 0 0; max-width: 34em; }
+  .cta-row { display: flex; flex-wrap: wrap; gap: 12px; margin-top: 30px; }
+  .cta-row .btn { padding: 12px 20px; font-size: 1rem; }
+  .disclaimer { margin-top: 20px; font-size: 0.82rem; color: var(--faint); }
+
+  /* ---- the live answer card (the thesis) ---- */
+  .card {
+    background: var(--bg-inset); border: 1px solid var(--border-strong);
+    border-radius: 14px; box-shadow: var(--shadow); overflow: hidden;
+  }
+  .card-top {
+    display: flex; align-items: center; gap: 8px; padding: 12px 16px;
+    border-bottom: 1px solid var(--border); background: color-mix(in srgb, var(--bg-raised) 60%, transparent);
+  }
+  .dot { width: 11px; height: 11px; border-radius: 50%; background: var(--border-strong); }
+  .card-title { margin-left: 8px; font-family: var(--mono); font-size: 0.76rem; color: var(--faint); letter-spacing: 0.02em; }
+  .card-body { padding: 20px 20px 22px; min-height: 292px; }
+  .ask { font-family: var(--mono); font-size: 0.86rem; color: var(--text); margin-bottom: 16px; }
+  .ask .q { color: var(--ember); }
+  .trace { font-family: var(--mono); font-size: 0.83rem; color: var(--muted); line-height: 1.85; }
+  .trace .row { opacity: 0; transform: translateY(3px); transition: opacity .35s, transform .35s; }
+  .trace .row.show { opacity: 1; transform: none; }
+  .answer {
+    margin-top: 16px; opacity: 0; transition: opacity .5s; font-size: 0.95rem; line-height: 1.6;
+  }
+  .answer.show { opacity: 1; }
+  .answer code, .code {
+    font-family: var(--mono); background: color-mix(in srgb, var(--ember) 12%, transparent);
+    color: var(--ember); padding: 1px 5px; border-radius: 4px; font-size: 0.86em;
+  }
+  .snippet {
+    margin: 12px 0 4px; padding: 11px 13px; border-radius: 8px; background: var(--bg);
+    border: 1px solid var(--border); font-family: var(--mono); font-size: 0.8rem;
+    color: var(--text); overflow-x: auto; line-height: 1.5;
+  }
+  .snippet .kw { color: var(--ember); }
+  .snippet .cm { color: var(--faint); }
+  .cite {
+    display: inline-flex; align-items: center; gap: 6px; margin-top: 14px;
+    font-family: var(--mono); font-size: 0.75rem; color: var(--verified);
+    border: 1px solid color-mix(in srgb, var(--verified) 40%, transparent);
+    border-radius: 999px; padding: 4px 11px; text-decoration: none;
+  }
+  .cite:hover { background: color-mix(in srgb, var(--verified) 12%, transparent); }
+  .caret { display: inline-block; width: 8px; height: 1.05em; vertical-align: -0.16em;
+    background: var(--ember); margin-left: 2px; animation: blink 1.05s steps(1) infinite; }
+  @keyframes blink { 50% { opacity: 0; } }
+
+  /* ---- generic section ---- */
+  section.band { padding: 64px 0; border-top: 1px solid var(--border); }
+  .band-head { max-width: 40em; margin-bottom: 40px; }
+  .band-head h2 { font-size: clamp(1.7rem, 3vw, 2.3rem); letter-spacing: -0.02em; font-weight: 800; margin-top: 12px; }
+  .band-head p { color: var(--muted); font-size: 1.06rem; margin: 14px 0 0; }
+
+  /* thesis strip */
+  .thesis { font-size: clamp(1.3rem, 2.6vw, 1.9rem); font-weight: 600; letter-spacing: -0.02em; line-height: 1.4; max-width: 20em; }
+  .thesis .hl { color: var(--ember); }
+  .thesis .em2 { color: var(--verified); }
+
+  /* feature grid */
+  .features { display: grid; grid-template-columns: repeat(3, 1fr); gap: 18px; }
+  @media (max-width: 860px) { .features { grid-template-columns: 1fr; } }
+  .feat {
+    background: var(--bg-raised); border: 1px solid var(--border); border-radius: 12px;
+    padding: 22px 22px 24px; transition: border-color .2s, transform .2s;
+  }
+  .feat:hover { border-color: var(--border-strong); transform: translateY(-2px); }
+  .feat .tag { font-family: var(--mono); font-size: 0.7rem; letter-spacing: 0.12em; text-transform: uppercase; color: var(--faint); }
+  .feat h3 { font-size: 1.12rem; margin: 12px 0 8px; letter-spacing: -0.01em; }
+  .feat p { color: var(--muted); font-size: 0.95rem; margin: 0; }
+  .feat .rule { width: 26px; height: 3px; border-radius: 2px; background: var(--ember); margin-bottom: 16px; }
+
+  /* pipeline (a real sequence) */
+  .pipe { display: grid; grid-template-columns: repeat(4, 1fr); gap: 0; }
+  @media (max-width: 860px) { .pipe { grid-template-columns: 1fr; } }
+  .step { position: relative; padding: 24px 22px; border: 1px solid var(--border); background: var(--bg-raised); }
+  .pipe .step:not(:last-child) { border-right: none; }
+  @media (max-width: 860px) { .pipe .step:not(:last-child) { border-right: 1px solid var(--border); border-bottom: none; } }
+  .step:first-child { border-radius: 12px 0 0 12px; }
+  .step:last-child { border-radius: 0 12px 12px 0; }
+  @media (max-width: 860px) {
+    .step:first-child { border-radius: 12px 12px 0 0; }
+    .step:last-child { border-radius: 0 0 12px 12px; }
+  }
+  .step .n { font-family: var(--mono); font-size: 0.78rem; color: var(--ember); font-weight: 600; }
+  .step h3 { font-family: var(--mono); font-size: 0.98rem; margin: 10px 0 8px; letter-spacing: 0.01em; }
+  .step p { color: var(--muted); font-size: 0.88rem; margin: 0; }
+  .step .arrow { position: absolute; right: -9px; top: 50%; transform: translateY(-50%); z-index: 2;
+    width: 18px; height: 18px; color: var(--ember); }
+  @media (max-width: 860px) { .step .arrow { display: none; } }
+
+  /* use it — two tracks */
+  .tracks { display: grid; grid-template-columns: 1fr 1fr; gap: 22px; align-items: start; }
+  @media (max-width: 860px) { .tracks { grid-template-columns: 1fr; } }
+  .track { background: var(--bg-raised); border: 1px solid var(--border); border-radius: 12px; padding: 26px; }
+  .track h3 { font-size: 1.18rem; letter-spacing: -0.01em; }
+  .track .kicker { font-family: var(--mono); font-size: 0.72rem; letter-spacing: 0.12em; text-transform: uppercase; color: var(--ember); }
+  .track ol { margin: 16px 0 0; padding-left: 20px; color: var(--muted); }
+  .track ol li { margin-bottom: 9px; }
+  .track ol li::marker { color: var(--faint); font-family: var(--mono); font-size: 0.85em; }
+  pre.block {
+    margin: 16px 0 0; padding: 16px; border-radius: 10px; background: var(--bg-inset);
+    border: 1px solid var(--border); overflow-x: auto;
+    font-family: var(--mono); font-size: 0.82rem; line-height: 1.7; color: var(--text);
+  }
+  pre.block .cm { color: var(--faint); }
+  pre.block .kw { color: var(--ember); }
+  .ex { display: flex; flex-wrap: wrap; gap: 8px; margin-top: 16px; }
+  .ex span {
+    font-family: var(--mono); font-size: 0.78rem; color: var(--muted);
+    border: 1px solid var(--border); border-radius: 999px; padding: 5px 11px; background: var(--bg);
+  }
+
+  /* docs links */
+  .doclist { display: grid; grid-template-columns: repeat(3, 1fr); gap: 12px; }
+  @media (max-width: 860px) { .doclist { grid-template-columns: 1fr 1fr; } }
+  @media (max-width: 520px) { .doclist { grid-template-columns: 1fr; } }
+  .doc {
+    display: block; text-decoration: none; padding: 16px 18px; border-radius: 10px;
+    border: 1px solid var(--border); background: var(--bg-raised); transition: border-color .2s, transform .2s;
+  }
+  .doc:hover { border-color: var(--ember); transform: translateY(-2px); }
+  .doc .p { font-family: var(--mono); color: var(--ember); font-size: 0.9rem; }
+  .doc .d { color: var(--muted); font-size: 0.86rem; margin-top: 5px; }
+
+  /* footer */
+  footer { border-top: 1px solid var(--border); padding: 40px 0 56px; margin-top: 8px; }
+  .foot { display: flex; flex-wrap: wrap; gap: 18px 30px; align-items: center; color: var(--muted); font-size: 0.9rem; }
+  .foot a { color: var(--muted); text-decoration: none; }
+  .foot a:hover { color: var(--ember); }
+  .foot .sep { flex: 1; }
+
+  @media (prefers-reduced-motion: reduce) {
+    .trace .row, .answer { opacity: 1 !important; transform: none !important; transition: none !important; }
+    .caret { animation: none; }
+    .btn, .feat, .doc { transition: none; }
+  }
+</style>
+</head>
+<body>
+
+
+<header class="bar">
+  <div class="wrap bar-inner">
+    <span class="brand">
+      <svg class="flame" viewBox="0 0 24 24" aria-hidden="true"><path fill="currentColor" style="color:var(--ember)" d="M13.5 1.5c.4 3-1.2 4.7-2.8 6.3C9 9.6 7.2 11.3 7.2 14.2a4.8 4.8 0 0 0 9.6.3c0-1.6-.7-2.9-1.4-4-.3 1-1 1.6-1.8 1.7.8-2.3.3-5.3-2-8.4l1.9-2.3ZM10 15.4c0-1.3.8-2.2 1.6-3 .3 1 .1 2 .9 2.9.5-.3.8-.8.9-1.4.5.7.9 1.5.9 2.4a2.6 2.6 0 0 1-5.2 0c0-.5.1-.9.3-1.3.1.6.3 1 .6 1.3-.1-.9-.4-1.6 0-2.3-.1.7 0 1.1 0 1.4Z"/></svg>
+      TorchDocs Agent
+    </span>
+    <nav class="links">
+      <a class="nav-hide" href="#what">What</a>
+      <a class="nav-hide" href="#how">How it works</a>
+      <a class="nav-hide" href="#use">Use it</a>
+      <a href="https://github.com/eliezeravihail/TorchDocsAgent/tree/main/docs">Docs</a>
+      <button class="toggle" id="themeBtn" aria-label="Toggle colour theme" title="Toggle theme">
+        <svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><circle cx="12" cy="12" r="4.5"/><path d="M12 2v2M12 20v2M4 12H2M22 12h-2M5 5l1.5 1.5M17.5 17.5 19 19M19 5l-1.5 1.5M6.5 17.5 5 19"/></svg>
+      </button>
+    </nav>
+  </div>
+</header>
+
+<main>
+  <!-- HERO -->
+  <section class="hero">
+    <div class="wrap hero-grid">
+      <div>
+        <div class="eyebrow">PyTorch docs · grounded RAG agent</div>
+        <h1 class="head">Ask PyTorch anything.<br>Get an answer <span class="flame-word">grounded in the docs</span>.</h1>
+        <p class="lede">A chat agent that answers from the official PyTorch documentation — every claim traced to the exact page it cites, and an honest referral when the question reaches past the docs. No invented APIs.</p>
+        <div class="cta-row">
+          <a class="btn btn-primary" href="https://huggingface.co/spaces/eliezeravihail/torchdocs-agent">
+            Try it live
+            <svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M13 6l6 6-6 6"/></svg>
+          </a>
+          <a class="btn btn-ghost" href="#how">See how it works</a>
+        </div>
+        <p class="disclaimer">A personal project — not affiliated with or endorsed by the PyTorch team.</p>
+      </div>
+
+      <!-- the live answer card: the product's signature UX -->
+      <div class="card" id="demo" aria-label="Example of the agent answering, with its reasoning shown">
+        <div class="card-top">
+          <span class="dot"></span><span class="dot"></span><span class="dot"></span>
+          <span class="card-title">torchdocs-agent</span>
+        </div>
+        <div class="card-body">
+          <div class="ask">&gt; <span class="q">How do I use SGD with momentum?</span></div>
+          <div class="trace" id="trace">
+            <div class="row" data-row>🔍 searched “SGD momentum”</div>
+            <div class="row" data-row>📄 found: torch.optim.SGD · Per-parameter options</div>
+            <div class="row" data-row>✍️ writing the answer</div>
+          </div>
+          <div class="answer" id="answer">
+            Pass <code>momentum</code> to <code>torch.optim.SGD</code> — a value around <code>0.9</code> is typical:
+            <div class="snippet"><span class="kw">optimizer</span> = torch.optim.SGD(model.parameters(),
+      lr=<span class="kw">0.01</span>, momentum=<span class="kw">0.9</span>)  <span class="cm"># accelerates in consistent directions</span></div>
+            <a class="cite" href="https://docs.pytorch.org/docs/stable/generated/torch.optim.SGD.html">✓ torch.optim.SGD ↗</a><span class="caret" id="caret"></span>
+          </div>
+        </div>
+      </div>
+    </div>
+  </section>
+
+  <!-- THESIS -->
+  <section class="band" id="what">
+    <div class="wrap">
+      <div class="eyebrow sub-eyebrow">The point</div>
+      <p class="thesis" style="margin-top:16px">
+        A general model will happily <span class="hl">invent an API that doesn’t exist</span>.
+        This one can’t: every answer is built only from <span class="em2">retrieved documentation</span> —
+        or it tells you it’s beyond the docs and points you to the source.
+      </p>
+    </div>
+  </section>
+
+  <!-- FEATURES -->
+  <section class="band">
+    <div class="wrap">
+      <div class="band-head">
+        <div class="eyebrow">What makes it different</div>
+        <h2>Grounded, current, and honest about its limits.</h2>
+      </div>
+      <div class="features">
+        <div class="feat">
+          <div class="rule"></div>
+          <div class="tag">Citations</div>
+          <h3>Grounded, or it refers you out</h3>
+          <p>Every claim traces to a retrieved doc section; citations open the exact page and anchor. Source-code internals get a DeepWiki / GitHub referral — never a guess dressed as fact.</p>
+        </div>
+        <div class="feat">
+          <div class="rule"></div>
+          <div class="tag">Freshness</div>
+          <h3>The index heals itself</h3>
+          <p>After answering, it revalidates the cited pages against the live docs. If they drifted, it re-embeds the changed sections in place and corrects the answer — no rebuild, no stale cache.</p>
+        </div>
+        <div class="feat">
+          <div class="rule"></div>
+          <div class="tag">Transparency</div>
+          <h3>You watch it think</h3>
+          <p>The reasoning streams live in grey — what it searched, what it found — then resolves into the answer in black. Honest progress for a payload that can only arrive whole.</p>
+        </div>
+        <div class="feat">
+          <div class="rule"></div>
+          <div class="tag">Retrieval</div>
+          <h3>Hybrid search, lean stack</h3>
+          <p>Dense pgvector + keyword tsvector, RRF-merged over Neon Postgres. Embeddings run on a local 384-dim model on CPU: no embedding API, no quota, no cost.</p>
+        </div>
+        <div class="feat">
+          <div class="rule"></div>
+          <div class="tag">Resilience</div>
+          <h3>Won’t fall over</h3>
+          <p>A provider fallback chain — OpenRouter → Gemini → Anthropic — with per-model cooldowns. One rate-limited or broken key self-heals to the next instead of taking the app down.</p>
+        </div>
+        <div class="feat">
+          <div class="rule"></div>
+          <div class="tag">Discipline</div>
+          <h3>Checked before it ships</h3>
+          <p>Every snippet is statically checked — it must parse, import only torch/stdlib, and mention only symbols that exist in the index. A failed check triggers one regeneration, not a bluff.</p>
+        </div>
+      </div>
+    </div>
+  </section>
+
+  <!-- HOW IT WORKS -->
+  <section class="band" id="how">
+    <div class="wrap">
+      <div class="band-head">
+        <div class="eyebrow">The pipeline</div>
+        <h2>From the docs site to a cited answer, in four stages.</h2>
+        <p>Data flows one way. Each stage is a package in the repo, documented on its own.</p>
+      </div>
+      <div class="pipe">
+        <div class="step">
+          <div class="n">01</div>
+          <h3>ingest</h3>
+          <p>Discover every page (Sphinx <span class="code">objects.inv</span> + sitemaps), crawl it, and heading-chunk the docs site into an on-disk snapshot.</p>
+          <svg class="arrow" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.4" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M13 6l6 6-6 6"/></svg>
+        </div>
+        <div class="step">
+          <div class="n">02</div>
+          <h3>index</h3>
+          <p>Embed into Neon (pgvector + tsvector). Hybrid retrieval returns pointers; content is hydrated straight from the database.</p>
+          <svg class="arrow" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.4" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M13 6l6 6-6 6"/></svg>
+        </div>
+        <div class="step">
+          <div class="n">03</div>
+          <h3>agent</h3>
+          <p>Guard → route → a single grounded pass or a tool loop (<span class="code">search_docs</span> / <span class="code">read_page</span> / <span class="code">ask_source</span>) → a validated answer.</p>
+          <svg class="arrow" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.4" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M13 6l6 6-6 6"/></svg>
+        </div>
+        <div class="step">
+          <div class="n">04</div>
+          <h3>app</h3>
+          <p>A Gradio Space streams the reasoning trace and renders the answer with clickable citations — then quietly checks the docs are still current.</p>
+        </div>
+      </div>
+    </div>
+  </section>
+
+  <!-- USE IT -->
+  <section class="band" id="use">
+    <div class="wrap">
+      <div class="band-head">
+        <div class="eyebrow">Use it</div>
+        <h2>Two ways in.</h2>
+      </div>
+      <div class="tracks">
+        <div class="track">
+          <div class="kicker">No install</div>
+          <h3>Ask the live agent</h3>
+          <ol>
+            <li>Open the Space — it runs as a public web app.</li>
+            <li>Type a question <strong>in English</strong> and press Enter.</li>
+            <li>Read the answer; click a citation to land on the exact doc section.</li>
+          </ol>
+          <div class="ex">
+            <span>How do I use torch.optim.SGD with momentum?</span>
+            <span>What LR schedulers are supported?</span>
+            <span>How do I build a CNN to classify images?</span>
+          </div>
+          <div class="cta-row" style="margin-top:22px">
+            <a class="btn btn-primary" href="https://huggingface.co/spaces/eliezeravihail/torchdocs-agent">Open the Space
+              <svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M13 6l6 6-6 6"/></svg>
+            </a>
+          </div>
+        </div>
+        <div class="track">
+          <div class="kicker">Self-host</div>
+          <h3>Run or deploy your own</h3>
+          <p style="color:var(--muted); margin:14px 0 0; font-size:0.95rem;">Embeddings run locally on CPU, so building the index needs only a Neon connection string.</p>
+          <pre class="block"><span class="cm"># 1 · install</span>
+git clone https://github.com/eliezeravihail/TorchDocsAgent
+<span class="kw">cd</span> TorchDocsAgent && pip install -e .
+
+<span class="cm"># 2 · build the index (crawl + embed, CPU, minutes)</span>
+<span class="kw">export</span> NEON_URL=postgres://...
+python scripts/build_index.py
+
+<span class="cm"># 3 · run locally — or deploy to HF Spaces</span>
+python -m app.main</pre>
+        </div>
+      </div>
+    </div>
+  </section>
+
+  <!-- ENGINEERING DOCS -->
+  <section class="band">
+    <div class="wrap">
+      <div class="band-head">
+        <div class="eyebrow">Built in the open</div>
+        <h2>Read the reasoning, not just the code.</h2>
+        <p>Every package is documented with its boundary, its logic, the design decisions behind it, and the tools it chose — and why.</p>
+      </div>
+      <div class="doclist">
+        <a class="doc" href="https://github.com/eliezeravihail/TorchDocsAgent/tree/main/docs/agent"><div class="p">agent/</div><div class="d">Guard, router, grounded &amp; loop paths, LLM dispatch.</div></a>
+        <a class="doc" href="https://github.com/eliezeravihail/TorchDocsAgent/tree/main/docs/index"><div class="p">index/</div><div class="d">Neon/pgvector storage, hybrid retrieval, freshness.</div></a>
+        <a class="doc" href="https://github.com/eliezeravihail/TorchDocsAgent/tree/main/docs/ingest"><div class="p">ingest/</div><div class="d">Discover → crawl → chunk the docs into a snapshot.</div></a>
+        <a class="doc" href="https://github.com/eliezeravihail/TorchDocsAgent/tree/main/docs/eval"><div class="p">eval/</div><div class="d">Static checks, retrieval metrics, LLM-as-judge.</div></a>
+        <a class="doc" href="https://github.com/eliezeravihail/TorchDocsAgent/tree/main/docs/app"><div class="p">app/</div><div class="d">The Gradio Space: serving + the live trace.</div></a>
+        <a class="doc" href="https://github.com/eliezeravihail/TorchDocsAgent/blob/main/docs/design-content-and-agent-flow.md"><div class="p">design ↗</div><div class="d">The architecture bible — corpus, tools, flow.</div></a>
+      </div>
+    </div>
+  </section>
+</main>
+
+<footer>
+  <div class="wrap foot">
+    <span class="brand"><svg class="flame" viewBox="0 0 24 24" aria-hidden="true"><path fill="currentColor" style="color:var(--ember)" d="M13.5 1.5c.4 3-1.2 4.7-2.8 6.3C9 9.6 7.2 11.3 7.2 14.2a4.8 4.8 0 0 0 9.6.3c0-1.6-.7-2.9-1.4-4-.3 1-1 1.6-1.8 1.7.8-2.3.3-5.3-2-8.4l1.9-2.3Z"/></svg> TorchDocs Agent</span>
+    <span class="sep"></span>
+    <a href="https://github.com/eliezeravihail/TorchDocsAgent">GitHub</a>
+    <a href="https://huggingface.co/spaces/eliezeravihail/torchdocs-agent">Live Space</a>
+    <a href="https://github.com/eliezeravihail/TorchDocsAgent/tree/main/docs">Docs</a>
+    <a href="https://github.com/pytorch/pytorch/blob/main/LICENSE">Docs license · BSD-3</a>
+  </div>
+</footer>
+
+<script>
+  // theme toggle — stamps data-theme, overriding the media query in both directions
+  (function () {
+    var btn = document.getElementById("themeBtn");
+    btn && btn.addEventListener("click", function () {
+      var root = document.documentElement;
+      var dark = getComputedStyle(root).getPropertyValue("--bg").trim().indexOf("#1") === 0
+        || matchMedia("(prefers-color-scheme: dark)").matches;
+      var cur = root.getAttribute("data-theme");
+      if (!cur) cur = matchMedia("(prefers-color-scheme: dark)").matches ? "dark" : "light";
+      root.setAttribute("data-theme", cur === "dark" ? "light" : "dark");
+    });
+  })();
+
+  // hero: play the reasoning trace, then reveal the answer (reduced-motion → final state)
+  (function () {
+    var reduce = matchMedia("(prefers-reduced-motion: reduce)").matches;
+    var rows = Array.prototype.slice.call(document.querySelectorAll("#trace [data-row]"));
+    var answer = document.getElementById("answer");
+    if (reduce) { rows.forEach(function (r) { r.classList.add("show"); }); answer.classList.add("show"); return; }
+    var i = 0;
+    function next() {
+      if (i < rows.length) { rows[i].classList.add("show"); i++; setTimeout(next, 620); }
+      else { setTimeout(function () { answer.classList.add("show"); }, 260); }
+    }
+    setTimeout(next, 420);
+  })();
+</script>
+
+</body>
+</html>