QuOTE questions in the enrichment pipeline + one-click eval default (#88)

* QuOTE-style hypothetical questions join the content-enrichment pipeline

The reranker measurement left five genuinely-buried pages — Linear, Conv2d,
random_split, WeightedRandomSampler, einsum — where a descriptive question
matches nothing the page carries, not even its gloss (Linear still at true
dense rank ~3,400 post-gloss). The literature's index-side answer (QuOTE,
arXiv:2502.10976; HyPE) is to index the QUESTIONS a page answers, turning
question→document matching into question→question matching, paid once at
index time rather than per query like HyDE.

scripts/generate_questions.py mirrors the gloss pipeline exactly: batched,
flushed per batch, resumable via the shared existing_urls_of() check, core
pages first. 5 short task-vocabulary questions per api page, at most one
naming the symbol — the vocabulary bridge is the point. Output is committed
to index/questions.jsonl.

index/embed.py folds the questions into indexed_text() (after the gloss,
before the heading), feeding BOTH the vector and the tsvector — the exact
channel pair the reranker measurement proved decisive (CrossEntropyLoss
flipped via tsvector+rerank). The questions file gets its own recipe stamp
(shared _enrichment_stamp), and the shape change bumps the recipe to v7, so
the next Build Index re-embeds once automatically.

Both workflows run the new pass: Generate glosses gains a questions step
(continue-on-error so a failure never discards the glosses written above),
and Build Index's crawl → enrich → embed sequence now glosses AND questions
new pages before embedding. Provider defaults flipped to openai-compat (hy3
with credit carries the corpus; gemini stays the fallback).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LX1gm8fuzK2ZeEWi3Zar1b

* Eval default suite = retrieval+judge: the app-dispatch button must measure everything

The GitHub mobile app cannot set workflow_dispatch inputs — it fires with the
defaults. So the default must BE the measurement one click should give: a new
retrieval+judge suite option (recall/MRR + the answer-quality anchor) becomes
the default, and both job conditions accept it. agentic stays opt-in — it is
the slow, LLM-heavy benchmark that should not run on every click.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LX1gm8fuzK2ZeEWi3Zar1b

---------

Co-authored-by: Claude <noreply@anthropic.com>

.github/workflows/build-index.yml

@@ -82,16 +82,26 @@ jobs:
           OPENAI_COMPAT_API_KEY: ${{ secrets.OPENROUTER }}
           TORCHDOCS_OPENAI_COMPAT_MODEL: ${{ inputs.llm || 'tencent/hy3:free,meta-llama/llama-3.3-70b-instruct:free' }}
         run: python scripts/generate_glosses.py --limit 0 --batch 30 --sleep 5
-      - name: Commit any new glosses (also on gloss failure — partial progress is saved)
+      - name: Hypothetical questions for new pages (QuOTE-style, incremental)
+        continue-on-error: true # same contract as the gloss step
+        env:
+          PYTHONUNBUFFERED: "1"
+          TORCHDOCS_PROVIDER: ${{ inputs.provider || 'openai-compat' }}
+          GEMINI_API_KEY: ${{ secrets.GOOGLE_API }}
+          OPENAI_COMPAT_BASE_URL: https://openrouter.ai/api/v1
+          OPENAI_COMPAT_API_KEY: ${{ secrets.OPENROUTER }}
+          TORCHDOCS_OPENAI_COMPAT_MODEL: ${{ inputs.llm || 'tencent/hy3:free,meta-llama/llama-3.3-70b-instruct:free' }}
+        run: python scripts/generate_questions.py --limit 0 --batch 10 --sleep 5
+      - name: Commit any new enrichment (also on failure — partial progress is saved)
         if: always()
         run: |
           git config user.name "github-actions[bot]"
           git config user.email "github-actions[bot]@users.noreply.github.com"
-          git add -f index/glosses.jsonl
-          git diff --cached --quiet || git commit -m "index: glosses from Actions run [skip ci]"
+          git add -f index/glosses.jsonl index/questions.jsonl
+          git diff --cached --quiet || git commit -m "index: content enrichment from Actions run [skip ci]"
           git pull --rebase origin main  # main may have moved during the run
           git push
-      - name: Embed into Neon (now with the glosses in indexed_text)
+      - name: Embed into Neon (now with glosses + questions in indexed_text)
         env:
           PYTHONUNBUFFERED: "1"
           NEON_URL: ${{ secrets.NEON }}

.github/workflows/eval.yml

@@ -1,23 +1,28 @@
 name: Eval
 
 # The measurement loop, one button:
-#   suite=retrieval (default) — recall@k + MRR over the 100-question set
-#     against the LIVE index, plus the rank diagnostic (where do the known-miss
-#     pages actually sit in their kind-pool). No LLM needed.
+#   suite=retrieval+judge (default) — recall@k + MRR over the 100-question set
+#     against the LIVE index (plus the rank diagnostic), AND the LLM-as-judge
+#     answer-quality anchor. The default is the full one-click measurement
+#     because the GitHub mobile app cannot set workflow inputs.
+#   suite=retrieval — recall/MRR only. No LLM needed.
 #   suite=agentic — the agent-loop benchmark (loop vs single-shot coverage
 #     delta). Needs LLM keys; slow on free tiers.
 #   suite=judge — LLM-as-judge answer quality (faithfulness / relevance /
 #     citation-correctness) over the grounded path. Needs LLM keys.
-#   suite=all — all three.
+#   suite=all — everything.
 # Results are committed back to main for before/after diffing.
 on:
   workflow_dispatch:
     inputs:
       suite:
-        description: "what to measure"
+        description: "what to measure. The default runs retrieval AND judge —
+          deliberately, because the GitHub mobile app can't set inputs, so the
+          default must be the measurement we actually want one click to give:
+          recall/MRR plus the answer-quality anchor. agentic stays opt-in."
         type: choice
-        options: [retrieval, agentic, judge, all]
-        default: retrieval
+        options: [retrieval+judge, retrieval, agentic, judge, all]
+        default: retrieval+judge
       set:
         description: "retrieval eval set (v1 = 100 questions, v0 = original 13)"
         type: string
@@ -63,7 +68,7 @@ permissions:
 
 jobs:
   retrieval:
-    if: ${{ inputs.suite == 'retrieval' || inputs.suite == 'all' }}
+    if: ${{ inputs.suite == 'retrieval' || inputs.suite == 'retrieval+judge' || inputs.suite == 'all' }}
     runs-on: ubuntu-latest
     timeout-minutes: 15
     steps:
@@ -140,7 +145,7 @@ jobs:
           git push
 
   judge:
-    if: ${{ inputs.suite == 'judge' || inputs.suite == 'all' }}
+    if: ${{ inputs.suite == 'judge' || inputs.suite == 'retrieval+judge' || inputs.suite == 'all' }}
     runs-on: ubuntu-latest
     timeout-minutes: 120
     steps:

.github/workflows/generate-glosses.yml

@@ -1,21 +1,26 @@
 name: Generate glosses
 
-# Contextual Retrieval, index side: LLM-write a one-sentence plain-language
-# gloss for every api reference page in the corpus snapshot (batched, ~300
-# calls for 3.6K pages). Resumable: already-glossed URLs are skipped and
-# partial progress is committed even on failure — re-run to fill gaps.
-# After glosses land, run Build Index (the changed recipe stamp forces the
-# one-time full re-embed), then Eval retrieval to measure the effect.
+# Index-side content enrichment, two passes over the corpus snapshot:
+#   1. glosses — a one-sentence plain-language description per api page
+#      (Contextual Retrieval)
+#   2. hypothetical questions — a few task-vocabulary questions each api page
+#      answers (QuOTE-style), for pages whose gloss alone couldn't bridge the
+#      vocabulary gap (measured 2026-07-09: Linear at dense rank ~3,400
+#      post-gloss)
+# Both are batched and resumable: already-covered URLs are skipped and partial
+# progress is committed even on failure — re-run to fill gaps. After they
+# land, run Build Index (the changed recipe stamp forces the one-time full
+# re-embed), then Eval retrieval to measure the effect.
 on:
   workflow_dispatch:
     inputs:
       provider:
-        description: "LLM provider. Free quotas are tight (measured 2026-07-08):
-          gemini = 20 req/day PER MODEL, OpenRouter = 50 req/day total without
-          credits. A failed provider falls through to the other (both keys set)."
+        description: "LLM provider. openai-compat = OpenRouter (hy3 with credit
+          carries the full corpus); gemini free = 20 req/day per model. A failed
+          provider falls through to the other (both keys set)."
         type: choice
-        options: [gemini, openai-compat]
-        default: gemini
+        options: [openai-compat, gemini]
+        default: openai-compat
       gemini_model:
         description: "gemini model — the free 20/day quota is per model, so
           re-running with a different one (gemini-2.5-flash-lite,
@@ -28,19 +33,27 @@ on:
         type: string
         default: "tencent/hy3:free,meta-llama/llama-3.3-70b-instruct:free"
       limit:
-        description: "gloss at most N pages this run (0 = all remaining)"
+        description: "cover at most N pages per pass this run (0 = all remaining)"
         type: string
         default: "0"
 
 permissions:
   contents: write
 
-concurrency: generate-glosses # two writers would duplicate gloss lines
+concurrency: generate-glosses # two writers would duplicate enrichment lines
 
 jobs:
   glosses:
     runs-on: ubuntu-latest
-    timeout-minutes: 180
+    timeout-minutes: 300
+    env:
+      PYTHONUNBUFFERED: "1"
+      TORCHDOCS_PROVIDER: ${{ inputs.provider }}
+      GEMINI_API_KEY: ${{ secrets.GOOGLE_API }}
+      TORCHDOCS_GEMINI_MODEL: ${{ inputs.gemini_model }}
+      OPENAI_COMPAT_BASE_URL: https://openrouter.ai/api/v1
+      OPENAI_COMPAT_API_KEY: ${{ secrets.OPENROUTER }}
+      TORCHDOCS_OPENAI_COMPAT_MODEL: ${{ inputs.model }}
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
@@ -55,23 +68,21 @@ jobs:
           restore-keys: corpus-
       - run: pip install -e .
       - name: Write glosses
-        env:
-          PYTHONUNBUFFERED: "1"
-          TORCHDOCS_PROVIDER: ${{ inputs.provider }}
-          GEMINI_API_KEY: ${{ secrets.GOOGLE_API }}
-          TORCHDOCS_GEMINI_MODEL: ${{ inputs.gemini_model }}
-          OPENAI_COMPAT_BASE_URL: https://openrouter.ai/api/v1
-          OPENAI_COMPAT_API_KEY: ${{ secrets.OPENROUTER }}
-          TORCHDOCS_OPENAI_COMPAT_MODEL: ${{ inputs.model }}
         # batch 30 → ~121 calls for the whole corpus; the run is resumable, so
         # each dispatch chips away at whatever today's quotas allow
         run: python scripts/generate_glosses.py --limit "${{ inputs.limit }}" --batch 30 --sleep 5
+      - name: Write hypothetical questions (QuOTE-style)
+        # a question failure must not throw away the glosses written above —
+        # the commit step below runs regardless and both passes are resumable
+        continue-on-error: true
+        # 5 questions/page is a longer reply than a gloss → smaller batches
+        run: python scripts/generate_questions.py --limit "${{ inputs.limit }}" --batch 10 --sleep 5
       - name: Commit whatever was written (also on failure — the run is resumable)
         if: always()
         run: |
           git config user.name "github-actions[bot]"
           git config user.email "github-actions[bot]@users.noreply.github.com"
-          git add index/glosses.jsonl
-          git diff --cached --quiet || git commit -m "index: glosses from Actions run [skip ci]"
+          git add index/glosses.jsonl index/questions.jsonl
+          git diff --cached --quiet || git commit -m "index: content enrichment from Actions run [skip ci]"
           git pull --rebase origin main  # main may have moved during the run
           git push

index/embed.py

@@ -36,6 +36,13 @@ MAX_EMBED_CHARS = 2000  # bge context is 512 tokens; beyond it is truncated anyw
 # near the descriptive questions users actually ask.
 GLOSSES_PATH = Path(__file__).parent / "glosses.jsonl"
 
+# QuOTE-style hypothetical questions: {url: [task-vocabulary questions the page
+# answers]}, generated by scripts/generate_questions.py and committed.
+# indexed_text() folds them in — question→question matching for pages whose
+# one-sentence gloss couldn't bridge the vocabulary gap (measured 2026-07-09:
+# Linear still at true dense rank ~3,400 post-gloss).
+QUESTIONS_PATH = Path(__file__).parent / "questions.jsonl"
+
 
 @cache
 def load_glosses() -> dict[str, str]:
@@ -45,26 +52,42 @@ def load_glosses() -> dict[str, str]:
     return {row["url"]: row["gloss"] for row in rows}
 
 
-def _gloss_stamp() -> str:
-    """Content hash of the gloss file — folded into the recipe below, so any
-    gloss change (or first arrival) forces the one-time full re-embed itself;
-    no manual version bump to forget."""
-    if not GLOSSES_PATH.exists():
+@cache
+def load_questions() -> dict[str, list[str]]:
+    if not QUESTIONS_PATH.exists():
+        return {}
+    rows = (json.loads(line) for line in QUESTIONS_PATH.open() if line.strip())
+    return {row["url"]: row["questions"] for row in rows}
+
+
+def _enrichment_stamp(path: Path) -> str:
+    """Content hash of a committed enrichment file (glosses / questions) —
+    folded into the recipe below, so any content change (or first arrival)
+    forces the one-time full re-embed itself; no manual version bump to
+    forget."""
+    if not path.exists():
         return "none"
-    return hashlib.sha256(GLOSSES_PATH.read_bytes()).hexdigest()[:8]
+    return hashlib.sha256(path.read_bytes()).hexdigest()[:8]
+
+
+def _gloss_stamp() -> str:
+    return _enrichment_stamp(GLOSSES_PATH)
+
+
+def _questions_stamp() -> str:
+    return _enrichment_stamp(QUESTIONS_PATH)
 
 
 # bump the version prefix when indexed_text()'s SHAPE changes → forces a
 # one-time full re-embed (dims same, so the row-skip check would otherwise keep
-# stale vectors). v6: synopsis extraction fixed to read the Sphinx <dl>
-# description line — v5 accidentally extracted the signature+github-url head,
-# duplicating the chunk's own opening (a semantic no-op, measured: distances
-# unchanged to 3 decimals). The model tag is folded in so swapping models is
-# itself a recipe change: a same-dims swap (e.g. two 768d models) still forces
-# a re-embed, and index_meta stays honest about which model's vectors are live
-# (a dims change also rebuilds the table outright — see index/db.ensure_schema).
-# The gloss stamp does the same for gloss-content changes.
-EMBED_RECIPE = f"v6-{EMBED_MODEL.split('/')[-1]}-g{_gloss_stamp()}"
+# stale vectors). v7: QuOTE-style hypothetical questions join the indexed text
+# (v6 fixed synopsis extraction to read the Sphinx <dl> description line). The
+# model tag is folded in so swapping models is itself a recipe change: a
+# same-dims swap (e.g. two 768d models) still forces a re-embed, and index_meta
+# stays honest about which model's vectors are live (a dims change also
+# rebuilds the table outright — see index/db.ensure_schema). The gloss and
+# question stamps do the same for enrichment-content changes.
+EMBED_RECIPE = f"v7-{EMBED_MODEL.split('/')[-1]}-g{_gloss_stamp()}-q{_questions_stamp()}"
 
 
 def chunk_key(unit: dict) -> str:
@@ -89,7 +112,8 @@ def symbol_from_url(url: str) -> str:
 
 
 def indexed_text(unit: dict) -> str:
-    """What we embed AND tsvector: symbol + synopsis + gloss + heading + body.
+    """What we embed AND tsvector: symbol + synopsis + gloss + questions +
+    heading + body.
 
     The symbol gives the page strong lexical weight for symbol-typed queries.
     The synopsis (the page's own first prose sentence, extracted at chunk time)
@@ -97,7 +121,10 @@ def indexed_text(unit: dict) -> str:
     to descriptive questions — reference pages' own text is signature-shaped
     and embeds far from "which loss takes raw logits?", so the plain-language
     sentences are prepended to every chunk of the page, feeding both the
-    vector and the tsvector.
+    vector and the tsvector. The hypothetical questions (QuOTE-style) go one
+    step further for pages the gloss couldn't move: they put the QUESTION
+    phrasing itself into both channels, so a user's descriptive query matches
+    question→question instead of question→signature.
     """
     parts = []
     symbol = symbol_from_url(unit["url"])
@@ -108,6 +135,9 @@ def indexed_text(unit: dict) -> str:
     gloss = load_glosses().get(unit["url"], "")
     if gloss:
         parts.append(gloss)
+    questions = load_questions().get(unit["url"], [])
+    if questions:
+        parts.append(" ".join(questions))
     heading = " > ".join(unit.get("heading_path", []))
     if heading:
         parts.append(heading)

scripts/generate_glosses.py

@@ -106,10 +106,16 @@ def parse_glosses(raw: str, n: int) -> dict[int, str]:
     return out
 
 
-def existing_urls() -> set[str]:
-    if not GLOSSES_PATH.exists():
+def existing_urls_of(path: Path) -> set[str]:
+    """URLs already covered in a jsonl enrichment file — the resume check,
+    shared with generate_questions.py (same append-and-skip pipeline shape)."""
+    if not path.exists():
         return set()
-    return {json.loads(line)["url"] for line in GLOSSES_PATH.open() if line.strip()}
+    return {json.loads(line)["url"] for line in path.open() if line.strip()}
+
+
+def existing_urls() -> set[str]:
+    return existing_urls_of(GLOSSES_PATH)
 
 
 def main() -> int:

scripts/generate_questions.py

@@ -0,0 +1,149 @@
+"""Generate hypothetical questions for API reference pages (QuOTE-style).
+
+Why: the reranker measurement (2026-07-09) left a residue of genuinely-buried
+pages — Linear, Conv2d, random_split, WeightedRandomSampler, einsum — where a
+descriptive question ("what's the standard fully-connected layer?") matches
+NOTHING the page carries, not even its one-sentence gloss (Linear still sat at
+true dense rank ~3,400 post-gloss). The literature's index-side answer
+(QuOTE, arXiv:2502.10976; HyPE) is to index the QUESTIONS a page answers,
+turning question→document matching into question→question matching — paid
+once at index time, not per query like HyDE.
+
+What: for every api-kind page in the corpus snapshot, ask an LLM for a few
+short questions a user would ask that this page answers — phrased in everyday
+task vocabulary, mostly WITHOUT naming the symbol (the vocabulary bridge is
+the whole point; the symbol token is already in the index). The questions are
+folded into indexed_text() by index/embed.py — feeding both the page's vector
+and its tsvector, the exact channel pair that flipped CrossEntropyLoss.
+
+Output: index/questions.jsonl — {"url", "questions": [...]} per line,
+committed. Same shape as the gloss pipeline: batched, flushed per batch,
+resumable (already-covered URLs are skipped) — rate-limit deaths just mean
+"run it again".
+
+Usage:  python scripts/generate_questions.py [--limit N] [--batch N]
+        (needs an LLM key; corpus snapshot must exist — run the crawl first)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import time
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from dotenv import load_dotenv
+
+from scripts.generate_glosses import api_pages, existing_urls_of
+
+QUESTIONS_PATH = Path(__file__).parent.parent / "index" / "questions.jsonl"
+
+QUESTIONS_PER_PAGE = 5
+QUESTION_MAX_CHARS = 160
+
+SYSTEM = (
+    "You write search-bridging questions for PyTorch documentation reference "
+    f"pages. For each numbered page (symbol/title + excerpt) write "
+    f"{QUESTIONS_PER_PAGE} distinct short questions (8-20 words) a PyTorch "
+    "user would ask that THIS page answers. Phrase them the way users talk "
+    "about the TASK, in everyday ML vocabulary; at most one question may name "
+    "the symbol itself — the rest must describe what it does or when you need "
+    "it (e.g. for Linear: 'What's the standard fully-connected layer that "
+    "applies a weight matrix and bias?'). Reply with a JSON array only, one "
+    'item per page, no other text: [{"i": 0, "questions": ["...", ...]}, ...]'
+)
+
+
+def batch_prompt(batch: list[dict]) -> str:
+    from index.embed import symbol_from_url
+
+    blocks = []
+    for i, page in enumerate(batch):
+        symbol = symbol_from_url(page["url"]) or page["title"]
+        blocks.append(f"### {i}\nsymbol: {symbol}\nexcerpt: {page['excerpt']}")
+    return "\n\n".join(blocks) + f"\n\nJSON array with {len(batch)} question sets:"
+
+
+def parse_questions(raw: str, n: int) -> dict[int, list[str]]:
+    """{index: [questions]} from the model's reply; malformed items are dropped."""
+    start, end = raw.find("["), raw.rfind("]")
+    if start == -1 or end == -1:
+        return {}
+    try:
+        items = json.loads(raw[start : end + 1])
+    except json.JSONDecodeError:
+        return {}
+    out: dict[int, list[str]] = {}
+    for item in items if isinstance(items, list) else []:
+        if not isinstance(item, dict):
+            continue
+        i, qs = item.get("i"), item.get("questions")
+        if not (isinstance(i, int) and 0 <= i < n and isinstance(qs, list)):
+            continue
+        clean = [q.strip()[:QUESTION_MAX_CHARS] for q in qs if isinstance(q, str) and q.strip()]
+        if clean:
+            out[i] = clean[:QUESTIONS_PER_PAGE]
+    return out
+
+
+def main() -> int:
+    load_dotenv()
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--limit", type=int, default=0, help="cover at most N pages (0 = all)")
+    parser.add_argument("--batch", type=int, default=10, help="pages per LLM call")
+    parser.add_argument("--sleep", type=float, default=2.0, help="pause between calls (s)")
+    args = parser.parse_args()
+
+    from agent.llm import GenerationError, _raw_completion
+    from ingest.crawl import CORPUS_DIR
+
+    if not CORPUS_DIR.exists() or not any(CORPUS_DIR.rglob("*.md")):
+        print("corpus snapshot is empty — run the crawl (Build Index) first", flush=True)
+        return 1
+
+    done = existing_urls_of(QUESTIONS_PATH)
+    todo = [p for p in api_pages(CORPUS_DIR) if p["url"] not in done]
+    if args.limit:
+        todo = todo[: args.limit]
+    print(f"[questions] {len(done)} pages already covered, {len(todo)} to go", flush=True)
+    if not todo:
+        return 0
+
+    written = failed_batches = 0
+    with QUESTIONS_PATH.open("a") as out:
+        for at in range(0, len(todo), args.batch):
+            batch = todo[at : at + args.batch]
+            try:
+                raw = _raw_completion(batch_prompt(batch), system=SYSTEM, timeout=120.0)
+            except GenerationError as exc:
+                print(f"[questions] batch at {at} failed: {exc}", flush=True)
+                failed_batches += 1
+                if failed_batches >= 5:
+                    print("[questions] 5 failed batches — provider looks down, stopping",
+                          flush=True)
+                    break
+                continue
+            sets = parse_questions(raw, len(batch))
+            if not sets:
+                print(f"[questions] batch at {at}: unparseable reply, skipped", flush=True)
+                failed_batches += 1
+                continue
+            for i, qs in sorted(sets.items()):
+                out.write(json.dumps({"url": batch[i]["url"], "questions": qs},
+                                     ensure_ascii=False) + "\n")
+            out.flush()  # checkpoint: kill/rate-limit here loses nothing
+            written += len(sets)
+            print(f"[questions] {at + len(batch)}/{len(todo)} pages seen, "
+                  f"{written} question sets written", flush=True)
+            time.sleep(args.sleep)
+
+    print(f"[questions] done: {written} new question sets → {QUESTIONS_PATH}", flush=True)
+    # partial success is success (resumable); total failure is loud
+    return 0 if written else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tests/index/test_embed.py

@@ -99,6 +99,40 @@ def test_gloss_stamp_changes_the_recipe_with_gloss_content(monkeypatch, tmp_path
     assert first not in ("none", embed._gloss_stamp())
 
 
+def test_indexed_text_folds_in_the_pages_hypothetical_questions(monkeypatch):
+    # QuOTE-style: the question phrasing itself joins both channels, so a
+    # descriptive query matches question→question instead of question→signature
+    from index.embed import indexed_text
+
+    url = "https://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html"
+    questions = [
+        "What's the standard fully-connected layer that applies a weight matrix and bias?",
+        "How do I add a dense layer to my network?",
+    ]
+    monkeypatch.setattr("index.embed.load_glosses", lambda: {})
+    monkeypatch.setattr("index.embed.load_questions", lambda: {url: questions})
+    unit = {"url": url, "heading_path": ["Linear"], "content": "in_features (int)"}
+    text = indexed_text(unit)
+    for q in questions:
+        assert q in text
+    # questions sit between the symbol block and the body, like the gloss
+    assert text.index("torch.nn.Linear") < text.index(questions[0]) < text.index("in_features")
+    # a page with no questions is unchanged
+    assert questions[0] not in indexed_text({**unit, "url": url.replace("Linear", "GELU")})
+
+
+def test_questions_stamp_changes_the_recipe_with_question_content(monkeypatch, tmp_path):
+    # same contract as the gloss stamp: new/changed questions force the re-embed
+    from index import embed
+
+    monkeypatch.setattr(embed, "QUESTIONS_PATH", tmp_path / "questions.jsonl")
+    assert embed._questions_stamp() == "none"
+    embed.QUESTIONS_PATH.write_text('{"url": "u", "questions": ["q"]}\n')
+    first = embed._questions_stamp()
+    embed.QUESTIONS_PATH.write_text('{"url": "u", "questions": ["other"]}\n')
+    assert first not in ("none", embed._questions_stamp())
+
+
 def test_iter_corpus_units_walks_snapshot(tmp_path):
     save_page("https://docs.pytorch.org/docs/stable/optim.html", "core", HTML, tmp_path)
     units = list(iter_corpus_units(tmp_path))

tests/scripts/test_generate_questions.py

@@ -0,0 +1,78 @@
+"""Question generation: the pure parts — prompt shape, reply parsing, resume."""
+
+import json
+
+from scripts.generate_glosses import existing_urls_of
+from scripts.generate_questions import (
+    QUESTION_MAX_CHARS,
+    QUESTIONS_PER_PAGE,
+    batch_prompt,
+    parse_questions,
+)
+
+
+def _page(url, title="T", excerpt="Applies a linear transformation."):
+    return {"url": url, "title": title, "excerpt": excerpt}
+
+
+def test_batch_prompt_numbers_pages_and_uses_symbol():
+    prompt = batch_prompt(
+        [
+            _page("https://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html"),
+            _page("https://docs.pytorch.org/docs/stable/amp.html", title="AMP page"),
+        ]
+    )
+    assert "### 0" in prompt and "### 1" in prompt
+    assert "torch.nn.Linear" in prompt  # symbol derived from the url
+    assert "AMP page" in prompt  # no symbol in the url → the title stands in
+
+
+def test_parse_questions_plain_json():
+    raw = json.dumps(
+        [
+            {"i": 0, "questions": ["How do I add a fully-connected layer?", "  spaced  "]},
+            {"i": 1, "questions": ["What splits a dataset randomly?"]},
+        ]
+    )
+    out = parse_questions(raw, 2)
+    assert out[0][0] == "How do I add a fully-connected layer?"
+    assert out[0][1] == "spaced"
+    assert out[1] == ["What splits a dataset randomly?"]
+
+
+def test_parse_questions_survives_fences_and_prose():
+    raw = 'Sure!\n```json\n[{"i": 0, "questions": ["q1"]}]\n```\nDone.'
+    assert parse_questions(raw, 1) == {0: ["q1"]}
+
+
+def test_parse_questions_drops_malformed_items():
+    raw = json.dumps(
+        [
+            {"i": 5, "questions": ["out of range"]},  # index beyond the batch
+            {"i": 0, "questions": []},  # empty list → dropped
+            {"i": 1, "questions": ["ok", 7, ""]},  # non-strings/blanks filtered
+            "not a dict",
+        ]
+    )
+    assert parse_questions(raw, 2) == {1: ["ok"]}
+
+
+def test_parse_questions_caps_count_and_length():
+    many = [f"question {i}?" for i in range(QUESTIONS_PER_PAGE + 4)]
+    long = "x" * (QUESTION_MAX_CHARS + 50)
+    out = parse_questions(json.dumps([{"i": 0, "questions": many + [long]}]), 1)
+    assert len(out[0]) == QUESTIONS_PER_PAGE
+    assert all(len(q) <= QUESTION_MAX_CHARS for q in out[0])
+
+
+def test_parse_questions_non_json_is_empty():
+    assert parse_questions("the pages look great", 1) == {}
+
+
+def test_existing_urls_of_resumes_from_the_output_file(tmp_path):
+    path = tmp_path / "questions.jsonl"
+    assert existing_urls_of(path) == set()  # no file yet → nothing covered
+    path.write_text(
+        '{"url": "https://a", "questions": ["q"]}\n\n{"url": "https://b", "questions": ["q"]}\n'
+    )
+    assert existing_urls_of(path) == {"https://a", "https://b"}