# Implementation plan

Build order for the Node application specified in this folder. Do not skip ingest quality for a prettier UI.

## Phase 0 — Repo skeleton

- `package.json`, `tsconfig.json`, `.gitignore` (`data/lancedb`, `.env`, `node_modules`)
- Root `README.md`: install, `cp .env.example .env`, `npm run ingest`, `npm start`
- `src/config.ts`, `src/types.ts`

## Phase 1 — Parsers and chunks (no OpenAI)

- HTML section splitter + tests on `disease-cancer.html` and one process step
- JSON tree flattener + tests on Cancer timing leaves
- BMI enriched row formatter
- Write `data/debug-chunks.json` when `DEBUG_CHUNKS=1` for inspection

**Exit:** ~400–600 well-labeled chunks on disk without embeddings.

## Phase 2 — Vector store + ingest CLI

- LanceDB table, upsert by `id`
- OpenAI embeddings batch
- Manifest
- `npm run ingest` idempotent

**Exit:** `/health` can later report chunk count > 0.

## Phase 3 — Retrieve + generate

- `retrieve.ts` + `prompt.ts` + `generate.ts` (`gpt-4o-mini`)
- `POST /v1/query`
- Smoke questions:
  - BMI SI Level vs Preferred
  - Cancer 2–3 years (synthesis vs dump)
  - Mortgage process step 6 DecisionResult
  - Missing page (carrier ruleset HTML) → insufficient context

## Phase 4 — Hardening

- Zod validation, error mapper
- `MIN_RELEVANCE` + `insufficient_context`
- Dual-layer instruction in system prompt
- Optional `ans_code` exact lookup

## Phase 5 — Optional

- Minimal static UI (textarea + citations)
- Hybrid search
- Auth

## Acceptance criteria (v1)

1. Ingest completes against current `html/` without manual file lists.
2. Query about **mortgage SI BMI ceilings** cites `bmi-build-chart-review.html` and/or dataset page.
3. Query about **legacy decline counts** for a tree leaf cites `disease_trees.json` layer `legacy_2022_dump`.
4. Answers include disclaimer language from the system prompt.
5. No web browsing; no training on user chats.

## Suggested stack pin

- Node 22
- `openai` SDK v4+
- `@lancedb/lancedb` (Node)
- Express 4 or 5

If LanceDB install is painful on Windows, swap store to **Chroma persistent** or **JSON + cosine in memory** for v1 (corpus is small enough that in-memory embeddings in `data/embeddings.json` is an acceptable fallback — document the swap in `vectorStore.ts` only).