# Project structure (complete Node application)

Target layout after implementation. Files marked *(spec only)* exist as documentation today; the rest are to be created.

```
f:/agent/
├── docs/                          # This specification
│   ├── README.md
│   ├── architecture.md
│   ├── project-structure.md
│   ├── data-sources.md
│   ├── rag-pipeline.md
│   ├── api.md
│   ├── learning-model.md          # where learning data is saved
│   └── implementation-plan.md
├── html/                          # SOURCE knowledge (authored; ingest reads, never overwrites)
│   ├── bmi-build-chart-review.html
│   ├── diseases/
│   ├── vertical-datasets/
│   ├── vertical-processes/
│   └── data/
├── data/                          # LEARNING + runtime (gitignored except .gitkeep)
│   ├── .gitkeep
│   ├── chunks/
│   │   └── chunks.jsonl           # exported chunk text (audit)
│   ├── lancedb/                   # LEARNED VECTORS (primary “model memory”)
│   ├── ingest-manifest.json
│   ├── embeddings.fallback.json   # only if LanceDB unused
│   └── learning/                  # query logs, feedback, operator corrections
│       ├── queries.jsonl
│       ├── feedback.jsonl
│       └── corrections/
├── src/
│   ├── index.ts                   # HTTP server bootstrap
│   ├── config.ts                  # env + defaults
│   ├── types.ts                   # Chunk, QueryRequest, QueryResponse
│   ├── ingest/
│   │   ├── cli.ts                 # npm run ingest
│   │   ├── loadFiles.ts
│   │   ├── parseHtml.ts
│   │   ├── parseJson.ts
│   │   ├── chunk.ts
│   │   └── embedAndStore.ts
│   ├── rag/
│   │   ├── embed.ts
│   │   ├── retrieve.ts
│   │   ├── prompt.ts
│   │   └── generate.ts
│   ├── store/
│   │   └── vectorStore.ts         # LanceDB adapter (swap later)
│   ├── learning/
│   │   ├── logQuery.ts            # writes data/learning/queries.jsonl
│   │   ├── feedback.ts            # writes data/learning/feedback.jsonl
│   │   └── loadCorrections.ts     # reads data/learning/corrections/
│   └── api/
│       ├── app.ts
│       ├── routes.ts
│       └── errors.ts
├── scripts/
│   └── smoke-query.ts             # one-off CLI question
├── tests/
│   ├── chunk.test.ts
│   ├── parseHtml.test.ts
│   └── retrieve.test.ts
├── .env.example
├── .gitignore
├── package.json
├── tsconfig.json
└── README.md                      # operator README (how to run)
```

## package.json scripts (intended)

| Script | Command | Purpose |
|--------|---------|---------|
| `ingest` | `tsx src/ingest/cli.ts` | Parse corpus, embed, persist vectors |
| `start` | `tsx src/index.ts` | API on `PORT` (default 3000) |
| `dev` | `tsx watch src/index.ts` | Reload server |
| `query` | `tsx scripts/smoke-query.ts` | CLI RAG without HTTP |
| `test` | `node --test` or `vitest` | Unit tests |
| `typecheck` | `tsc --noEmit` | Types |

## Dependencies (intended)

**Runtime**

- `express` — HTTP
- `openai` — embeddings + chat
- `@lancedb/lancedb` — local vector table (or `chromadb` if preferred)
- `cheerio` — HTML to text/sections
- `zod` — request validation
- `dotenv` — env

**Dev**

- `typescript`, `tsx`, `@types/express`, `@types/node`

## Environment (see also api.md)

```
OPENAI_API_KEY=
OPENAI_CHAT_MODEL=gpt-4o-mini
OPENAI_EMBED_MODEL=text-embedding-3-small
CORPUS_DIR=./html
VECTOR_DIR=./data/lancedb
PORT=3000
RETRIEVE_K=8
MIN_RELEVANCE=0.25
```

## Module boundaries

- `ingest/*` never calls chat completion.
- `rag/generate.ts` never reads `html/` directly; only retrieved chunks.
- `store/vectorStore.ts` is the only persistence API (`upsertChunks`, `search`).
- `config.ts` is the only place that reads `process.env`.

## Chunk record (canonical)

```ts
type KnowledgeChunk = {
  id: string;                 // stable: sha256(source + section + index)
  source_path: string;        // e.g. html/diseases/disease-cancer.html
  doc_type: "html_narrative" | "interview_tree" | "bmi_table" | "product" | "stats" | "process" | "dataset";
  module: string;             // cancer | diabetes | bmi | mortgage_process | ...
  title: string;
  text: string;               // what gets embedded
  metadata: Record<string, string | number | boolean>;
};
```

`id` must be **stable across ingest runs** so re-ingest updates rather than duplicates.