# 05 — Search Redesign (Phase 4B)

Search is the **flagship**. The goal: make people *want to search rather than browse* —
an experience that's **inviting, visual, forgiving, ranked, and explainable**, works for
*"that gym receipt from March"* (not just `from:gym has:attachment`), and runs **locally
and privately**. Grounded in the existing Postgres FTS + `IAiProvider` seam (see
[01](01-architecture-review.md)); AI modes are **optional** — with AI off, lexical +
structured search is still excellent.

---

## The layered search engine
Four cooperating layers, each usable alone; higher layers *degrade gracefully* to lower
ones when AI is disabled.

```
┌ Layer 4 · AI-ASSISTED  (optional, local) ────────────────────────────┐
│  NL→query · conversational "ask your inbox" · explanations · related   │
├ Layer 3 · SEMANTIC  (optional, local embeddings + pgvector) ──────────┤
│  meaning-based recall · "similar to this" · concept search             │
├ Layer 2 · LEXICAL + RANKING  (Postgres FTS, always on) ───────────────┤
│  websearch_to_tsquery · ts_rank · pg_trgm fuzzy · multi-field          │
├ Layer 1 · STRUCTURED  (SQL WHERE, always on) ─────────────────────────┤
│  sender/date/labels/flags/size/category · filter chips                 │
└───────────────────────────────────────────────────────────────────────┘
                 ▲ hybrid fusion (RRF) ranks Layer 2+3 together ▲
```

**Hybrid ranking (RRF):** lexical and semantic candidate sets are merged via *Reciprocal
Rank Fusion*, then boosted by **recency**, **sender importance**, and **unread**. This
replaces today's date-only ordering — the single biggest search win.

---

## The experience: search as the front door
- **Hero search field** in the top bar + a **search-home** when the box is focused/empty:
  **recent searches**, **suggested searches** ("Unread from people you reply to",
  "Large attachments", "Receipts this month"), and **saved searches ★**.
- Type freely → **live results** with **filter chips** the user can add/remove by click
  (pointer-first). Behind an unobtrusive "＋ Filters" for the full builder.
- Every result shows **"why it matched"** (highlighted terms, or the semantic concept, or
  the parsed NL interpretation as *editable chips*).
- **Zero-result recovery:** *Did you mean* · *Broaden* · *Search all mail/trash*.
- Fully keyboard-drivable as an **accelerator** (⌘K → type → arrow → enter), but never required.

---

## Search modes & features
Each with **Problem solved · Implementation · Complexity · Effort · User value.**
Effort = rough dev-days for one engineer; Complexity = S/M/L/XL.

| Feature | Problem solved | Implementation | Cx | Effort | Value |
|---------|----------------|----------------|----|--------|-------|
| **Relevance ranking** | Results sorted by date, not usefulness | `ts_rank_cd` + hybrid RRF + recency/sender/unread boosts; top-N by score | M | 3–5d | 🔴🔴🔴 |
| **Multi-field FTS** | Only Subject+Body indexed | Extend generated `tsvector` (sender name/addr, attachment filenames, labels) with weights (A/B/C/D) | S | 2–3d | 🔴🔴 |
| **Fuzzy / typo tolerance** | "recieved" finds nothing | `pg_trgm` GIN + similarity fallback when FTS is empty | S | 1–2d | 🔴🔴 |
| **Phrase / prefix / boolean** | `plainto_tsquery` too blunt | Swap to `websearch_to_tsquery` (quotes, OR, -exclude) | S | 1d | 🔴 |
| **Interactive filter chips** | Operators are a syntax wall | UI chips ↔ structured DTO; add/remove live | M | 3–4d | 🔴🔴🔴 |
| **Advanced query builder** | Power users want precision | Visual builder → same DTO; operators still work | M | 3–4d | 🔵🔵 |
| **Natural-language search** | "gym receipt March" | Local LLM (or rules-first) parses intent → chips + terms, shown editable | L | 5–8d | 🔴🔴🔴 |
| **Semantic search** | Recall by *meaning*, not keywords | `IEmbeddingProvider` (Ollama) → `pgvector` (HNSW); backfill job; hybrid with FTS | L | 8–12d | 🔴🔴🔴 |
| **Conversational "ask your inbox"** | "What did Sarah say about the invoice?" | RAG: semantic retrieve → local LLM answers with **citations** to source emails | XL | 10–15d | 🔴🔴🔴 |
| **"Why this matched"** | Trust + learnability | `ts_headline` highlights (lexical); nearest-concept + snippet (semantic); parsed-intent chips (NL) | M | 3–4d | 🔴🔴 |
| **Saved searches ★** | Repeated queries | Persist DTO; pin to sidebar; optional live count | S | 2d | 🔴🔴 |
| **Recent searches / history** | Re-find what you searched | Per-user history table; privacy-clearable | S | 1–2d | 🔴 |
| **Suggested searches** | Blank-box paralysis | Rules over user's own stats (top senders, unreplied, big attachments) | S | 2–3d | 🔴🔴 |
| **People search** | "Everything with this person" | Sender/recipient facet + aggregation; person profile panel | M | 3–5d | 🔴🔴 |
| **Attachment search** | Find files, not emails | Index filenames now; **content/OCR** later (see below) | S→L | 2d → +8d | 🔴🔴 |
| **Entity extraction** | Search by amounts/dates/orgs | Local NER (LLM or rules) → entity index; facets (money, dates, companies) | L | 6–10d | 🔴🔴 |
| **Timeline search** | "our thread over time" | Date-bucketed results + a timeline scrubber UI | M | 3–4d | 🔵🔵 |
| **Related conversations** | Surface adjacent context | Vector nearest-neighbours + same-participants/thread heuristics | M | 3–5d | 🔴🔴 |
| **Duplicate / near-dup detection** | Clutter, repeated sends | Hash (exact) + embedding similarity (near) → group/cleanup | M | 4–6d | 🔵🔵 |
| **Conversation intelligence** | Long threads are walls | Thread summary + extracted actions/decisions (local LLM) surfaced in results & reading | L | 8–12d | 🔴🔴🔴 |

### Additional ideas (beyond the brief)
- **Search-driven bulk actions** — run cleanup/label/unsub on a *result set* ("archive all newsletters older than 6 months").
- **Scoped search** — constrain to this folder / label / person / thread inline.
- **"Find similar to this email"** — one-click semantic neighbours from any message.
- **Zero-result recovery** — *did-you-mean* (trigram), *broaden*, *search trash/all*.
- **Search from selection** — highlight text → "search inbox for this".
- **Sender reputation / safety facet** — filter by phishing/spam confidence (ties to AI).
- **Search templates** — parameterised saved searches ("invoices from {client}").
- **Search analytics (for the user)** — "you search 'invoice' most" → suggests a saved search / smart folder.

---

## Ranking design (detail)
```
score = RRF(lexical_rank, semantic_rank)
        + w_recency   · decay(sentAt)
        + w_sender    · senderImportance(userReplyRate, frequency)
        + w_unread    · isUnread
```
- Weights are configurable; defaults tuned so **exact/lexical hits never lose to fuzzy noise**.
- `senderImportance` derives from the user's *own* behaviour (reply rate, frequency) —
  computable from existing data, no AI required.

## Performance
- **Indexes:** GIN on `SearchVector`; GIN `pg_trgm` on sender/subject; **HNSW** (or IVFFlat)
  on the `pgvector` embedding column.
- **Keyset/cursor pagination** for lexical/structured (replaces offset `Skip/Take`); score-
  ordered top-N for ranked/semantic (fetch N, no deep offset).
- **Async embedding backfill** worker (reuse the hosted-worker pattern) so semantic search
  builds in the background; new mail embedded on sync. Batch to respect the 3080's VRAM.
- Cache suggestions/recent; debounce live search (~120ms).
- Target: **<150ms** lexical, **<400ms** hybrid on 100k emails (local).

## Privacy
- Embeddings and NER computed **locally via Ollama**, stored **locally** in Postgres.
- Nothing leaves the machine unless the user explicitly selects a cloud provider.
- **AI-off fallback:** Layers 1–2 (structured + ranked lexical + fuzzy) — still a top-tier
  search. Semantic/NL/conversational simply hide when AI is disabled.

## Mapping to the existing code (extend, don't rewrite)
- **`GmailQueryParser` → `SearchIntentParser`:** keep operator parsing; add a rules-first
  NL layer, optional local-LLM disambiguation, emit the same `SearchRequestDto` (+ new fields).
- **`SearchService`:** add ranking (`ts_rank_cd`), `websearch_to_tsquery`, `pg_trgm`
  fallback, multi-field vector, hybrid RRF, keyset pagination, `ts_headline` explanations.
- **`Email.SearchVector`:** widen the generated column (weighted A/B/C/D) + add a
  `pgvector` `Embedding` column + `EmailEntities`/`ThreadSummary` tables.
- **AI seam:** extend to `IEmbeddingProvider.EmbedAsync` and a structured
  `IAiProvider.CompleteStructuredAsync` (see [06 AI Strategy](06-ai-strategy.md)); Null
  provider returns "unavailable" so callers fall back to lexical.

## Phasing (feeds the roadmap)
- **MVP search:** relevance ranking · multi-field FTS · fuzzy · chips · saved/recent/suggested · "why matched" (lexical) · keyset pagination.
- **v1.1:** natural-language parse · people search · attachment (filename) · search-driven bulk.
- **v1.2:** semantic (pgvector) · related · find-similar · thread summaries in results.
- **v2.0:** conversational "ask your inbox" (RAG + citations) · entity search · duplicate detection · attachment OCR/content.