docs: discovery blueprint + multi-provider design (#8)

2026-07-01 19:59:53 +02:00
parent 9ee5d757f5
commit 4ce2df0a2b
27 changed files with 2032 additions and 0 deletions
@@ -0,0 +1,129 @@
+# 05 — Search Redesign (Phase 4B)
+
+Search is the **flagship**. The goal: make people *want to search rather than browse* —
+an experience that's **inviting, visual, forgiving, ranked, and explainable**, works for
+*"that gym receipt from March"* (not just `from:gym has:attachment`), and runs **locally
+and privately**. Grounded in the existing Postgres FTS + `IAiProvider` seam (see
+[01](01-architecture-review.md)); AI modes are **optional** — with AI off, lexical +
+structured search is still excellent.
+
+---
+
+## The layered search engine
+Four cooperating layers, each usable alone; higher layers *degrade gracefully* to lower
+ones when AI is disabled.
+
+```
+┌ Layer 4 · AI-ASSISTED  (optional, local) ────────────────────────────┐
+│  NL→query · conversational "ask your inbox" · explanations · related   │
+├ Layer 3 · SEMANTIC  (optional, local embeddings + pgvector) ──────────┤
+│  meaning-based recall · "similar to this" · concept search             │
+├ Layer 2 · LEXICAL + RANKING  (Postgres FTS, always on) ───────────────┤
+│  websearch_to_tsquery · ts_rank · pg_trgm fuzzy · multi-field          │
+├ Layer 1 · STRUCTURED  (SQL WHERE, always on) ─────────────────────────┤
+│  sender/date/labels/flags/size/category · filter chips                 │
+└───────────────────────────────────────────────────────────────────────┘
+                 ▲ hybrid fusion (RRF) ranks Layer 2+3 together ▲
+```
+
+**Hybrid ranking (RRF):** lexical and semantic candidate sets are merged via *Reciprocal
+Rank Fusion*, then boosted by **recency**, **sender importance**, and **unread**. This
+replaces today's date-only ordering — the single biggest search win.
+
+---
+
+## The experience: search as the front door
+- **Hero search field** in the top bar + a **search-home** when the box is focused/empty:
+  **recent searches**, **suggested searches** ("Unread from people you reply to",
+  "Large attachments", "Receipts this month"), and **saved searches ★**.
+- Type freely → **live results** with **filter chips** the user can add/remove by click
+  (pointer-first). Behind an unobtrusive "＋ Filters" for the full builder.
+- Every result shows **"why it matched"** (highlighted terms, or the semantic concept, or
+  the parsed NL interpretation as *editable chips*).
+- **Zero-result recovery:** *Did you mean* · *Broaden* · *Search all mail/trash*.
+- Fully keyboard-drivable as an **accelerator** (⌘K → type → arrow → enter), but never required.
+
+---
+
+## Search modes & features
+Each with **Problem solved · Implementation · Complexity · Effort · User value.**
+Effort = rough dev-days for one engineer; Complexity = S/M/L/XL.
+
+| Feature | Problem solved | Implementation | Cx | Effort | Value |
+|---------|----------------|----------------|----|--------|-------|
+| **Relevance ranking** | Results sorted by date, not usefulness | `ts_rank_cd` + hybrid RRF + recency/sender/unread boosts; top-N by score | M | 3–5d | 🔴🔴🔴 |
+| **Multi-field FTS** | Only Subject+Body indexed | Extend generated `tsvector` (sender name/addr, attachment filenames, labels) with weights (A/B/C/D) | S | 2–3d | 🔴🔴 |
+| **Fuzzy / typo tolerance** | "recieved" finds nothing | `pg_trgm` GIN + similarity fallback when FTS is empty | S | 1–2d | 🔴🔴 |
+| **Phrase / prefix / boolean** | `plainto_tsquery` too blunt | Swap to `websearch_to_tsquery` (quotes, OR, -exclude) | S | 1d | 🔴 |
+| **Interactive filter chips** | Operators are a syntax wall | UI chips ↔ structured DTO; add/remove live | M | 3–4d | 🔴🔴🔴 |
+| **Advanced query builder** | Power users want precision | Visual builder → same DTO; operators still work | M | 3–4d | 🔵🔵 |
+| **Natural-language search** | "gym receipt March" | Local LLM (or rules-first) parses intent → chips + terms, shown editable | L | 5–8d | 🔴🔴🔴 |
+| **Semantic search** | Recall by *meaning*, not keywords | `IEmbeddingProvider` (Ollama) → `pgvector` (HNSW); backfill job; hybrid with FTS | L | 8–12d | 🔴🔴🔴 |
+| **Conversational "ask your inbox"** | "What did Sarah say about the invoice?" | RAG: semantic retrieve → local LLM answers with **citations** to source emails | XL | 10–15d | 🔴🔴🔴 |
+| **"Why this matched"** | Trust + learnability | `ts_headline` highlights (lexical); nearest-concept + snippet (semantic); parsed-intent chips (NL) | M | 3–4d | 🔴🔴 |
+| **Saved searches ★** | Repeated queries | Persist DTO; pin to sidebar; optional live count | S | 2d | 🔴🔴 |
+| **Recent searches / history** | Re-find what you searched | Per-user history table; privacy-clearable | S | 1–2d | 🔴 |
+| **Suggested searches** | Blank-box paralysis | Rules over user's own stats (top senders, unreplied, big attachments) | S | 2–3d | 🔴🔴 |
+| **People search** | "Everything with this person" | Sender/recipient facet + aggregation; person profile panel | M | 3–5d | 🔴🔴 |
+| **Attachment search** | Find files, not emails | Index filenames now; **content/OCR** later (see below) | S→L | 2d → +8d | 🔴🔴 |
+| **Entity extraction** | Search by amounts/dates/orgs | Local NER (LLM or rules) → entity index; facets (money, dates, companies) | L | 6–10d | 🔴🔴 |
+| **Timeline search** | "our thread over time" | Date-bucketed results + a timeline scrubber UI | M | 3–4d | 🔵🔵 |
+| **Related conversations** | Surface adjacent context | Vector nearest-neighbours + same-participants/thread heuristics | M | 3–5d | 🔴🔴 |
+| **Duplicate / near-dup detection** | Clutter, repeated sends | Hash (exact) + embedding similarity (near) → group/cleanup | M | 4–6d | 🔵🔵 |
+| **Conversation intelligence** | Long threads are walls | Thread summary + extracted actions/decisions (local LLM) surfaced in results & reading | L | 8–12d | 🔴🔴🔴 |
+
+### Additional ideas (beyond the brief)
+- **Search-driven bulk actions** — run cleanup/label/unsub on a *result set* ("archive all newsletters older than 6 months").
+- **Scoped search** — constrain to this folder / label / person / thread inline.
+- **"Find similar to this email"** — one-click semantic neighbours from any message.
+- **Zero-result recovery** — *did-you-mean* (trigram), *broaden*, *search trash/all*.
+- **Search from selection** — highlight text → "search inbox for this".
+- **Sender reputation / safety facet** — filter by phishing/spam confidence (ties to AI).
+- **Search templates** — parameterised saved searches ("invoices from {client}").
+- **Search analytics (for the user)** — "you search 'invoice' most" → suggests a saved search / smart folder.
+
+---
+
+## Ranking design (detail)
+```
+score = RRF(lexical_rank, semantic_rank)
+        + w_recency   · decay(sentAt)
+        + w_sender    · senderImportance(userReplyRate, frequency)
+        + w_unread    · isUnread
+```
+- Weights are configurable; defaults tuned so **exact/lexical hits never lose to fuzzy noise**.
+- `senderImportance` derives from the user's *own* behaviour (reply rate, frequency) —
+  computable from existing data, no AI required.
+
+## Performance
+- **Indexes:** GIN on `SearchVector`; GIN `pg_trgm` on sender/subject; **HNSW** (or IVFFlat)
+  on the `pgvector` embedding column.
+- **Keyset/cursor pagination** for lexical/structured (replaces offset `Skip/Take`); score-
+  ordered top-N for ranked/semantic (fetch N, no deep offset).
+- **Async embedding backfill** worker (reuse the hosted-worker pattern) so semantic search
+  builds in the background; new mail embedded on sync. Batch to respect the 3080's VRAM.
+- Cache suggestions/recent; debounce live search (~120ms).
+- Target: **<150ms** lexical, **<400ms** hybrid on 100k emails (local).
+
+## Privacy
+- Embeddings and NER computed **locally via Ollama**, stored **locally** in Postgres.
+- Nothing leaves the machine unless the user explicitly selects a cloud provider.
+- **AI-off fallback:** Layers 1–2 (structured + ranked lexical + fuzzy) — still a top-tier
+  search. Semantic/NL/conversational simply hide when AI is disabled.
+
+## Mapping to the existing code (extend, don't rewrite)
+- **`GmailQueryParser` → `SearchIntentParser`:** keep operator parsing; add a rules-first
+  NL layer, optional local-LLM disambiguation, emit the same `SearchRequestDto` (+ new fields).
+- **`SearchService`:** add ranking (`ts_rank_cd`), `websearch_to_tsquery`, `pg_trgm`
+  fallback, multi-field vector, hybrid RRF, keyset pagination, `ts_headline` explanations.
+- **`Email.SearchVector`:** widen the generated column (weighted A/B/C/D) + add a
+  `pgvector` `Embedding` column + `EmailEntities`/`ThreadSummary` tables.
+- **AI seam:** extend to `IEmbeddingProvider.EmbedAsync` and a structured
+  `IAiProvider.CompleteStructuredAsync` (see [06 AI Strategy](06-ai-strategy.md)); Null
+  provider returns "unavailable" so callers fall back to lexical.
+
+## Phasing (feeds the roadmap)
+- **MVP search:** relevance ranking · multi-field FTS · fuzzy · chips · saved/recent/suggested · "why matched" (lexical) · keyset pagination.
+- **v1.1:** natural-language parse · people search · attachment (filename) · search-driven bulk.
+- **v1.2:** semantic (pgvector) · related · find-similar · thread summaries in results.
+- **v2.0:** conversational "ask your inbox" (RAG + citations) · entity search · duplicate detection · attachment OCR/content.