cesnimda
4ce2df0a2b
CI / backend (push) Successful in 52s
CI / frontend (push) Successful in 14s
Deploy Staging / deploy (push) Successful in 18s
CI / backend (pull_request) Successful in 52s
CI / frontend (pull_request) Successful in 15s
Security / secrets (push) Successful in 4s
Security / dependencies (push) Successful in 55s
Security / secrets (pull_request) Successful in 4s
Security / dependencies (pull_request) Successful in 54s
130 lines
9.4 KiB
Markdown
130 lines
9.4 KiB
Markdown
# 05 — Search Redesign (Phase 4B)
|
||
|
||
Search is the **flagship**. The goal: make people *want to search rather than browse* —
|
||
an experience that's **inviting, visual, forgiving, ranked, and explainable**, works for
|
||
*"that gym receipt from March"* (not just `from:gym has:attachment`), and runs **locally
|
||
and privately**. Grounded in the existing Postgres FTS + `IAiProvider` seam (see
|
||
[01](01-architecture-review.md)); AI modes are **optional** — with AI off, lexical +
|
||
structured search is still excellent.
|
||
|
||
---
|
||
|
||
## The layered search engine
|
||
Four cooperating layers, each usable alone; higher layers *degrade gracefully* to lower
|
||
ones when AI is disabled.
|
||
|
||
```
|
||
┌ Layer 4 · AI-ASSISTED (optional, local) ────────────────────────────┐
|
||
│ NL→query · conversational "ask your inbox" · explanations · related │
|
||
├ Layer 3 · SEMANTIC (optional, local embeddings + pgvector) ──────────┤
|
||
│ meaning-based recall · "similar to this" · concept search │
|
||
├ Layer 2 · LEXICAL + RANKING (Postgres FTS, always on) ───────────────┤
|
||
│ websearch_to_tsquery · ts_rank · pg_trgm fuzzy · multi-field │
|
||
├ Layer 1 · STRUCTURED (SQL WHERE, always on) ─────────────────────────┤
|
||
│ sender/date/labels/flags/size/category · filter chips │
|
||
└───────────────────────────────────────────────────────────────────────┘
|
||
▲ hybrid fusion (RRF) ranks Layer 2+3 together ▲
|
||
```
|
||
|
||
**Hybrid ranking (RRF):** lexical and semantic candidate sets are merged via *Reciprocal
|
||
Rank Fusion*, then boosted by **recency**, **sender importance**, and **unread**. This
|
||
replaces today's date-only ordering — the single biggest search win.
|
||
|
||
---
|
||
|
||
## The experience: search as the front door
|
||
- **Hero search field** in the top bar + a **search-home** when the box is focused/empty:
|
||
**recent searches**, **suggested searches** ("Unread from people you reply to",
|
||
"Large attachments", "Receipts this month"), and **saved searches ★**.
|
||
- Type freely → **live results** with **filter chips** the user can add/remove by click
|
||
(pointer-first). Behind an unobtrusive "＋ Filters" for the full builder.
|
||
- Every result shows **"why it matched"** (highlighted terms, or the semantic concept, or
|
||
the parsed NL interpretation as *editable chips*).
|
||
- **Zero-result recovery:** *Did you mean* · *Broaden* · *Search all mail/trash*.
|
||
- Fully keyboard-drivable as an **accelerator** (⌘K → type → arrow → enter), but never required.
|
||
|
||
---
|
||
|
||
## Search modes & features
|
||
Each with **Problem solved · Implementation · Complexity · Effort · User value.**
|
||
Effort = rough dev-days for one engineer; Complexity = S/M/L/XL.
|
||
|
||
| Feature | Problem solved | Implementation | Cx | Effort | Value |
|
||
|---------|----------------|----------------|----|--------|-------|
|
||
| **Relevance ranking** | Results sorted by date, not usefulness | `ts_rank_cd` + hybrid RRF + recency/sender/unread boosts; top-N by score | M | 3–5d | 🔴🔴🔴 |
|
||
| **Multi-field FTS** | Only Subject+Body indexed | Extend generated `tsvector` (sender name/addr, attachment filenames, labels) with weights (A/B/C/D) | S | 2–3d | 🔴🔴 |
|
||
| **Fuzzy / typo tolerance** | "recieved" finds nothing | `pg_trgm` GIN + similarity fallback when FTS is empty | S | 1–2d | 🔴🔴 |
|
||
| **Phrase / prefix / boolean** | `plainto_tsquery` too blunt | Swap to `websearch_to_tsquery` (quotes, OR, -exclude) | S | 1d | 🔴 |
|
||
| **Interactive filter chips** | Operators are a syntax wall | UI chips ↔ structured DTO; add/remove live | M | 3–4d | 🔴🔴🔴 |
|
||
| **Advanced query builder** | Power users want precision | Visual builder → same DTO; operators still work | M | 3–4d | 🔵🔵 |
|
||
| **Natural-language search** | "gym receipt March" | Local LLM (or rules-first) parses intent → chips + terms, shown editable | L | 5–8d | 🔴🔴🔴 |
|
||
| **Semantic search** | Recall by *meaning*, not keywords | `IEmbeddingProvider` (Ollama) → `pgvector` (HNSW); backfill job; hybrid with FTS | L | 8–12d | 🔴🔴🔴 |
|
||
| **Conversational "ask your inbox"** | "What did Sarah say about the invoice?" | RAG: semantic retrieve → local LLM answers with **citations** to source emails | XL | 10–15d | 🔴🔴🔴 |
|
||
| **"Why this matched"** | Trust + learnability | `ts_headline` highlights (lexical); nearest-concept + snippet (semantic); parsed-intent chips (NL) | M | 3–4d | 🔴🔴 |
|
||
| **Saved searches ★** | Repeated queries | Persist DTO; pin to sidebar; optional live count | S | 2d | 🔴🔴 |
|
||
| **Recent searches / history** | Re-find what you searched | Per-user history table; privacy-clearable | S | 1–2d | 🔴 |
|
||
| **Suggested searches** | Blank-box paralysis | Rules over user's own stats (top senders, unreplied, big attachments) | S | 2–3d | 🔴🔴 |
|
||
| **People search** | "Everything with this person" | Sender/recipient facet + aggregation; person profile panel | M | 3–5d | 🔴🔴 |
|
||
| **Attachment search** | Find files, not emails | Index filenames now; **content/OCR** later (see below) | S→L | 2d → +8d | 🔴🔴 |
|
||
| **Entity extraction** | Search by amounts/dates/orgs | Local NER (LLM or rules) → entity index; facets (money, dates, companies) | L | 6–10d | 🔴🔴 |
|
||
| **Timeline search** | "our thread over time" | Date-bucketed results + a timeline scrubber UI | M | 3–4d | 🔵🔵 |
|
||
| **Related conversations** | Surface adjacent context | Vector nearest-neighbours + same-participants/thread heuristics | M | 3–5d | 🔴🔴 |
|
||
| **Duplicate / near-dup detection** | Clutter, repeated sends | Hash (exact) + embedding similarity (near) → group/cleanup | M | 4–6d | 🔵🔵 |
|
||
| **Conversation intelligence** | Long threads are walls | Thread summary + extracted actions/decisions (local LLM) surfaced in results & reading | L | 8–12d | 🔴🔴🔴 |
|
||
|
||
### Additional ideas (beyond the brief)
|
||
- **Search-driven bulk actions** — run cleanup/label/unsub on a *result set* ("archive all newsletters older than 6 months").
|
||
- **Scoped search** — constrain to this folder / label / person / thread inline.
|
||
- **"Find similar to this email"** — one-click semantic neighbours from any message.
|
||
- **Zero-result recovery** — *did-you-mean* (trigram), *broaden*, *search trash/all*.
|
||
- **Search from selection** — highlight text → "search inbox for this".
|
||
- **Sender reputation / safety facet** — filter by phishing/spam confidence (ties to AI).
|
||
- **Search templates** — parameterised saved searches ("invoices from {client}").
|
||
- **Search analytics (for the user)** — "you search 'invoice' most" → suggests a saved search / smart folder.
|
||
|
||
---
|
||
|
||
## Ranking design (detail)
|
||
```
|
||
score = RRF(lexical_rank, semantic_rank)
|
||
+ w_recency · decay(sentAt)
|
||
+ w_sender · senderImportance(userReplyRate, frequency)
|
||
+ w_unread · isUnread
|
||
```
|
||
- Weights are configurable; defaults tuned so **exact/lexical hits never lose to fuzzy noise**.
|
||
- `senderImportance` derives from the user's *own* behaviour (reply rate, frequency) —
|
||
computable from existing data, no AI required.
|
||
|
||
## Performance
|
||
- **Indexes:** GIN on `SearchVector`; GIN `pg_trgm` on sender/subject; **HNSW** (or IVFFlat)
|
||
on the `pgvector` embedding column.
|
||
- **Keyset/cursor pagination** for lexical/structured (replaces offset `Skip/Take`); score-
|
||
ordered top-N for ranked/semantic (fetch N, no deep offset).
|
||
- **Async embedding backfill** worker (reuse the hosted-worker pattern) so semantic search
|
||
builds in the background; new mail embedded on sync. Batch to respect the 3080's VRAM.
|
||
- Cache suggestions/recent; debounce live search (~120ms).
|
||
- Target: **<150ms** lexical, **<400ms** hybrid on 100k emails (local).
|
||
|
||
## Privacy
|
||
- Embeddings and NER computed **locally via Ollama**, stored **locally** in Postgres.
|
||
- Nothing leaves the machine unless the user explicitly selects a cloud provider.
|
||
- **AI-off fallback:** Layers 1–2 (structured + ranked lexical + fuzzy) — still a top-tier
|
||
search. Semantic/NL/conversational simply hide when AI is disabled.
|
||
|
||
## Mapping to the existing code (extend, don't rewrite)
|
||
- **`GmailQueryParser` → `SearchIntentParser`:** keep operator parsing; add a rules-first
|
||
NL layer, optional local-LLM disambiguation, emit the same `SearchRequestDto` (+ new fields).
|
||
- **`SearchService`:** add ranking (`ts_rank_cd`), `websearch_to_tsquery`, `pg_trgm`
|
||
fallback, multi-field vector, hybrid RRF, keyset pagination, `ts_headline` explanations.
|
||
- **`Email.SearchVector`:** widen the generated column (weighted A/B/C/D) + add a
|
||
`pgvector` `Embedding` column + `EmailEntities`/`ThreadSummary` tables.
|
||
- **AI seam:** extend to `IEmbeddingProvider.EmbedAsync` and a structured
|
||
`IAiProvider.CompleteStructuredAsync` (see [06 AI Strategy](06-ai-strategy.md)); Null
|
||
provider returns "unavailable" so callers fall back to lexical.
|
||
|
||
## Phasing (feeds the roadmap)
|
||
- **MVP search:** relevance ranking · multi-field FTS · fuzzy · chips · saved/recent/suggested · "why matched" (lexical) · keyset pagination.
|
||
- **v1.1:** natural-language parse · people search · attachment (filename) · search-driven bulk.
|
||
- **v1.2:** semantic (pgvector) · related · find-similar · thread summaries in results.
|
||
- **v2.0:** conversational "ask your inbox" (RAG + citations) · entity search · duplicate detection · attachment OCR/content.
|