Inboxintel/docs/discovery/05-search-redesign.md

05 — Search Redesign (Phase 4B)

Search is the flagship. The goal: make people want to search rather than browse — an experience that's inviting, visual, forgiving, ranked, and explainable, works for "that gym receipt from March" (not just from:gym has:attachment), and runs locally and privately. Grounded in the existing Postgres FTS + IAiProvider seam (see 01); AI modes are optional — with AI off, lexical + structured search is still excellent.

---

The layered search engine

Four cooperating layers, each usable alone; higher layers degrade gracefully to lower ones when AI is disabled.

┌ Layer 4 · AI-ASSISTED  (optional, local) ────────────────────────────┐
│  NL→query · conversational "ask your inbox" · explanations · related   │
├ Layer 3 · SEMANTIC  (optional, local embeddings + pgvector) ──────────┤
│  meaning-based recall · "similar to this" · concept search             │
├ Layer 2 · LEXICAL + RANKING  (Postgres FTS, always on) ───────────────┤
│  websearch_to_tsquery · ts_rank · pg_trgm fuzzy · multi-field          │
├ Layer 1 · STRUCTURED  (SQL WHERE, always on) ─────────────────────────┤
│  sender/date/labels/flags/size/category · filter chips                 │
└───────────────────────────────────────────────────────────────────────┘
                 ▲ hybrid fusion (RRF) ranks Layer 2+3 together ▲

Hybrid ranking (RRF): lexical and semantic candidate sets are merged via Reciprocal Rank Fusion, then boosted by recency, sender importance, and unread. This replaces today's date-only ordering — the single biggest search win.

---

The experience: search as the front door

• Hero search field in the top bar + a search-home when the box is focused/empty: recent searches, suggested searches ("Unread from people you reply to", "Large attachments", "Receipts this month"), and saved searches ★.
• Type freely → live results with filter chips the user can add/remove by click (pointer-first). Behind an unobtrusive "＋ Filters" for the full builder.
• Every result shows "why it matched" (highlighted terms, or the semantic concept, or the parsed NL interpretation as editable chips).
• Zero-result recovery: Did you mean · Broaden · Search all mail/trash.
• Fully keyboard-drivable as an accelerator (⌘K → type → arrow → enter), but never required.

---

Search modes & features

Each with Problem solved · Implementation · Complexity · Effort · User value. Effort = rough dev-days for one engineer; Complexity = S/M/L/XL.

Feature	Problem solved	Implementation	Cx	Effort	Value
Relevance ranking	Results sorted by date, not usefulness	ts_rank_cd + hybrid RRF + recency/sender/unread boosts; top-N by score	M	3–5d	🔴🔴🔴
Multi-field FTS	Only Subject+Body indexed	Extend generated tsvector (sender name/addr, attachment filenames, labels) with weights (A/B/C/D)	S	2–3d	🔴🔴
Fuzzy / typo tolerance	"recieved" finds nothing	pg_trgm GIN + similarity fallback when FTS is empty	S	1–2d	🔴🔴
Phrase / prefix / boolean	plainto_tsquery too blunt	Swap to websearch_to_tsquery (quotes, OR, -exclude)	S	1d	🔴
Interactive filter chips	Operators are a syntax wall	UI chips ↔ structured DTO; add/remove live	M	3–4d	🔴🔴🔴
Advanced query builder	Power users want precision	Visual builder → same DTO; operators still work	M	3–4d	🔵🔵
Natural-language search	"gym receipt March"	Local LLM (or rules-first) parses intent → chips + terms, shown editable	L	5–8d	🔴🔴🔴
Semantic search	Recall by meaning, not keywords	IEmbeddingProvider (Ollama) → pgvector (HNSW); backfill job; hybrid with FTS	L	8–12d	🔴🔴🔴
Conversational "ask your inbox"	"What did Sarah say about the invoice?"	RAG: semantic retrieve → local LLM answers with citations to source emails	XL	10–15d	🔴🔴🔴
"Why this matched"	Trust + learnability	ts_headline highlights (lexical); nearest-concept + snippet (semantic); parsed-intent chips (NL)	M	3–4d	🔴🔴
Saved searches ★	Repeated queries	Persist DTO; pin to sidebar; optional live count	S	2d	🔴🔴
Recent searches / history	Re-find what you searched	Per-user history table; privacy-clearable	S	1–2d	🔴
Suggested searches	Blank-box paralysis	Rules over user's own stats (top senders, unreplied, big attachments)	S	2–3d	🔴🔴
People search	"Everything with this person"	Sender/recipient facet + aggregation; person profile panel	M	3–5d	🔴🔴
Attachment search	Find files, not emails	Index filenames now; content/OCR later (see below)	S→L	2d → +8d	🔴🔴
Entity extraction	Search by amounts/dates/orgs	Local NER (LLM or rules) → entity index; facets (money, dates, companies)	L	6–10d	🔴🔴
Timeline search	"our thread over time"	Date-bucketed results + a timeline scrubber UI	M	3–4d	🔵🔵
Related conversations	Surface adjacent context	Vector nearest-neighbours + same-participants/thread heuristics	M	3–5d	🔴🔴
Duplicate / near-dup detection	Clutter, repeated sends	Hash (exact) + embedding similarity (near) → group/cleanup	M	4–6d	🔵🔵
Conversation intelligence	Long threads are walls	Thread summary + extracted actions/decisions (local LLM) surfaced in results & reading	L	8–12d	🔴🔴🔴

Additional ideas (beyond the brief)

• Search-driven bulk actions — run cleanup/label/unsub on a result set ("archive all newsletters older than 6 months").
• Scoped search — constrain to this folder / label / person / thread inline.
• "Find similar to this email" — one-click semantic neighbours from any message.
• Zero-result recovery — did-you-mean (trigram), broaden, search trash/all.
• Search from selection — highlight text → "search inbox for this".
• Sender reputation / safety facet — filter by phishing/spam confidence (ties to AI).
• Search templates — parameterised saved searches ("invoices from {client}").
• Search analytics (for the user) — "you search 'invoice' most" → suggests a saved search / smart folder.

---

Ranking design (detail)

score = RRF(lexical_rank, semantic_rank)
        + w_recency   · decay(sentAt)
        + w_sender    · senderImportance(userReplyRate, frequency)
        + w_unread    · isUnread

• Weights are configurable; defaults tuned so exact/lexical hits never lose to fuzzy noise.
• senderImportance derives from the user's own behaviour (reply rate, frequency) — computable from existing data, no AI required.

Performance

• Indexes: GIN on SearchVector; GIN pg_trgm on sender/subject; HNSW (or IVFFlat) on the pgvector embedding column.
• Keyset/cursor pagination for lexical/structured (replaces offset Skip/Take); score- ordered top-N for ranked/semantic (fetch N, no deep offset).
• Async embedding backfill worker (reuse the hosted-worker pattern) so semantic search builds in the background; new mail embedded on sync. Batch to respect the 3080's VRAM.
• Cache suggestions/recent; debounce live search (~120ms).
• Target: <150ms lexical, <400ms hybrid on 100k emails (local).

Privacy

• Embeddings and NER computed locally via Ollama, stored locally in Postgres.
• Nothing leaves the machine unless the user explicitly selects a cloud provider.
• AI-off fallback: Layers 1–2 (structured + ranked lexical + fuzzy) — still a top-tier search. Semantic/NL/conversational simply hide when AI is disabled.

Mapping to the existing code (extend, don't rewrite)

• GmailQueryParser → SearchIntentParser: keep operator parsing; add a rules-first NL layer, optional local-LLM disambiguation, emit the same SearchRequestDto (+ new fields).
• SearchService: add ranking (ts_rank_cd), websearch_to_tsquery, pg_trgm fallback, multi-field vector, hybrid RRF, keyset pagination, ts_headline explanations.
• Email.SearchVector: widen the generated column (weighted A/B/C/D) + add a pgvector Embedding column + EmailEntities/ThreadSummary tables.
• AI seam: extend to IEmbeddingProvider.EmbedAsync and a structured IAiProvider.CompleteStructuredAsync (see 06 AI Strategy); Null provider returns "unavailable" so callers fall back to lexical.

Phasing (feeds the roadmap)

• MVP search: relevance ranking · multi-field FTS · fuzzy · chips · saved/recent/suggested · "why matched" (lexical) · keyset pagination.
• v1.1: natural-language parse · people search · attachment (filename) · search-driven bulk.
• v1.2: semantic (pgvector) · related · find-similar · thread summaries in results.
• v2.0: conversational "ask your inbox" (RAG + citations) · entity search · duplicate detection · attachment OCR/content.