diff --git a/docs/discovery/00-design-brief.md b/docs/discovery/00-design-brief.md new file mode 100644 index 0000000..9c173db --- /dev/null +++ b/docs/discovery/00-design-brief.md @@ -0,0 +1,45 @@ +# Design Brief (locked via interview) + +The confirmed design direction from the Phase 4A interview. This governs the UX/UI +redesign, design system, themes, and search experience. + +## Direction in one line +**An approachable, professional, "anyone can pick it up" email tool in the spirit of +Notion & Arc — a polished, responsive web app that is fast and premium without being +intimidating.** + +## Confirmed preferences +| Dimension | Decision | Design implication | +|-----------|----------|--------------------| +| Reference feel | **Notion / Arc, but professional** | Friendly, spatial, low learning curve — not a power-user speed tool (not Superhuman). | +| Density | **Balanced** | Comfortable, readable rows; efficient but not cramped. | +| Input model | **Pointer-first / discoverable** | Clickable UI is the backbone. Command palette + shortcuts are **accelerators for power users**, layered on top — never required. | +| App metaphor | **Polished web app** | Exceptional in-browser experience; shareable URLs; no native-desktop assumptions. | +| Platforms | **Desktop (Win + Mac/Linux), Tablet, Mobile** | **Fully responsive** is mandatory — layouts must gracefully collapse desktop → mobile. | +| Default theme | **Dark-first** | Design primarily for a layered dark UI; light theme fully first-class (genuine white, not grey). | +| Motion | **Subtle & smooth** | Gentle, quick transitions and tasteful micro-interactions. Respect `prefers-reduced-motion`. | +| Colour personality | **Warm ∩ cool blend, green accent** | Accent built around **`#3ba31f`** (refined into an accessible ramp); neutrals lean subtly warm. | +| Icons | **Clean line, rounded** (Lucide-style) | Consistent stroke weight, soft corners. | +| Speed vs polish | **Balanced** | Polish is welcome only when it never costs perceptible speed. | +| Audience | **Mainstream, power-capable** | Simple by default; power features revealed progressively. | +| Accessibility | **Deferred (nice-to-have)** | AA basics baked in cheaply; deeper a11y is a pre-launch backlog item. See below. | + +## The pivotal insight +The original brief leaned "keyboard-first"; the interview corrected this to +**discoverable-first**. That reframes the flagship **search** away from a syntax you +must learn (`from:x after:y`) toward an **inviting, visual, assisted** experience — +filter chips, live suggestions, and natural language — with the power syntax still +available underneath. This single decision shapes the entire redesign. + +## Accent colour note +`#3ba31f` = rgb(59,163,31). It's a strong differentiator (most email apps default to +blue) and bridges warm/cool. It will be developed into a full ramp (50→900); the +**interactive** shade will be nudged for WCAG AA contrast on dark surfaces and on +buttons, and green will **never be the sole signal** for selection/status (always +paired with icon/shape) so the design is colour-blind-safe by construction. + +## Deferred: accessibility +Per the user, accessibility is a **nice-to-have to revisit before public launch**, not +a hard requirement now. Cheap AA basics (contrast, focus rings, reduced-motion, +non-colour-only state) are still baked in; high-contrast mode, full screen-reader +semantics, font-scaling controls, and formal colour-blind audits are **backlog**. diff --git a/docs/discovery/01-architecture-review.md b/docs/discovery/01-architecture-review.md new file mode 100644 index 0000000..21f55bb --- /dev/null +++ b/docs/discovery/01-architecture-review.md @@ -0,0 +1,88 @@ +# 01 — Architecture Review (current state) + +Phase 1 deliverable: a grounded assessment of InboxIntel as it exists today, from the +source. This is the baseline the redesign builds on. + +## Architecture +**Clean Architecture, .NET 8** — strict dependency rule `Api → Infrastructure → Application → Domain`. + +| Layer | Responsibility | Key contents | +|-------|----------------|--------------| +| **Domain** | Entities + enums, no external deps | `Email`, `Sender`, `Domain`, `MailThread`, `Label`/`EmailLabel`, `Attachment`; `EmailCategory` | +| **Application** | Interfaces, DTOs, validation, parsing | `ISearchService` et al., `GmailQueryParser`, FluentValidation | +| **Infrastructure** | EF Core/Npgsql, integrations, workers | `SearchService`, `CleanupService`, `AnalyticsService`, `HeuristicClassifier`, **AI providers**, exports, `GmailSyncWorker`, `DigestWorker` | +| **Api** | ASP.NET Core Web API | Thin controllers, DI, Serilog, OAuth | +| **Frontend** | React 18 + Vite SPA | Chart.js, react-grid-layout (draggable dashboard), Tailwind | + +- **Data:** PostgreSQL (EF Core + Npgsql). `Email` designed for 100k+ rows/user; a + **generated `tsvector`** column backs full-text search. +- **Background:** hosted workers — `GmailSyncWorker` (scheduled sync), `DigestWorker` (digest). +- **AI seam (already present):** `IAiProvider` with `NullAiProvider` (disabled), + `OllamaProvider` (local `/api/chat`), `OpenAiProvider` (cloud). Contract today is a + single `CompleteAsync(systemPrompt, userPrompt)`. +- **Security:** Google OAuth2 (read-only Gmail), cookie session + JWT, refresh tokens + encrypted via Data Protection API, **IDOR-safe global query filters**, SSRF egress + guard, non-root containers, destructive actions require `Confirmed` + server preview. +- **Delivery:** Docker Compose (Postgres/API/frontend/optional nginx) + Gitea CI/CD. + +## Feature set +Gmail connect + sync → Postgres · analytics dashboard (draggable widgets) · advanced +search (Gmail operators + FTS) · safe bulk cleanup · unsubscribe management +(List-Unsubscribe / one-click) · heuristic categorisation (smart folders) · exports +(PDF/CSV/JSON) · optional **advisory** AI · sender/domain aggregation. + +## Existing search implementation (flagship) +1. **`GmailQueryParser`** (Application) parses `from: to: domain: after: before: + is:unread|read has:attachment`; remaining text → free-text. +2. **`SearchService`** (Infrastructure) composes structured filters as EF `WHERE` + clauses, and free text via Postgres FTS: + `SearchVector.Matches(PlainToTsQuery('english', term))`, where + `SearchVector = to_tsvector('english', coalesce(Subject,'') || ' ' || coalesce(BodyText,''))`. +3. Results are per-user filtered (IDOR-safe), **ordered by `SentAtUtc DESC`**, + offset-paginated, projected to `EmailSummaryDto`. + +## Strengths +- Clean, testable layering; DI throughout; 39 tests + CI/CD gate. +- **Real Postgres FTS** (generated tsvector), not naïve `LIKE`. +- Security-forward (IDOR filters, encrypted tokens, SSRF guard, confirmed destructive ops). +- **AI already decoupled behind `IAiProvider`** — the "no-AI / Ollama / future" requirement is architecturally seeded. +- Read-only scope + advisory AI = privacy-respecting. + +## Weaknesses +- **Search is single-mode**: ordered by **date, not relevance** (no `ts_rank`); no + fuzzy/typo tolerance, no semantic/vector search, no grouping, no "why matched," no + saved/recent/suggested searches. +- FTS is **English-only** and covers **only Subject + Body** (not sender, attachment + names, labels). +- Sender/domain filters use `.Contains()` → **non-sargable ILIKE scans** (no `pg_trgm`). +- Category is a **single heuristic enum** — no multi-label, confidence, or learning. +- **`IAiProvider` is chat-only** — no embeddings/classification/extraction contract, so + semantic search & structured extraction can't yet be expressed. +- Threading (`ThreadId`) exists but isn't surfaced as conversation intelligence. + +## Technical debt +- EF global-query-filter vs required `Email↔EmailLabel` relationship warning (in logs). +- Hardcoded English FTS config. +- **678 KB single JS chunk** (no code-splitting) — cold-load cost. +- Relevance-blind ordering. +- AI interface too narrow for the roadmap. + +## Performance bottlenecks +- `.Contains()` sender/domain → sequential scans (need `pg_trgm` GIN indexes). +- **Offset pagination** (`Skip/Take` + `Count`) degrades on deep pages → keyset/cursor. +- No score-based top-N (no ranking) → sorts full match set by date. +- Single JS chunk (~219 KB gzip) slows cold loads. +- Full-mailbox sync latency at 100k+ (batching/Polly present, but per-message fetch). + +## Documentation quality +Strong for a scaffold: `README`, `docs/ARCHITECTURE.md`, OAuth setup, `docs/specs/*`, +`AGENTS.md`, plus `WORKFLOW.md`/`CHANGELOG`; meaningful XML-doc comments. +**Gaps:** no API reference, no ERD/data-model doc, no ADRs, no search/AI design docs — +which this discovery produces. + +## Verdict +A clean, secure, well-tested foundation with a **genuine FTS base** and an **AI seam +already in place**. The biggest opportunities are exactly where the product wants to +win: **relevance-ranked, multi-mode, assisted search**; a **richer AI contract** +(embeddings/classification/extraction); **conversation intelligence**; and a **modern, +approachable UX**. None of these require a rewrite — they extend the existing seams. diff --git a/docs/discovery/02-competitor-analysis.md b/docs/discovery/02-competitor-analysis.md new file mode 100644 index 0000000..85303cb --- /dev/null +++ b/docs/discovery/02-competitor-analysis.md @@ -0,0 +1,73 @@ +# 02 — Competitor Analysis (Phase 2) + +Goal: understand the modern email landscape, isolate what users consistently praise +and complain about, and find **where InboxIntel can be genuinely better** — without +copying anyone. Analysis is framed against the [Design Brief](00-design-brief.md): +approachable-professional, fast, **local/private AI**, search-centric. + +## Landscape snapshot +| Product | Positioning | Standout strength | Recurring weakness | +|---------|-------------|-------------------|--------------------| +| **Gmail** | Default mass-market webmail | Powerful search operators; scale; free; Gemini bolt-ons | Cluttered; privacy optics; AI feels tacked-on; dated triage | +| **Outlook** | Enterprise mail + calendar | Calendar/email fusion; Copilot; Rules | Heavy; **search is famously flaky/slow**; inconsistent "new Outlook" | +| **Superhuman** | Speed tool for pros | **Blazing speed, keyboard triage, polish**; Superhuman AI | $30/mo; Gmail/Outlook-only; **intimidating for casual users** | +| **Proton Mail** | Privacy / E2EE | Zero-access encryption; Swiss; open source; local-ish Scribe AI | **Search limited** (encrypted); fewer productivity features; slower | +| **Spark** | Smart inbox + team email | Collaboration (shared drafts, comments), smart inbox, cross-platform | Privacy history concerns; can feel cluttered; sync hiccups | +| **Shortwave** | AI-native Gmail client | **AI search + assistant that genuinely work**; bundles; fast | Gmail-only; **cloud AI (privacy)**; pricing crept up | +| **Thunderbird** | Open-source local client | Free, private, extensible, local storage, multi-account | UX still catching up; **no built-in AI**; search not modern | +| **HEY** | Opinionated workflow | Novel triage (Screener/Imbox/Feed/Paper Trail) | Locked-in; no folders/search-first; pricey | +| **Fastmail** | Fast private power-mail | Excellent fast search; reliable; standards-based | No AI; power-user aesthetic; niche | +| **Missive** | Team shared inbox | Best-in-class collaboration/assignment/rules | Team-first; overkill for individuals | + +## Deep dive on the two axes that decide this market + +### Search (InboxIntel's flagship battleground) +- **Gmail / Fastmail**: fast, operator-rich, but **syntax-first** and browse-anchored — great for people who already know `from:` / `has:attachment`, opaque to everyone else. +- **Outlook**: powerful on paper, but **unreliable/slow search is its most complained-about feature** for years. +- **Shortwave**: the modern bar — **AI/semantic search + "ask your inbox"** questions ("what did Sarah say about the invoice?"). Genuinely loved, but **cloud-processed**. +- **Proton/Thunderbird**: privacy-strong but **search is a known weak point** (encryption / dated indexing). +- **Gap:** nobody offers **fast, relevance-ranked, semantic, *explainable* search that is also local/private and approachable to non-power-users.** That is precisely InboxIntel's opening. + +### AI +- The market has **split into two camps**: + 1. **AI, but cloud** — Gmail/Gemini, Outlook/Copilot, Shortwave, Spark. Useful, but your email is processed off-device. + 2. **Private, but little/no AI** — Proton, Thunderbird, Fastmail. +- Proton's **Scribe** (privacy-first, can run locally) hints at the future but is narrow (writing only). +- Common complaints: AI feels **generic/bolted-on**, **unexplained**, and **can't be turned off cleanly**. +- **Gap:** **genuinely useful AI that runs 100% locally (Ollama), is modular, explainable, and fully optional** — nobody mainstream owns this. + +## What users consistently PRAISE (cross-product) +1. **Speed** — instant search/open/triage (Superhuman, Fastmail). +2. **AI that saves real time** — summaries of long threads, "ask your inbox," draft replies (Shortwave, Superhuman AI). +3. **Privacy you can trust** — Proton's whole brand. +4. **Effortless triage** — Split Inbox / Bundles / Screener (Superhuman, Shortwave, HEY). +5. **Collaboration** — assign, comment, share drafts (Missive, Spark). +6. **Clean, calm, modern UI** that reduces overwhelm. + +## What users consistently COMPLAIN about +1. **Search that's slow, flaky, or syntax-only** (Outlook; Gmail/Proton on mobile). +2. **Privacy cost of AI** — "I want the AI but not to send my mail to a server." +3. **Clutter & overwhelm** — noisy inboxes, ads, too many features. +4. **AI that's generic and unexplained** — "why did it say that / categorise that?" +5. **Power tools that alienate casual users** (Superhuman's learning curve; HEY's rigidity). +6. **Price** — best experiences gated behind $10–30/mo. +7. **Opaque automation/categorisation** you can't understand or correct. + +## Opportunity gaps → where InboxIntel wins (ranked) +1. **Local, private, *useful* AI.** Resolve the market's central tension (AI **vs** privacy) by delivering both via Ollama. This is the headline differentiator and aligns with the existing `IAIProvider` seam and read-only scope. +2. **Approachable power-search as the home screen.** Discoverable, visual, assisted (filter chips + NL + suggestions) so a *mainstream* user searches confidently — with operators/semantic power underneath for those who grow into it. +3. **Explainability everywhere.** "Why this result matched," "why this was categorised/prioritised." No competitor does this well; it builds trust *and* teaches the product — perfect for the "anyone can pick it up" audience. +4. **Own-your-data intelligence.** Analytics, cleanup, unsubscribe, and local AI combine into a "your inbox, your machine, your intelligence" story that Gmail/Outlook structurally can't tell. +5. **Fast *and* friendly.** Superhuman is fast-but-intimidating; casual apps are friendly-but-slow. The brief's sweet spot (balanced, pointer-first, polished) is largely unoccupied. +6. **Honest, correctable automation.** Categories/priority a user can see, understand, and fix — the antidote to Gmail's opaque tabs. + +## Explicitly NOT copying +- Not Superhuman's keyboard-only speed-sport (brief = discoverable-first). +- Not HEY's rigid, search-hostile opinionation. +- Not cloud-AI-at-any-cost (brief = local/private). +- Not enterprise-collaboration-first (individual mainstream user is primary; collaboration is a later opportunity, see roadmap). + +## Positioning statement (draft) +> **InboxIntel** — the email tool that makes **search the fastest way to think about your +> inbox**, with **AI that runs on your own machine** and always explains itself. +> Powerful enough for pros, simple enough for anyone. diff --git a/docs/discovery/03-user-research.md b/docs/discovery/03-user-research.md new file mode 100644 index 0000000..44e6828 --- /dev/null +++ b/docs/discovery/03-user-research.md @@ -0,0 +1,107 @@ +# 03 — User Research: Personas & Journey Maps (Phase 3) + +Personas are prioritised against the [Design Brief](00-design-brief.md) audience: +**mainstream, power-capable.** So the **primary** personas are everyday individuals and +small-business owners; **secondary** personas are high-volume role users whose needs we +support *progressively* (power features revealed as people grow into them). + +Legend: 🟢 primary · 🔵 secondary + +--- + +## 🟢 Priya — Personal user (mainstream) +Busy professional with a personal Gmail full of subscriptions, receipts, travel, and the +occasional important thread buried in noise. +- **Goals:** find things fast; not miss the important stuff; keep the inbox from feeling overwhelming. +- **Pain points:** can't remember where things are; newsletters bury real mail; unsubscribing is tedious; search needs the "right words." +- **Daily workflow:** skim inbox on phone/desktop → star/leave a few → hunt for a receipt or booking → ignore the rest. +- **Search needs:** *natural, forgiving* ("flight to Lisbon", "gym receipt March") — no operators. Attachment/receipt finding. Typo tolerance. +- **AI opportunities:** thread summaries, "what needs a reply," smart unsubscribe suggestions, receipt/booking extraction, gentle priority. + +## 🟢 Marcus — Business owner / solopreneur (mainstream, high-value) +Runs a small business from one inbox: clients, vendors, invoices, admin — **no assistant**. +- **Goals:** never drop a client ball; find any past agreement/invoice instantly; spend less time in email. +- **Pain points:** follow-ups slip; important buried under admin; digging for "what did we agree?"; context-switching. +- **Daily workflow:** triage first thing → reply to clients → chase unpaid invoices → search for prior context mid-reply. +- **Search needs:** people-centric ("everything with this client"), attachment/invoice search, timeline ("our thread about the contract"), "did they reply?" +- **AI opportunities:** follow-up detection, "you haven't heard back" nudges, thread/relationship summaries, task/commitment extraction, draft replies with context. + +## 🔵 Dev — Developer / power user +Technical inbox: GitHub, CI alerts, monitoring, mailing lists, plus real mail. +- **Goals:** cut notification noise; automate; keep signal. **Strongly values local/private AI** and self-hostable. +- **Pain points:** alert floods; wants rules/automation and keyboard speed; distrusts cloud AI on their mail. +- **Daily workflow:** bulk-archive alerts → scan PR/issue threads → deep-search history → automate the repetitive. +- **Search needs:** operators + regex-ish precision, entity/sender/domain, saved searches, fast keyboard-driven search. +- **AI opportunities:** local summarisation of long threads, auto-labelling/rules suggestions, semantic search across archives, duplicate/near-duplicate detection. + +## 🔵 Rina — Recruiter (role power user, high volume) +Hundreds of candidate threads, CV attachments, scheduling, multi-stage follow-ups. +- **Goals:** find any candidate/CV instantly; track stage; never lose a promising lead. +- **Pain points:** attachments scattered; who's at what stage; follow-up timing; duplicate applicants. +- **Search needs:** **attachment/document search** (search *inside* CVs), people search, stage/timeline, entity extraction (skills, roles). +- **AI opportunities:** CV/document understanding, candidate summaries, follow-up detection, duplicate detection, relationship mapping. + +## 🔵 Sam — Sales · 🔵 Tom — Support (role power users) +- **Sam:** pipeline follow-ups, response tracking, templates. Needs "who hasn't replied," saved searches, sentiment, reply suggestions. +- **Tom:** triage by category/SLA, canned responses, categorisation accuracy. Needs reliable auto-categorisation, priority prediction, phishing/spam confidence, thread summaries. +- (Both hint at **collaboration/shared-inbox** — deliberately a *later* opportunity; individual mainstream user is primary.) + +--- + +## Cross-persona signal +| Need | Who feels it most | Priority | +|------|-------------------|----------| +| **Forgiving, natural search** | Priya, Marcus (everyone) | 🔴 Highest | +| **Find people / attachments / past context** | Marcus, Rina, Dev | 🔴 High | +| **Follow-up / "did they reply?" detection** | Marcus, Sam, Rina | 🔴 High | +| **Thread & relationship summaries** | Marcus, Dev, Rina | 🟠 Medium-high | +| **Noise reduction (unsubscribe, bulk, categorise)** | Priya, Dev, Tom | 🟠 Medium-high | +| **Local/private AI** | Dev (loud), all (latent) | 🔴 Strategic | +| **Explainability of results/priority** | everyone (trust) | 🟠 Medium-high | +| Collaboration / shared inbox | Sam, Tom | 🟢 Later | + +--- + +## Journey maps +Format: **stage → what they do → current pain → InboxIntel opportunity.** + +### J1 — Morning triage ("what needs *me* today?") — Priya, Marcus +| Stage | Action | Current pain | Opportunity | +|-------|--------|--------------|-------------| +| Arrive | Open inbox | Wall of mixed noise + important | AI **priority lane** + summary of "needs you" (local, explainable) | +| Scan | Skim subjects | No signal of what's urgent/awaiting reply | **Follow-up/awaiting-reply** badges; thread one-liners | +| Act | Reply/defer/clean | Repetitive, context-switch heavy | One-click defer/snooze; context-aware **draft reply**; bulk clean noise | +| Exit | Close, hope nothing missed | Anxiety of missed items | "You're caught up on what matters" confidence state | + +### J2 — Find one specific thing (the flagship moment) — everyone +| Stage | Action | Current pain | Opportunity | +|-------|--------|--------------|-------------| +| Recall | Remember fragments ("invoice, that vendor, spring") | Must guess exact keywords/operators | **Natural-language + semantic** search; typo-tolerant | +| Query | Type in search | Syntax anxiety; blank box | **Suggestions, recent, filter chips**; search-as-home | +| Scan results | Look through hits | Date-sorted, not relevant; no context | **Relevance ranking** + **"why this matched"** + instant preview | +| Confirm | Open the right one | Re-open several to be sure | Grouped results, inline preview, people/attachment facets | + +### J3 — Reduce the noise — Priya, Dev, Tom +| Stage | Action | Current pain | Opportunity | +|-------|--------|--------------|-------------| +| Notice | "Too many newsletters" | Unsub links hidden/dark-patterned | **One-click unsubscribe** (already seeded) + AI suggestions of what to cut | +| Decide | Which to keep? | Manual, per-email | AI **bulk suggestions** by sender/category with confidence + preview | +| Act | Unsubscribe / bulk clean | Risky, irreversible-feeling | **Confirmed + previewed** bulk actions (already a strength) | +| Trust | Did it do the right thing? | Opaque | Explainable "why suggested," undo, audit | + +### J4 — Track a commitment / follow-up — Marcus, Sam, Rina +| Stage | Action | Current pain | Opportunity | +|-------|--------|--------------|-------------| +| Send | Email a client/candidate | Then forget | AI **follow-up detection**: "expecting a reply?" | +| Wait | Time passes | No system nudges you | "**No reply in N days**" surfaced automatically | +| Recall | "What did we agree?" | Dig through thread | Thread **summary + extracted commitments/tasks** | +| Resolve | Chase / close | Re-draft from scratch | Context-aware **draft nudge** referencing the thread | + +--- + +## Research-driven product principles +1. **Search is the front door**, and it must work for someone who types *"that gym receipt"* — not just `from:gym has:attachment`. +2. **Explain everything** the system decides (match, category, priority) — it builds trust and teaches the app. +3. **AI removes toil, never control** — advisory, previewed, undoable, and fully optional/local. +4. **Progressive power** — mainstream-simple by default; operators, saved searches, automation revealed as users grow. +5. **Reduce anxiety** — "you're caught up," undo, and confidence states matter as much as features. diff --git a/docs/discovery/04-ux-redesign-and-design-system.md b/docs/discovery/04-ux-redesign-and-design-system.md new file mode 100644 index 0000000..4afcd62 --- /dev/null +++ b/docs/discovery/04-ux-redesign-and-design-system.md @@ -0,0 +1,182 @@ +# 04 — UX/UI Redesign, Information Architecture & Design System (Phase 4A) + +A **complete redesign**, not an iterative tweak. Nothing about the current layout is +assumed to survive. Governed by the [Design Brief](00-design-brief.md): Notion/Arc- +professional, pointer-first & discoverable, balanced density, dark-first, green +`#3ba31f`, subtle motion, fully responsive, mainstream-friendly. + +--- + +## Part A — UX Research (per-screen evaluation) +Evaluating the *jobs*, not the current pixels. For each screen: **goal · is it intuitive +· what's unnecessary · how to simplify · click reduction · what must be obvious · sources +of cognitive overload.** + +| Screen | User goal | Redesign verdict | +|--------|-----------|------------------| +| **Dashboard (current draggable widgets)** | Understand my inbox | Becomes the **Analytics** view, not the home. A draggable widget grid is *configuration overhead* most users never want. Default = a curated overview; customisation is progressive. | +| **Inbox/list** | Triage what matters | Reframe from "all mail newest-first" to **lanes** (Needs you · Awaiting reply · Everything). Obvious: who/subject/one-line intent/time. Overload source: undifferentiated noise → fix with grouping + priority. | +| **Reading a message** | Understand + act | Add a **thread summary** header, **extracted actions/dates**, and inline reply. Reduce clicks: reply/snooze/label as one-key or one-click from the pane. | +| **Search** | Find a specific thing | **Promote to the front door.** Today it's an operator box; redesign to inviting, visual, assisted (see [05](05-search-redesign.md)). | +| **Cleanup / unsubscribe** | Reduce noise safely | Strong bones (confirm+preview). Make it **suggestion-led** ("cut these 12 newsletters?") with confidence + undo. | +| **Settings** | Configure incl. AI | Add a clear **AI panel**: off / local (Ollama) / provider, model status, VRAM. AI-off must feel first-class, not degraded. | + +**Cross-cutting UX principles:** search-as-home · lanes over one big list · explain every +decision · progressive disclosure of power · confidence/undo everywhere · one primary +action per screen. + +--- + +## Part B — Information Architecture + +### Navigation shell (responsive 3-pane → 1-pane) +``` +┌───────────────────────────────────────────────────────────────┐ +│ TopBar: [ 🔍 Search your inbox… ⌘K ] ☾ ⚙ 👤 │ +├──────────┬──────────────────────────┬─────────────────────────┤ +│ Sidebar │ List / Results │ Reading / Preview │ +│ (collaps)│ (virtualised) │ (thread + AI summary) │ +│ │ │ │ +│ Search │ ▸ Needs you (lane) │ Subject │ +│ Priority │ ▸ Awaiting reply │ ⟶ AI summary (local) │ +│ Unread │ ▸ Everything │ ⟶ extracted actions │ +│ Saved ★ │ │ body … │ +│ Categories │ [Reply] [Snooze] […] │ +│ Cleanup │ │ │ +│ Analytics│ │ │ +│ ─────────│ │ │ +│ 👤 acct │ │ │ +└──────────┴──────────────────────────┴─────────────────────────┘ +``` +- **Search sits at the top of everything** (top bar) *and* as the first sidebar item — reinforcing search-as-home. +- **Responsive collapse:** 3-pane (wide desktop) → 2-pane (list+reading, laptop) → 1-pane with push navigation (tablet/mobile). Reading opens as an overlay sheet on mobile. +- **Progressive disclosure:** advanced filters, operators, saved-search management, and automation are revealed on demand — never in a beginner's face. +- **Command palette (⌘K / Ctrl+K):** an *accelerator* — navigate, act, and search — layered on top of the fully clickable UI (pointer-first per brief). +- **Right-click context menus** on rows/senders/threads (archive, label, unsubscribe, "find similar," "everything from this sender"). +- **Multi-window / dockable panels:** deferred (web app); pop-out reading view is a v-later opportunity. + +### Screen hierarchy +1. **Search-home** (front door) · 2. **List/Results** (lanes, ranked) · 3. **Reading/thread** +· 4. **Cleanup** · 5. **Analytics** · 6. **Settings (incl. AI)**. + +--- + +## Part C — Design System + +### Colour — accent ramp (from `#3ba31f`) +``` +green-50 #f890? → use tint set: +--green-50: #f1f9ec --green-300: #93d07d --green-600: #2f8419 +--green-100: #dcf0d0 --green-400: #63b84a --green-700: #266a15 +--green-200: #bde3ab --green-500: #3ba31f --green-800: #1e5312 + (brand base) --green-900: #163a0e +``` +**Usage rules (dark-first):** +- Brand/base = `green-500`. On dark surfaces, interactive fills use `green-500`/`green-400`; **foreground on accent is contrast-checked** (near-black `#0f1a0b` on light greens, white on `green-600`+). +- Accent is used **sparingly** — primary actions, selection, active nav, positive status. +- **Never colour-only:** selection also shows a left-bar/checkbox; status pairs green with an icon/label (colour-blind-safe by construction, even though formal a11y is deferred). + +### Colour — neutrals (warm-leaning) +| Token | Dark (default) | Light | +|-------|----------------|-------| +| `--bg` | `#1a1917` (warm charcoal, **not** pure black) | `#ffffff` (genuine white) | +| `--surface-1` | `#211f1d` | `#faf9f7` | +| `--surface-2` | `#2a2724` | `#f4f2ee` | +| `--surface-3` | `#34302c` | `#ebe8e2` | +| `--border` | `#3a3632` | `#e4e0d9` | +| `--text` | `#f2efe9` (warm off-white) | `#1c1a17` | +| `--text-muted` | `#a8a29a` | `#6b6459` | +| `--text-subtle` | `#7a746c` | `#928b7e` | +| semantic | `info #4a90d9 · warn #d9a441 · danger #d95a4a · success = green-500` | same, contrast-tuned | + +### Typography +- **UI font:** Inter (or system fallback) — clean, neutral, highly legible. +- **Optional mono:** JetBrains Mono / ui-monospace for addresses, IDs, data. +- **Scale (px / line-height), UI base = 14 for balanced density:** + `xs 12/16 · sm 13/18 · base 14/20 · md 16/24 · lg 18/26 · xl 20/28 · 2xl 24/32 · 3xl 30/38` +- Weights: 400 body · 500 UI/labels · 600 headings/emphasis. Avoid 700 except brand. + +### Spacing (4px base) & layout +`space: 2, 4, 6, 8, 12, 16, 20, 24, 32, 40, 48, 64`. +Grid: 12-col fluid content area; sidebar fixed (240px, collapsible to 56px icon rail); +reading pane min 420px. Density "balanced" → row height ~44px, 8–12px internal padding. + +### Radius / elevation / motion +- **Radius:** `sm 4 · md 6 (buttons/inputs) · lg 8 (cards/panels, default) · xl 12 (modals) · pill 999`. +- **Elevation:** dark = surface-layering + faint shadow + 1px border; light = soft shadows + `e1 0 1 2 /6% · e2 0 4 12 /10% · e3 0 12 32 /16%`. Levels: e0 flat · e1 menus · e2 popovers · e3 modals. +- **Motion:** durations `120 / 180 / 240ms`; easing `cubic-bezier(0.2,0,0,1)` (ease-out) for enters, `cubic-bezier(0.4,0,1,1)` for exits; a spring only for selection/drag. **Respect `prefers-reduced-motion`.** + +### Icons & illustration +- **Lucide** (line, rounded), stroke 1.5px, 20px default (16px dense, 24px feature). +- Illustration: minimal, single-accent line spot-art for empty states — friendly, not corporate stock. + +### States (must all be designed) +- **Loading:** skeleton rows (list), shimmer summary card (reading) — never spinners for content. +- **Empty:** friendly line-art + one clear CTA ("Nothing here yet — connect Gmail" / "No results — try broader terms" with a *Did you mean* / *Broaden* action). +- **Error:** calm, specific, recoverable ("Couldn't reach Gmail — Retry"), never a raw stack. +- **Success:** toast + inline confirmation; destructive actions show **preview → confirm → undo**. + +### Components (catalogue) +Buttons (`primary` green / `secondary` surface / `ghost` / `danger`) · icon-button · input +· **search field** (hero variant) · **filter chip** (removable, typed) · segmented control · +toggle · dropdown menu · **context menu** · **command palette** · **email row** (avatar, +sender, subject, one-line intent, badges, time, hover-actions) · **sender chip/avatar** · +**thread summary card** · **category badge** · **priority indicator** · tabs · tooltip · +toast · modal · **side sheet** (mobile reading) · skeletons · empty-state · pagination / +**virtualised infinite scroll** · avatar/initials · progress/VRAM meter (AI panel). + +--- + +## Part D — Themes (both fully polished) + +### Dark (default) +Layered **warm charcoal** surfaces (`#1a1917` → `#34302c`), warm off-white text, green +accent nudged for on-dark contrast, faint borders to separate layers. Avoids pure black; +depth via surface elevation + hairline borders, not heavy shadow. + +### Light +**Genuine white** base (`#ffffff`) with warm off-white surfaces — *not grey-pretending-to- +be-white*. Generous whitespace, soft shadows for elevation, restrained green accent. +Premium, low-noise, highly readable. + +Both share tokens; only the neutral map + shadow strategy differ. Theme follows a +`data-theme` attribute; **dark is the design source of truth**, light is derived and +independently QA'd. + +--- + +## Part E — Interaction design +- **Hover:** rows raise to `surface-2`, reveal quick-actions (archive/snooze/label/unsub). +- **Selection:** checkbox on hover + click-row-to-open; shift/⌘-click multi-select; a + sticky **bulk action bar** slides up when >1 selected. +- **Search:** instant results, **live filtering** as chips are added/removed, suggestions + + recent on focus (see [05](05-search-redesign.md)). +- **Previews:** hover peek + inline reading; attachments preview in a lightbox. +- **Drag & drop:** rows → labels/categories/cleanup; respects reduced-motion. +- **Notifications:** toasts (non-blocking), with undo for reversible actions. +- **Transitions:** pane content crossfades; mobile reading slides up as a sheet. +- **Scrolling:** **virtualised list** (mandatory for 100k+ rows) with sticky lane headers. + +--- + +## Part F — Accessibility review (baseline; deeper a11y deferred) +Baked in cheaply now: AA-tuned contrast via the ramps, visible focus rings, `prefers- +reduced-motion`, **non-colour-only** state, semantic HTML + ARIA on lists/dialogs/menus, +full keyboard reachability of core actions. **Deferred to pre-launch backlog** (per brief): +high-contrast mode, formal screen-reader passes, font-scaling controls, colour-blind audit. + +--- + +## Part G — Migration strategy (strangler, low-risk) +1. **Introduce tokens + component library** (Tailwind config from this doc) — no behaviour change. +2. **Rebuild the shell** (sidebar / top-bar / search-home) around the existing API. +3. **Migrate screen-by-screen behind a feature flag** (`ui.v2`): Search → Reading → List → + Cleanup → Analytics (the old draggable dashboard *becomes* Analytics). +4. **Keep the API stable**; frontend-only migration. Delete old screens once parity is verified. +5. Ship per-screen via the `develop → staging` pipeline; each screen is its own epic (see [Git plan](10-git-plan.md)). + +## Part H — Future design opportunities +Pop-out / multi-window reading · dockable panels · custom accent picker · additional themes +· a Tauri/Electron shell if a true desktop build is ever wanted · plugin-contributed widgets +on the Analytics canvas · command-palette extensibility. diff --git a/docs/discovery/05-search-redesign.md b/docs/discovery/05-search-redesign.md new file mode 100644 index 0000000..6a8e702 --- /dev/null +++ b/docs/discovery/05-search-redesign.md @@ -0,0 +1,129 @@ +# 05 — Search Redesign (Phase 4B) + +Search is the **flagship**. The goal: make people *want to search rather than browse* — +an experience that's **inviting, visual, forgiving, ranked, and explainable**, works for +*"that gym receipt from March"* (not just `from:gym has:attachment`), and runs **locally +and privately**. Grounded in the existing Postgres FTS + `IAiProvider` seam (see +[01](01-architecture-review.md)); AI modes are **optional** — with AI off, lexical + +structured search is still excellent. + +--- + +## The layered search engine +Four cooperating layers, each usable alone; higher layers *degrade gracefully* to lower +ones when AI is disabled. + +``` +┌ Layer 4 · AI-ASSISTED (optional, local) ────────────────────────────┐ +│ NL→query · conversational "ask your inbox" · explanations · related │ +├ Layer 3 · SEMANTIC (optional, local embeddings + pgvector) ──────────┤ +│ meaning-based recall · "similar to this" · concept search │ +├ Layer 2 · LEXICAL + RANKING (Postgres FTS, always on) ───────────────┤ +│ websearch_to_tsquery · ts_rank · pg_trgm fuzzy · multi-field │ +├ Layer 1 · STRUCTURED (SQL WHERE, always on) ─────────────────────────┤ +│ sender/date/labels/flags/size/category · filter chips │ +└───────────────────────────────────────────────────────────────────────┘ + ▲ hybrid fusion (RRF) ranks Layer 2+3 together ▲ +``` + +**Hybrid ranking (RRF):** lexical and semantic candidate sets are merged via *Reciprocal +Rank Fusion*, then boosted by **recency**, **sender importance**, and **unread**. This +replaces today's date-only ordering — the single biggest search win. + +--- + +## The experience: search as the front door +- **Hero search field** in the top bar + a **search-home** when the box is focused/empty: + **recent searches**, **suggested searches** ("Unread from people you reply to", + "Large attachments", "Receipts this month"), and **saved searches ★**. +- Type freely → **live results** with **filter chips** the user can add/remove by click + (pointer-first). Behind an unobtrusive "＋ Filters" for the full builder. +- Every result shows **"why it matched"** (highlighted terms, or the semantic concept, or + the parsed NL interpretation as *editable chips*). +- **Zero-result recovery:** *Did you mean* · *Broaden* · *Search all mail/trash*. +- Fully keyboard-drivable as an **accelerator** (⌘K → type → arrow → enter), but never required. + +--- + +## Search modes & features +Each with **Problem solved · Implementation · Complexity · Effort · User value.** +Effort = rough dev-days for one engineer; Complexity = S/M/L/XL. + +| Feature | Problem solved | Implementation | Cx | Effort | Value | +|---------|----------------|----------------|----|--------|-------| +| **Relevance ranking** | Results sorted by date, not usefulness | `ts_rank_cd` + hybrid RRF + recency/sender/unread boosts; top-N by score | M | 3–5d | 🔴🔴🔴 | +| **Multi-field FTS** | Only Subject+Body indexed | Extend generated `tsvector` (sender name/addr, attachment filenames, labels) with weights (A/B/C/D) | S | 2–3d | 🔴🔴 | +| **Fuzzy / typo tolerance** | "recieved" finds nothing | `pg_trgm` GIN + similarity fallback when FTS is empty | S | 1–2d | 🔴🔴 | +| **Phrase / prefix / boolean** | `plainto_tsquery` too blunt | Swap to `websearch_to_tsquery` (quotes, OR, -exclude) | S | 1d | 🔴 | +| **Interactive filter chips** | Operators are a syntax wall | UI chips ↔ structured DTO; add/remove live | M | 3–4d | 🔴🔴🔴 | +| **Advanced query builder** | Power users want precision | Visual builder → same DTO; operators still work | M | 3–4d | 🔵🔵 | +| **Natural-language search** | "gym receipt March" | Local LLM (or rules-first) parses intent → chips + terms, shown editable | L | 5–8d | 🔴🔴🔴 | +| **Semantic search** | Recall by *meaning*, not keywords | `IEmbeddingProvider` (Ollama) → `pgvector` (HNSW); backfill job; hybrid with FTS | L | 8–12d | 🔴🔴🔴 | +| **Conversational "ask your inbox"** | "What did Sarah say about the invoice?" | RAG: semantic retrieve → local LLM answers with **citations** to source emails | XL | 10–15d | 🔴🔴🔴 | +| **"Why this matched"** | Trust + learnability | `ts_headline` highlights (lexical); nearest-concept + snippet (semantic); parsed-intent chips (NL) | M | 3–4d | 🔴🔴 | +| **Saved searches ★** | Repeated queries | Persist DTO; pin to sidebar; optional live count | S | 2d | 🔴🔴 | +| **Recent searches / history** | Re-find what you searched | Per-user history table; privacy-clearable | S | 1–2d | 🔴 | +| **Suggested searches** | Blank-box paralysis | Rules over user's own stats (top senders, unreplied, big attachments) | S | 2–3d | 🔴🔴 | +| **People search** | "Everything with this person" | Sender/recipient facet + aggregation; person profile panel | M | 3–5d | 🔴🔴 | +| **Attachment search** | Find files, not emails | Index filenames now; **content/OCR** later (see below) | S→L | 2d → +8d | 🔴🔴 | +| **Entity extraction** | Search by amounts/dates/orgs | Local NER (LLM or rules) → entity index; facets (money, dates, companies) | L | 6–10d | 🔴🔴 | +| **Timeline search** | "our thread over time" | Date-bucketed results + a timeline scrubber UI | M | 3–4d | 🔵🔵 | +| **Related conversations** | Surface adjacent context | Vector nearest-neighbours + same-participants/thread heuristics | M | 3–5d | 🔴🔴 | +| **Duplicate / near-dup detection** | Clutter, repeated sends | Hash (exact) + embedding similarity (near) → group/cleanup | M | 4–6d | 🔵🔵 | +| **Conversation intelligence** | Long threads are walls | Thread summary + extracted actions/decisions (local LLM) surfaced in results & reading | L | 8–12d | 🔴🔴🔴 | + +### Additional ideas (beyond the brief) +- **Search-driven bulk actions** — run cleanup/label/unsub on a *result set* ("archive all newsletters older than 6 months"). +- **Scoped search** — constrain to this folder / label / person / thread inline. +- **"Find similar to this email"** — one-click semantic neighbours from any message. +- **Zero-result recovery** — *did-you-mean* (trigram), *broaden*, *search trash/all*. +- **Search from selection** — highlight text → "search inbox for this". +- **Sender reputation / safety facet** — filter by phishing/spam confidence (ties to AI). +- **Search templates** — parameterised saved searches ("invoices from {client}"). +- **Search analytics (for the user)** — "you search 'invoice' most" → suggests a saved search / smart folder. + +--- + +## Ranking design (detail) +``` +score = RRF(lexical_rank, semantic_rank) + + w_recency · decay(sentAt) + + w_sender · senderImportance(userReplyRate, frequency) + + w_unread · isUnread +``` +- Weights are configurable; defaults tuned so **exact/lexical hits never lose to fuzzy noise**. +- `senderImportance` derives from the user's *own* behaviour (reply rate, frequency) — + computable from existing data, no AI required. + +## Performance +- **Indexes:** GIN on `SearchVector`; GIN `pg_trgm` on sender/subject; **HNSW** (or IVFFlat) + on the `pgvector` embedding column. +- **Keyset/cursor pagination** for lexical/structured (replaces offset `Skip/Take`); score- + ordered top-N for ranked/semantic (fetch N, no deep offset). +- **Async embedding backfill** worker (reuse the hosted-worker pattern) so semantic search + builds in the background; new mail embedded on sync. Batch to respect the 3080's VRAM. +- Cache suggestions/recent; debounce live search (~120ms). +- Target: **<150ms** lexical, **<400ms** hybrid on 100k emails (local). + +## Privacy +- Embeddings and NER computed **locally via Ollama**, stored **locally** in Postgres. +- Nothing leaves the machine unless the user explicitly selects a cloud provider. +- **AI-off fallback:** Layers 1–2 (structured + ranked lexical + fuzzy) — still a top-tier + search. Semantic/NL/conversational simply hide when AI is disabled. + +## Mapping to the existing code (extend, don't rewrite) +- **`GmailQueryParser` → `SearchIntentParser`:** keep operator parsing; add a rules-first + NL layer, optional local-LLM disambiguation, emit the same `SearchRequestDto` (+ new fields). +- **`SearchService`:** add ranking (`ts_rank_cd`), `websearch_to_tsquery`, `pg_trgm` + fallback, multi-field vector, hybrid RRF, keyset pagination, `ts_headline` explanations. +- **`Email.SearchVector`:** widen the generated column (weighted A/B/C/D) + add a + `pgvector` `Embedding` column + `EmailEntities`/`ThreadSummary` tables. +- **AI seam:** extend to `IEmbeddingProvider.EmbedAsync` and a structured + `IAiProvider.CompleteStructuredAsync` (see [06 AI Strategy](06-ai-strategy.md)); Null + provider returns "unavailable" so callers fall back to lexical. + +## Phasing (feeds the roadmap) +- **MVP search:** relevance ranking · multi-field FTS · fuzzy · chips · saved/recent/suggested · "why matched" (lexical) · keyset pagination. +- **v1.1:** natural-language parse · people search · attachment (filename) · search-driven bulk. +- **v1.2:** semantic (pgvector) · related · find-similar · thread summaries in results. +- **v2.0:** conversational "ask your inbox" (RAG + citations) · entity search · duplicate detection · attachment OCR/content. diff --git a/docs/discovery/06-ai-strategy.md b/docs/discovery/06-ai-strategy.md new file mode 100644 index 0000000..aca50d0 --- /dev/null +++ b/docs/discovery/06-ai-strategy.md @@ -0,0 +1,113 @@ +# 06 — AI Strategy & Model Recommendations (Phase 5) + +AI is **optional, modular, local-first, and explainable**. The app must be fully usable +with AI disabled. Primary hardware: **NVIDIA RTX 3080 (10 GB VRAM)**. All AI sits behind +an abstraction so models/providers swap without touching application logic. + +--- + +## Part A — The AI abstraction (extends the existing seam) +Today there is one interface, `IAiProvider.CompleteAsync(system, user)`, with +`NullAiProvider` / `OllamaProvider` / `OpenAiProvider`. We extend it into a small set of +capability interfaces plus a task-oriented facade. + +### Provider-level interfaces (Infrastructure) +```csharp +public interface IAiProvider { // chat / generation (exists) + AiProviderMode Mode { get; } + Task CompleteAsync(string system, string user, CancellationToken ct = default); + // NEW: schema-constrained JSON (Ollama `format`, OpenAI `response_format`) + Task CompleteStructuredAsync(string system, string user, CancellationToken ct = default); +} + +public interface IEmbeddingProvider { // NEW: vectors for semantic search/dedup + Task EmbedAsync(string text, CancellationToken ct = default); + Task> EmbedBatchAsync(IReadOnlyList texts, CancellationToken ct = default); +} + +public interface IAiCapabilities { // NEW: lets the UI hide what isn't available + bool Chat { get; } bool Embeddings { get; } bool Vision { get; } bool StructuredJson { get; } +} +``` +`NullAiProvider`/`NullEmbeddingProvider` return empty/"unavailable" so **every caller has a +graceful non-AI path** (fall back to lexical search, heuristics, or hide the feature). + +### Task facade (Application) — the only thing feature code calls +```csharp +public interface IInboxAi { + Task> SummarizeThreadAsync(Guid threadId, CancellationToken ct); + Task> ExtractAsync(string body, ExtractKinds kinds, CancellationToken ct); + Task>ClassifyAsync(string subject, string body, CancellationToken ct); + Task> AskAsync(string question, SearchScope scope, CancellationToken ct); // RAG + Task EmbedAsync(string text, CancellationToken ct); + Task> AssessPhishingAsync(EmailContext ctx, CancellationToken ct); +} +``` +- `AiResult` carries `{ value, available, model, latencyMs, explanation }` → powers the + **"explainability"** and graceful-degradation requirements everywhere. +- A **model router** maps *logical task → model name* from config, so swapping a model is a + config change: `Ai:Models:{Chat,Summarize,Embed,Classify,Extract,Vision}`. +- Prompt templates are **versioned files**, separate from code. +- Cross-cutting: async, `CancellationToken`, per-call timeout + **Polly** fallback to the + Null path, and a token/VRAM-aware concurrency limiter. + +### Config shape +``` +Ai:Mode = Disabled | LocalOllama | Cloud... +Ai:Ollama:BaseUrl, KeepAliveSeconds, VramBudgetMb +Ai:Models:Chat = qwen2.5:7b-instruct +Ai:Models:Embed = nomic-embed-text +Ai:Models:Vision = qwen2.5vl:7b (load-on-demand) +``` + +--- + +## Part B — Model recommendations for the RTX 3080 (10 GB) + +**Budget reality:** a 7–8B instruct model at Q4/Q5 (~5 GB + ~1 GB KV cache at 8k context) +runs **alongside** a small embedding model (~0.5 GB) inside 10 GB with headroom. 14B is +possible (~9 GB) but leaves no room for concurrency and is slower — **7–8B is the sweet spot.** + +| Task | Recommended (primary) | Alt | ~VRAM (Q4) | Latency | License | Notes | +|------|----------------------|-----|-----------|---------|---------|-------| +| **Chat / reasoning / RAG answers** | **Qwen2.5-7B-Instruct** | Llama-3.1-8B, Gemma-2-9B | ~5.5 GB | ~30–60 tok/s | Apache-2.0 | Strong reasoning, multilingual, great JSON | +| **Summarisation** | reuse **Qwen2.5-7B** | Llama-3.1-8B | (shared) | fast | Apache-2.0 | No separate model needed | +| **Extraction (tasks/dates/entities)** | reuse **Qwen2.5-7B** + `format:json` | Phi-4 | (shared) | fast | Apache-2.0 | Schema-constrained JSON output | +| **Classification / tagging** | **Qwen2.5-3B** (fast) *or* embeddings-zero-shot | rules-first (exists) | ~2.2 GB | very fast | Apache-2.0/Qwen | High-volume → small model or embeddings, not 7B | +| **Embeddings (semantic search, dedup, zero-shot class.)** | **nomic-embed-text** (768-d, 8k ctx) | mxbai-embed-large, **bge-m3** (multilingual) | ~0.5 GB | very fast | Apache-2.0 | **Keep permanently loaded** | +| **OCR / attachment vision** | **Qwen2.5-VL-7B** *or* **MiniCPM-V** | llama3.2-vision-11B, moondream (tiny) | ~7 GB | slow | Apache/varied | **Load-on-demand**; evicts the text LLM | +| **Language detection** | **library, not an LLM** (fastText-lid / CLD3) | — | ~0 | instant | MIT/Apache | Don't waste VRAM on this | +| **Translation** | reuse **Qwen2.5-7B** (multilingual) | NLLB-200 (dedicated MT) | (shared) | med | Apache-2.0 | Good for common pairs; on-demand | +| **Spam detection** | **rules + small classifier/embeddings** | 3B LLM for edge cases | ~0–2 GB | fast | — | Traditional-first; LLM only for ambiguity | +| **Phishing detection** | **rules/URL analysis + embeddings**, LLM **reasoning** for suspicious | Qwen2.5-7B for explanation | shared | on-demand | — | LLM explains *why*; runs async on flagged mail | +| **Duplicate detection** | **hashing (exact) + embedding cosine (near)** | — | (uses embed) | fast | — | No chat model required | + +**Selection rationale:** accuracy + **Apache-2.0 licensing** (commercial-safe) + fits 10 GB ++ strong **structured-JSON** and multilingual (mailboxes aren't English-only). Qwen2.5-7B is +a single versatile workhorse for chat/summarise/extract/translate; nomic-embed-text is a +tiny always-on retrieval engine; a vision model is a heavy, rare, on-demand guest. + +### Load policy (VRAM management) +| State | Models | Rationale | +|-------|--------|-----------| +| **Hot (permanent)** | `nomic-embed-text` (~0.5 GB) | Used constantly (search, dedup, classify, backfill) — must be instant | +| **Warm (keep-alive)** | `qwen2.5:7b-instruct` (~5.5 GB) | Loaded on first chat/summarise, kept warm ~5–10 min idle via Ollama `keep_alive`, then unloaded | +| **Cold (on-demand)** | `qwen2.5vl:7b` vision, `nllb` translation | Loaded only for the specific job; unloaded after (would otherwise exceed budget alongside the 7B) | +- Ollama handles load/unload; we set `keep_alive` per task and a **VRAM budget guard** that + serialises a vision job behind unloading the text LLM. +- Embedding **backfill** runs as a hosted background worker, batched, low priority, so it + never starves interactive requests. + +--- + +## Part C — Principles +1. **Traditional-first.** If rules/heuristics/library solve it well (language detection, + exact dedup, basic spam, most categorisation), use them — AI only where it *clearly* wins. +2. **Always optional.** Null providers + capability flags → the app is whole without AI. +3. **Local & private by default.** Ollama on-device; data never leaves unless a cloud + provider is explicitly chosen. +4. **Explainable.** Every AI output carries a short rationale + the model used. +5. **Swappable.** Logical-task→model routing via config; prompts versioned; no feature code + references a model name. +6. **Bounded.** Timeouts, cancellation, Polly fallback, VRAM-aware concurrency — AI can never + hang or crash the core experience. diff --git a/docs/discovery/07-ai-feature-catalogue.md b/docs/discovery/07-ai-feature-catalogue.md new file mode 100644 index 0000000..2c650b1 --- /dev/null +++ b/docs/discovery/07-ai-feature-catalogue.md @@ -0,0 +1,70 @@ +# 07 — AI Feature Catalogue (Phase 6) + +Every realistic AI feature, judged honestly against the guiding rule: **AI only where it +clearly beats traditional approaches.** Each entry: *problem · AI role & why · could +traditional solve it? · complexity · performance · privacy.* All AI is local (Ollama), +optional, advisory, and explainable (see [06](06-ai-strategy.md)). Complexity = S/M/L/XL. + +Legend for the verdict column: +🟢 **AI clearly wins** · 🟡 **traditional-first, AI for the hard tail** · ⚪ **not AI at all** + +--- + +## Group 1 — 🟢 AI clearly wins +LLM/embeddings are genuinely the right tool; traditional approaches are weak here. + +| Feature | Problem | Why AI (and why traditional falls short) | Cx | Perf | Privacy | +|---------|---------|------------------------------------------|----|------|---------| +| **Thread / conversation summary** | Long threads are walls of text | LLMs summarise free text; regex/extractive summaries miss nuance & context | L | Warm 7B; cache per-thread, invalidate on new msg | Body → local LLM only | +| **Conversational "ask your inbox"** (RAG) | "What did Sarah say about the invoice?" | Retrieval + generation over many emails; impossible with filters alone | XL | Semantic retrieve → 7B answer w/ **citations**; ~1–3s | Retrieval + gen fully local | +| **Reply suggestions / writing assistant** | Blank-page drafting, tone | LLM drafts context-aware replies; templates can't adapt to content | L | Warm 7B, streamed | Thread context → local | +| **Task / meeting / calendar / reminder extraction** | Commitments hide in prose | LLM structured-JSON extraction of {task, date, attendee}; regex catches only rigid formats | L | 7B `format:json`, async on read/sync | Body → local | +| **Entity extraction** (amounts, orgs, dates, order #s) | Can't search/facet by meaning | LLM/NER generalises across phrasings; regex is brittle per-vendor | L | 7B or small NER, batched at sync | Local | +| **Document / attachment understanding** | Can't search *inside* files | OCR/vision + summarise; no traditional equivalent for images/PDF meaning | XL | Vision model **on-demand** (heavy); OCR async | File content → local | +| **Cross-thread linking / related conversations** | Related context is scattered | Embedding nearest-neighbours find semantic links; keyword join misses paraphrase | M | pgvector HNSW; precomputed | Vectors local | +| **Relationship mapping / knowledge graph** | No view of who/what connects | Extraction + embeddings build a people/topic graph; not expressible in SQL alone | XL | Batch build; incremental | Local graph store | +| **Conversation insights** (decisions, sentiment shift) | "What was decided / how's this going?" | LLM reads intent/sentiment over a thread; rules can't | L | 7B; cache | Local | +| **Sentiment analysis** | Gauge tone (angry client?) | Small model/LLM classifies tone; lexicon methods are crude/misleading | M | small model or embeddings | Local | +| **Email comparison** ("what changed vs last quote?") | Manual diffing of prose | LLM semantic diff; text-diff shows characters, not meaning | M | 7B on two bodies | Local | +| **Explain search results / decisions** | Trust & learnability | For semantic/NL, only the model can say *why*; lexical uses `ts_headline` (non-AI) | M | cheap (reuse retrieval) | Local | + +## Group 2 — 🟡 Traditional-first, AI for the hard tail +Heuristics/rules do 70–90% cheaply and instantly; AI handles ambiguity and adds explanations. +The existing `HeuristicClassifier` and unsubscribe signals are the traditional backbone. + +| Feature | Problem | Traditional core | Where AI adds value | Cx | Perf / Privacy | +|---------|---------|------------------|---------------------|----|----------------| +| **Automatic categorisation** | Sort inbox into buckets | Rules on sender/domain/headers (exists) | Embedding zero-shot / small-LLM for the ambiguous long tail + confidence | M | Rules instant; LLM only on "unknown"; local | +| **Smart filing / smart labels** | Where should this go? | Rules + user's past filing patterns | LLM/embedding *suggestions* with confidence, user-correctable | M | Suggest async; local | +| **Priority prediction** | What needs me now? | Behavioural signals: your reply-rate to sender, frequency, VIPs, keywords, direct-to-me | ML/LLM refines ranking for edge cases | M | Mostly SQL/heuristic; local | +| **Follow-up / awaiting-reply detection** | Dropped balls | Heuristic: *you* sent, contains a question, no reply in N days | LLM confirms "expects a reply" & drafts nudge | M | Heuristic instant; LLM optional; local | +| **Smart notifications** | Notification fatigue | Rules over priority + quiet hours | LLM tunes "is this actually urgent" for borderline | S | Rules-first; local | +| **Spam detection** | Junk | Rules/Bayesian + provider signals | Small model for novel spam; LLM explains | M | Fast; local | +| **Phishing detection** | Safety | URL/domain analysis, SPF/DKIM hints, lookalike detection (+ existing SSRF guard) | **LLM reasons about social-engineering cues**; runs async on flagged mail, explains risk | L | Rules sync; LLM async on suspicious; local | +| **Duplicate email detection** | Clutter / repeats | **Exact hash** for identical | **Embedding cosine** for near-duplicates | M | hash instant; vector cheap; local | +| **Inbox assistant** (daily brief) | "Catch me up" | Compose from priority/follow-up/counts (rules) | LLM writes the natural-language brief over that structured data | L | 7B once/session; local | + +## Group 3 — ⚪ Not AI (don't waste VRAM) +| Feature | Do it with | Why not AI | +|---------|-----------|------------| +| **Language detection** | fastText-lid / CLD3 library | Instant, accurate, ~0 VRAM; an LLM is pure overhead | +| **Exact duplicate detection** | content hash | Deterministic and free | +| **Unsubscribe detection** | List-Unsubscribe header parsing (exists) | Structured signal already present | + +--- + +## Selection guidance (feeds the roadmap) +- **First AI wins (highest value / lowest risk):** thread summary · follow-up detection + (heuristic + AI confirm) · reply suggestions · NL search parse. All reuse the one warm 7B. +- **Semantic tier (needs pgvector + embeddings):** related/find-similar · near-dup · + conversation insights · categorisation long-tail. +- **Ambitious tier:** ask-your-inbox (RAG) · knowledge graph · attachment/vision understanding. +- **Never gate the core on any of these** — each has a non-AI fallback or simply hides when + AI is off. + +## Privacy posture (applies to all) +Email bodies and attachments are processed **on-device via Ollama**; embeddings, summaries, +extractions, and graphs are **stored locally in Postgres**. No content leaves the machine +unless the user deliberately configures a cloud provider — and even then, per-feature +consent should gate it. This is the product's defining trust advantage (see +[02](02-competitor-analysis.md)). diff --git a/docs/discovery/08-technical-architecture.md b/docs/discovery/08-technical-architecture.md new file mode 100644 index 0000000..b535040 --- /dev/null +++ b/docs/discovery/08-technical-architecture.md @@ -0,0 +1,115 @@ +# 08 — Technical Architecture (Phase 7) + +How the redesign is built on the **existing Clean Architecture** — extending seams, not +rewriting. Everything here preserves the dependency rule `Api → Infrastructure → +Application → Domain` and the security posture from [01](01-architecture-review.md). + +> **Update — multi-provider platform.** This architecture now assumes the +> [multi-provider design](multi-provider/README.md): the Gmail-centric `Email`/`Sender` +> model generalises to **account-scoped, provider-normalised** entities behind +> `IEmailProvider`; the app becomes **small-team multi-user** (Admin/Member) with +> **OAuth-as-login**, settings, and feature flags; and **AI gating is feature-flag-driven** +> (`ai.enabled` system flag → user pref). See that folder for the provider abstraction, +> DB schema, security model, and admin/settings design; this doc remains the search/AI/ +> caching/indexing reference they build on. + +## Overall +``` +React SPA (v2 shell) + │ REST (+ SSE for streaming AI/search) +Api ────────────────────────────────────────────── +Application: IInboxAi · ISearchService · SearchIntentParser · DTOs +Infrastructure: + ├ Search: SearchService (structured + lexical rank + hybrid) + ├ AI: IAiProvider / IEmbeddingProvider / model router / VRAM guard + ├ Jobs: EmbeddingBackfill · AiEnrichment · IndexMaintenance (IHostedService + Channel queue) + ├ Gmail/Sync/Cleanup/Analytics (existing) + └ Persistence: EF Core + Npgsql (+ pgvector) +Domain: entities/enums (+ Embedding, Entities, ThreadSummary, SavedSearch) +External (optional): Ollama (local, own container) · cloud AI (opt-in only) +``` + +## Search architecture +The 4-layer engine from [05](05-search-redesign.md): +- **Structured** (SQL WHERE) + **Lexical** (`websearch_to_tsquery` + `ts_rank_cd`, weighted + multi-field `tsvector`, `pg_trgm` fuzzy fallback) — **always on**. +- **Semantic** (pgvector, HNSW) + **AI-assisted** (intent parse, RAG) — **optional**. +- **Hybrid fusion (RRF)** merges lexical+semantic; boosts recency/sender/unread. +- **Keyset pagination** replaces offset; score-ordered top-N for ranked queries. +- `SearchIntentParser` (evolves `GmailQueryParser`): operators → rules-NL → optional LLM. + +## AI architecture +- Provider interfaces + `IInboxAi` facade + **model router** (task→model via config) + + **prompt templates** (versioned files) + **VRAM guard** (serialises heavy vision jobs + behind unloading the 7B; embeddings stay hot). See [06](06-ai-strategy.md). +- **Streaming** via SSE for summaries/replies/RAG answers (perceived speed). +- **Bounded**: per-call timeout, `CancellationToken`, Polly fallback to Null path. + +## Plugin / modularity architecture +- **Analyzer pipeline:** email enrichment is a set of `IEmailAnalyzer` plugins + (classifier, entity-extractor, summariser, phishing, dedup). Each declares required + capabilities (`Chat`/`Embeddings`/none) and is **skipped gracefully** if unavailable. + New AI features = new analyzers; no core changes. +- **Search providers** implement a common `ISearchLayer` so semantic/AI layers plug in. +- **AI providers** already pluggable (`IAiProvider`); adding a provider = one class + config. +- This is the "modular AI" requirement realised structurally. + +## Caching +- **Embeddings & summaries**: persisted in Postgres (compute once, invalidate on new msg). +- **Search suggestions / recent**: cached per-user (memory + DB); debounced live search. +- **Query results**: short-TTL cache for identical repeated queries; ETag on read endpoints. +- **Model warmth**: Ollama `keep_alive` keeps the 7B hot between calls. + +## Indexing +| Index | Column | Purpose | +|-------|--------|---------| +| GIN | `SearchVector` (weighted A/B/C/D) | Full-text | +| GIN `pg_trgm` | sender addr/name, subject | Fuzzy / substring (fixes non-sargable `.Contains`) | +| HNSW | `Email.Embedding` (pgvector) | Semantic k-NN | +| btree | `(UserId, SentAtUtc)`, `(UserId, ThreadId)` | Keyset pagination, threading | +- Built incrementally on sync; **EmbeddingBackfillWorker** batch-fills history (VRAM-aware, low priority). + +## Background jobs +- Keep the existing `IHostedService` worker pattern; add a **`Channel` in-process queue** + with a bounded concurrency worker for AI enrichment (summaries/entities/embeddings on new + mail). Idempotent, resumable, backpressured. (Upgrade path: durable queue if multi-node.) +- Jobs: `GmailSyncWorker` (exists), `DigestWorker` (exists), `EmbeddingBackfillWorker`, + `AiEnrichmentWorker`, `IndexMaintenanceWorker`. + +## Database schema improvements +- **Widen** `Email.SearchVector` (subject/sender/filename/labels, weighted). +- **Add**: `Email.Embedding vector(768)`; tables `EmailEntities`, `ThreadSummary`, + `SavedSearch`, `SearchHistory`, `AiJob` (status/retry), `SenderImportance` (materialised). +- **Fix debt**: resolve the `Email↔EmailLabel` global-query-filter warning (optional nav or + matching filters); make FTS **language-aware** (detect → per-language config) instead of hardcoded English. +- **Scale**: consider per-`UserId` list partitioning of `Emails` if single users exceed ~1M rows. + +## Scalability +- Single-user/self-host is the primary shape → vertical scaling + good indexes is enough. +- Connection pooling (Npgsql), keyset pagination, top-N-by-score, batched embeddings. +- **Ollama is the throughput bottleneck** → serialise/queue AI, cache aggressively, prefer + the small/embedding models for high-volume paths. +- Multi-tenant future: read replicas, per-user partitioning, durable job queue. + +## Deployment +- Extend the Compose stack (from the CI/CD we built): add an **optional `ollama` service** + (profile `ai`) and a **pgvector-enabled Postgres image**. AI-off deployments omit the + profile entirely. Dev→staging auto-deploy already proven; prod is tag-gated. + +## Offline support +- Data is **already local** (Postgres on the user's machine) — the product is local-first + by nature. SPA: service-worker cache for shell + last-viewed mail (read offline); queue + mutations (label/cleanup) to replay on reconnect. AI is inherently offline (Ollama local). + +## Security model +- Retain: OAuth **read-only** scope, encrypted refresh tokens (Data Protection), **IDOR + global query filters**, SSRF egress guard, non-root containers, confirmed+previewed + destructive actions. +- **New AI concerns:** + - **Prompt injection** — email content is untrusted input to the LLM. Treat model output + as **advisory only**; never let it trigger actions directly; sanitise/format; the LLM + can *suggest* but a human/rule confirms (aligns with "AI never acts"). + - **Attachment/vision** — sandbox parsing; size/type limits; never execute. + - **Local-only by default** — cloud provider is explicit opt-in with per-feature consent + and egress logging; the SSRF guard already constrains outbound calls. + - Secrets stay in `deploy/.env` / Actions secrets (never in git) — as established. diff --git a/docs/discovery/09-roadmap.md b/docs/discovery/09-roadmap.md new file mode 100644 index 0000000..787030d --- /dev/null +++ b/docs/discovery/09-roadmap.md @@ -0,0 +1,93 @@ +# 09 — Product Roadmap (Phase 8) + +Everything from discovery, split into releases and **prioritised by user value vs +engineering effort**. The guiding sequence: **ship the redesign + dramatically better +(deterministic) search first**, then layer **optional local AI** in value order, each tier +reusing infrastructure the previous one built. + +> **Update — multi-provider platform epic.** The +> [multi-provider + admin/settings design](multi-provider/README.md) is a **v1.x platform +> epic** (its own 6-phase plan) that the search/AI features build on. Sequencing note: the +> **provider abstraction, unified email model, settings, and feature-flag engine land early +> in v1.x** (they underpin multi-user + AI gating); the AI features from this roadmap then +> plug into that flag system. Both plans share the same flag-gated, ship-dark discipline. + +## Prioritisation framework +Score each item **Value (1–5) × (6 − Effort 1–5)**, then sequence so that (a) high-value/ +low-effort ships first, (b) risky/expensive AI comes only after its infrastructure exists, +and (c) **nothing in an early release depends on AI being enabled.** + +``` + Value ▲ ★ MVP first ● do next ○ later + 5 │ ★ranking ★redesign ●NL search ○ask-inbox + 4 │ ★fuzzy ★chips ★summary ●semantic ●brief ○knowledge graph + 3 │ ★perf/indexes ●people ●extract ○collaboration + 2 │ ●categorise-tail ○plugins ○mobile + 1 │ ○multi-provider + └───────────────────────────────────────────────────► + low effort ──────────────────────────► high effort +``` + +--- + +## MVP → **v1.0.0** — "The redesign + world-class deterministic search" +Rationale: the single biggest perceived-quality jump, built mostly on **deterministic** +tech (low risk). AI appears only as a few optional, reversible wins behind a toggle. +- **UX v2 shell + design system + both themes** (dark-first, green ramp) — the "world-class" feel. +- **Search core:** relevance ranking (RRF/`ts_rank`), multi-field weighted FTS, `pg_trgm` + fuzzy, **filter chips**, saved/recent/suggested, **"why matched"** (lexical), keyset pagination. +- **Perf/debt:** `pg_trgm` + FTS indexes, virtualised list, code-splitting, fix EF + query-filter warning, language-aware FTS. +- **AI foundation:** extended abstraction (`IAiProvider`+`IEmbeddingProvider`+facade+router), + Null + Ollama wired, **VRAM guard**, prompt templates. +- **First AI wins (optional):** thread summary · follow-up detection (heuristic + AI confirm) + · reply suggestions. +- **Why now:** delivers the flagship promise (fast, approachable, ranked, explainable search + + a premium UI) even with AI off. + +## v1.1 — "Assisted search & productivity" +Rationale: build on MVP's embedding groundwork; assist without heavy compute. +- **Natural-language search** parse (rules-first + optional LLM), shown as editable chips. +- **People search** · **attachment (filename) search** · **search-driven bulk actions**. +- **Inbox assistant / daily brief** · **task/calendar/reminder extraction**. +- **Categorisation long-tail** (embedding zero-shot + confidence) · **priority prediction v1**. +- **Why now:** the "assisted, not syntactic" search vision, plus the highest-value AI + productivity features — all still light on VRAM. + +## v1.2 — "Semantic tier" +Rationale: introduces `pgvector` + embedding backfill; unlocks meaning-based features. +- **Semantic search** (pgvector HNSW, hybrid RRF) · **related / find-similar**. +- **Near-duplicate detection** · **thread summaries surfaced in results** · **conversation insights**. +- **Relationship mapping (basics)**. +- **Why now:** semantic recall is a headline differentiator but needs the embedding + infrastructure and backfill worker to be mature and VRAM-safe. + +## v2.0 — "Ambitious AI" +Rationale: flagship, compute-heavy features that need a mature semantic + vision stack. +- **Conversational "ask your inbox"** (RAG + **citations**). +- **Entity search & facets** (amounts, orgs, dates) · **attachment OCR/content understanding** (vision, on-demand). +- **Phishing reasoning** (LLM over flagged mail, async, explained) · **knowledge graph**. +- **Why now:** highest wow-factor and the strongest "local private AI" story, but only + worthwhile once retrieval, extraction, and vision infra are proven. + +## v3.0 — "Platform" +Rationale: expand beyond the individual power-user once the core is best-in-class. +- **Collaboration / shared inbox** (assign, comment) · **automation / rules engine** + (a `docs/specs/feature-rules-engine.md` already exists — fold it in). +- **Multi-account & additional providers** (IMAP/Outlook) · **plugin ecosystem** (analyzer + API) · **mobile app** · **multi-window / desktop shell** (Tauri) · **multi-provider AI**. +- **Why now:** platform bets that only pay off on top of a beloved core product. + +--- + +## Sequencing principles (explained) +1. **Deterministic value before AI.** MVP's biggest wins (ranking, chips, redesign) need no + AI — they de-risk the release and prove the product before compute-heavy features. +2. **Infrastructure amortised.** Embeddings introduced once (v1.1 foundations → v1.2 usage) + power search, dedup, related, categorisation, and RAG — spread the cost. +3. **Value-first within a release.** Inside each version, highest value/effort ships first so + partial delivery is still shippable. +4. **AI is always additive.** Every release is complete and excellent with AI disabled — + protecting the "works without AI" mandate and the mainstream audience. +5. **Platform last.** Collaboration/plugins/mobile are large and only worthwhile once the + individual experience is genuinely best-in-class. diff --git a/docs/discovery/10-git-plan.md b/docs/discovery/10-git-plan.md new file mode 100644 index 0000000..2de279a --- /dev/null +++ b/docs/discovery/10-git-plan.md @@ -0,0 +1,60 @@ +# 10 — Git Implementation Plan (Phase 10) + +Turns the [roadmap](09-roadmap.md) into executable Git work, on the workflow already in +place ([../WORKFLOW.md](../WORKFLOW.md)): trunked `develop`/`main`, Conventional Commits, +PR-gated CI/Security, auto-deploy to staging, tag-gated production. + +## Structure: Epics → Features → Tasks +- **Epic** = a roadmap theme → a milestone + a long-lived integration effort. +- **Feature** = one shippable slice → one `feature/*` branch → one PR into `develop`. +- **Task** = one atomic commit (conventional) within a feature branch. + +## Epics (mapped to releases) +| Milestone | Epic | Example feature branches | +|-----------|------|--------------------------| +| **v1.0.0** | `epic/design-system` | `feature/design-tokens` · `feature/app-shell` · `feature/themes-dark-light` | +| **v1.0.0** | `epic/search-core` | `feature/search-ranking` · `feature/fts-multifield` · `feature/search-fuzzy-trgm` · `feature/search-chips` · `feature/saved-recent-searches` · `feature/search-why-matched` · `feature/keyset-pagination` | +| **v1.0.0** | `epic/perf-and-debt` | `feature/trgm-indexes` · `feature/virtualised-list` · `feature/code-splitting` · `fix/ef-queryfilter-warning` · `feature/language-aware-fts` | +| **v1.0.0** | `epic/ai-foundation` | `feature/ai-abstraction` · `feature/embedding-provider` · `feature/ai-vram-guard` · `feature/ai-thread-summary` · `feature/ai-followup-detect` · `feature/ai-reply-suggest` | +| **v1.1.0** | `epic/assisted-search` | `feature/nl-search-parse` · `feature/people-search` · `feature/attachment-filename-search` · `feature/search-bulk-actions` | +| **v1.1.0** | `epic/ai-productivity` | `feature/inbox-brief` · `feature/task-calendar-extract` · `feature/categorise-tail` · `feature/priority-v1` | +| **v1.2.0** | `epic/semantic` | `feature/pgvector-schema` · `feature/embedding-backfill` · `feature/semantic-search` · `feature/find-similar` · `feature/near-dup` · `feature/thread-insights` | +| **v2.0.0** | `epic/ambitious-ai` | `feature/ask-your-inbox-rag` · `feature/entity-facets` · `feature/attachment-ocr` · `feature/phishing-reasoning` · `feature/knowledge-graph` | +| **v3.0.0** | `epic/platform` | `feature/rules-engine` · `feature/collaboration` · `feature/multi-account` · `feature/plugin-api` · `feature/mobile` | + +## Branch naming +- `feature/` · `fix/` · `hotfix/` (off `main`) · optional + `release/x.y.0` for stabilisation. Epics tracked via milestone/label, not a long branch + (avoid merge hell); features integrate continuously into `develop`. + +## Commit strategy +- **Conventional Commits** (already used): `type(scope): summary`. Types drive SemVer: + `feat`→minor, `fix`→patch, `feat!`/`BREAKING CHANGE`→major. +- One logical change per commit; **docs updated in the same commit/PR** as the behaviour + they describe (enforced by review checklist — see below). + +## PR strategy +- `feature/* → develop`, **squash-merge**; `develop → main`, **merge commit** (release boundary). +- **Required checks** (already enforced): `CI/backend`, `CI/frontend`, `Security/secrets`, + `Security/dependencies`. Merge to `develop` auto-deploys **staging**. +- **PR checklist:** tests added (unit + integration for API changes) · docs updated · + AI features have a **Null/AI-off path** · no secret committed · perf-sensitive paths have + an index/plan note. + +## Release milestones (tags on `main`) +| Tag | Contents | Gate | +|-----|----------|------| +| `v1.0.0` | Redesign + deterministic search + AI foundation & first wins | Redesign parity + search benchmarks green | +| `v1.1.0` | Assisted search + AI productivity | NL parse accuracy + extraction quality bar | +| `v1.2.0` | Semantic tier | Backfill complete + hybrid-rank quality bar | +| `v2.0.0` | Ambitious AI (RAG, vision, graph) | RAG citation accuracy + VRAM stability | +| `v3.0.0` | Platform (collab, rules, plugins, mobile) | — | +- Cutting a tag = the production-promotion action (see [../WORKFLOW.md](../WORKFLOW.md) §5–6); + `deploy-prod.yml` fires on `v*` once the Linux server + its runner exist. +- Interim work ships as `0.x`/pre-release increments on `develop`; `v1.0.0` is the first + "world-class" cut. + +## Docs-alongside-code (hard rule) +Every feature PR updates the relevant doc: `docs/discovery/*` decisions graduate into +`docs/` living docs (architecture, search, AI, API reference) as they're implemented, and +`CHANGELOG.md` gains an entry. Discovery docs are the *source*; implementation keeps them true. diff --git a/docs/discovery/11-risks-and-future.md b/docs/discovery/11-risks-and-future.md new file mode 100644 index 0000000..317265c --- /dev/null +++ b/docs/discovery/11-risks-and-future.md @@ -0,0 +1,37 @@ +# 11 — Risk Assessment & Future Opportunities + +## Risk assessment +Likelihood (L) / Impact (I): H/M/L. + +| # | Risk | L | I | Mitigation | +|---|------|---|---|------------| +| R1 | **Full UI redesign destabilises a working app** | M | H | Strangler migration behind `ui.v2` flag, screen-by-screen; API untouched; ship via the proven `develop→staging` pipeline; keep old screens until parity verified | +| R2 | **VRAM (10 GB) can't hold desired models concurrently** | M | M | 7–8B sweet-spot (not 14B); embeddings hot + LLM warm + vision on-demand; VRAM guard serialises heavy jobs; small models for high-volume paths | +| R3 | **Local AI quality/latency disappoints** | M | M | Traditional-first (AI only where it clearly wins); AI advisory + optional; stream responses; cache; model routing lets us swap models without code change | +| R4 | **Prompt injection via email content** | M | H | Treat all model output as advisory; **AI never triggers actions**; human/rule confirms; sanitise; SSRF/egress guards; local-only by default | +| R5 | **Gmail API quota / sync scale at 100k+ mailboxes** | M | M | Batching + Polly backoff (exists); incremental sync; background enrichment queue with backpressure; keyset pagination | +| R6 | **Semantic infra (pgvector/embeddings) ops complexity** | M | M | Introduce once (v1.2), backfill worker VRAM-aware + resumable; HNSW tuning; feature hides if unavailable | +| R7 | **Search relevance regressions vs today** | L | M | Lexical hits never lose to fuzzy noise (weighting); benchmark suite as a release gate; keep date-sort as a user option | +| R8 | **Scope creep — trying to beat everyone at once** | H | M | Roadmap value/effort discipline; MVP is deterministic + small AI; platform features deferred to v3 | +| R9 | **Solo-dev bandwidth / single-machine staging** | H | M | Small shippable features; CI/CD automation already reduces toil; staging owned by automation (don't hand-run it — see memory note) | +| R10 | **Mainstream-vs-power tension dilutes the UX** | M | M | Progressive disclosure: simple default, power revealed on demand; pointer-first with keyboard as accelerator | +| R11 | **Privacy promise broken by a cloud provider option** | L | H | Local default; cloud is explicit per-feature opt-in with egress logging + consent; never silent | +| R12 | **pgvector image / Ollama container adds deploy friction** | L | L | Optional Compose profiles (`ai`); AI-off deployments omit them entirely | + +## Future opportunities (beyond v3.0) +- **Additional mail backends** — IMAP/JMAP, Outlook/Graph — become a true multi-provider client. +- **On-device personalisation** — light fine-tuning / user-preference adapters for priority & tone. +- **Calendar & tasks integration** — close the loop from extraction to action. +- **Voice** — dictate replies, "ask your inbox" by voice (local Whisper). +- **Plugin marketplace** — third-party analyzers/widgets on the analyzer + command-palette APIs. +- **Team knowledge base** — shared, permissioned knowledge graph across a team inbox. +- **Native desktop shell** (Tauri) for OS integration, global hotkey, tray, true multi-window. +- **Smart compose surfaces** — templates that learn, snippet library, per-recipient tone memory. +- **Local model upgrades** — swap in newer/quantised models as they ship (router makes it a config change). + +## Recommendation +The foundation is strong and the wedge is real. **Proceed with v1.0.0 (redesign + +deterministic search + AI foundation)** — it's high-value, low-risk, and independent of AI +being enabled — then layer local AI in value order. The biggest watch-items are **R1 +(migration discipline)** and **R8 (scope)**; both are controlled by the strangler approach +and the value/effort-sequenced roadmap. diff --git a/docs/discovery/EXECUTIVE-SUMMARY.md b/docs/discovery/EXECUTIVE-SUMMARY.md new file mode 100644 index 0000000..d955ce8 --- /dev/null +++ b/docs/discovery/EXECUTIVE-SUMMARY.md @@ -0,0 +1,70 @@ +# Executive Summary — InboxIntel Discovery Blueprint + +**Ambition:** become one of the best **email search and management** experiences available +— fast, scalable, secure, intuitive — with **AI used only where it clearly beats +traditional approaches**, running **locally and privately**, and **never as a hard +dependency**. + +## The opportunity (the wedge) +The market has split in two, and both halves leave a gap: +- **AI, but cloud** (Gmail/Gemini, Outlook/Copilot, Shortwave, Spark) — useful, but your + mail is processed off-device. +- **Private, but little/no AI** (Proton, Thunderbird, Fastmail). + +**No mainstream product owns "genuinely useful AI that runs on your own machine."** +InboxIntel can — and it already has the seams for it. Combined with three more openings — +**approachable power-search as the home screen**, **explainability** ("why this matched / +was categorised"), and **fast *and* friendly** (Superhuman is fast-but-intimidating; casual +apps are friendly-but-slow) — this is a defensible, differentiated position. + +> **Positioning:** *Make search the fastest way to think about your inbox, with AI that runs +> on your own machine and always explains itself. Powerful for pros, simple for anyone.* + +## Current state — verdict: extend, don't rewrite +A clean, secure, well-tested .NET 8 / React foundation with **real Postgres full-text +search** and **an AI provider abstraction already in place** (`NullAiProvider` / +`OllamaProvider` / `OpenAiProvider`). The gaps are exactly where the product wants to win: +relevance-ranked multi-mode search, a richer AI contract (embeddings/extraction), +conversation intelligence, and a modern approachable UX. **None require a rewrite.** + +## Design direction (from the interview) +Notion/Arc-**professional**, "anyone can pick it up": **pointer-first & discoverable** +(palette/shortcuts as accelerators), **balanced density**, **dark-first** (genuine light +too), **subtle motion**, **green `#3ba31f`** accent, clean rounded line icons, **fully +responsive** desktop→mobile. The pivotal decision — *discoverable-first over keyboard-first* +— reframes search from "a syntax you learn" into "an inviting, assisted experience." + +## Search & AI in a nutshell +- **Search:** a 4-layer engine — Structured → **Lexical+Ranking** (always on) → Semantic + (pgvector) → AI-assisted (NL / ask-your-inbox / explanations) — with **hybrid RRF ranking** + replacing today's date-only sort. Degrades gracefully with AI off. +- **AI:** extend the abstraction to embeddings + structured output behind a task facade with + **config-driven model routing**; on the **RTX 3080 (10 GB)**, **Qwen2.5-7B** (warm) + + **nomic-embed-text** (hot) do most jobs, vision on-demand. Traditional-first everywhere; + every AI feature has a non-AI fallback. + +## Roadmap shape +**v1.0** redesign + world-class *deterministic* search + AI foundation → **v1.1** assisted +search & productivity → **v1.2** semantic tier → **v2.0** ambitious AI (RAG, vision, graph) +→ **v3.0** platform (collaboration, rules, plugins, mobile). Sequenced so **deterministic +value ships before AI**, infrastructure is amortised, and **every release is complete with +AI disabled.** + +## Key risks +Migration of a full redesign (**mitigated by strangler + `ui.v2` flag**), VRAM limits +(**7–8B sweet spot + load policy**), prompt injection (**AI advisory-only, never acts**), and +scope creep (**value/effort-sequenced roadmap**). See [11](11-risks-and-future.md). + +## Recommendation +**Proceed to v1.0.0** — the redesign, ranked/assisted search, and AI foundation. It's the +biggest quality jump, it's low-risk and deterministic, and it stands entirely on its own +without AI. Then layer local, private, explainable AI in value order. + +--- +### Read the full blueprint +[01 Architecture](01-architecture-review.md) · [02 Competitors](02-competitor-analysis.md) · +[03 Users](03-user-research.md) · [04 UX + Design System](04-ux-redesign-and-design-system.md) · +[05 Search](05-search-redesign.md) · [06 AI Strategy](06-ai-strategy.md) · +[07 AI Features](07-ai-feature-catalogue.md) · [08 Architecture](08-technical-architecture.md) · +[09 Roadmap](09-roadmap.md) · [10 Git Plan](10-git-plan.md) · +[11 Risks & Future](11-risks-and-future.md) · [Design Brief](00-design-brief.md) diff --git a/docs/discovery/README.md b/docs/discovery/README.md new file mode 100644 index 0000000..8a8a9b0 --- /dev/null +++ b/docs/discovery/README.md @@ -0,0 +1,48 @@ +# InboxIntel — Discovery & Blueprint + +This folder is the **product discovery output**: a world-class specification that will +become the blueprint for the next generation of InboxIntel. It is a **planning +artifact** — no application code is changed during discovery. + +> Vision: become one of the best **email search and management** experiences available — +> extremely fast, scalable, secure, and intuitive. AI is introduced **only where it +> clearly beats traditional approaches**, and never as a hard dependency. + +## Non-negotiable constraints (carried into every phase) +- **AI is optional & modular.** The app must work fully with AI disabled. All AI sits + behind an `IAIProvider` abstraction (already seeded: `NullAiProvider` / `OllamaProvider` + / `OpenAiProvider`) so providers swap without touching application logic. +- **Local-first AI.** Primary target: Ollama on an NVIDIA RTX 3080 (10 GB). +- **Privacy.** Read-only Gmail scope; AI advisory-only; no destructive AI actions. +- **Speed & security are features**, not afterthoughts. + +## Document index +> **Start here:** [Executive Summary](EXECUTIVE-SUMMARY.md) + +| # | Document | Phase | Status | +|---|----------|-------|--------| +| — | [Executive Summary](EXECUTIVE-SUMMARY.md) | all | ✅ draft | +| 00 | [Design Brief](00-design-brief.md) | 4A (interview) | ✅ locked | +| 01 | [Architecture Review](01-architecture-review.md) | 1 | ✅ draft | +| 02 | [Competitor Analysis](02-competitor-analysis.md) | 2 | ✅ draft | +| 03 | [User Research & Journey Maps](03-user-research.md) | 3 | ✅ draft | +| 04 | [UX/UI Redesign + Design System](04-ux-redesign-and-design-system.md) | 4A | ✅ draft | +| 05 | [Search Redesign](05-search-redesign.md) | 4B | ✅ draft | +| 06 | [AI Strategy & Model Recommendations](06-ai-strategy.md) | 5 | ✅ draft | +| 07 | [AI Feature Catalogue](07-ai-feature-catalogue.md) | 6 | ✅ draft | +| 08 | [Technical Architecture](08-technical-architecture.md) | 7 | ✅ draft | +| 09 | [Product Roadmap (MVP→v3)](09-roadmap.md) | 8 | ✅ draft | +| 10 | [Git Implementation Plan](10-git-plan.md) | 10 | ✅ draft | +| 11 | [Risk Assessment & Future Opportunities](11-risks-and-future.md) | cross-cutting | ✅ draft | + +## Extension designs +- [**Multi-Provider Email Platform + Admin/Settings**](multi-provider/README.md) — evolves + InboxIntel into a small-team, multi-provider (Gmail/Outlook/IMAP) platform with OAuth-as- + login, settings, feature flags, and an admin panel. Reshapes the core assumptions above + (see the "Update" callouts in [08](08-technical-architecture.md) and [09](09-roadmap.md)). + +## Conventions +- Every recommendation is a durable markdown doc suitable for long-term maintenance. +- Each feature is specified with: **problem solved · why AI (or why not) · complexity · + estimated effort · user value · perf & privacy considerations**. +- Nothing here is implemented until explicitly approved and scheduled via the Git plan. diff --git a/docs/discovery/multi-provider/01-provider-abstraction.md b/docs/discovery/multi-provider/01-provider-abstraction.md new file mode 100644 index 0000000..506c554 --- /dev/null +++ b/docs/discovery/multi-provider/01-provider-abstraction.md @@ -0,0 +1,80 @@ +# 01 — Provider Abstraction (Part 1) + +A unified layer so Gmail, Outlook/Graph, and future IMAP look identical to the rest of the +app. **No provider-specific logic in Domain**; specifics live in Infrastructure adapters. + +## Layering +``` +Domain : Account, EmailMessage, EmailThread, Label (provider-agnostic) +Application : IEmailProvider (contract) · ISyncOrchestrator · DTOs +Infrastructure : GmailProvider · OutlookProvider · ImapProvider (adapters) + ProviderFactory (ProviderType → adapter) · ITokenStore +``` + +## The contract +```csharp +public enum ProviderType { Google, Microsoft, Imap } + +public interface IEmailProvider { + ProviderType Type { get; } + ProviderCapabilities Capabilities { get; } // read, modifyFlags, folders, delta, send? + + // Auth (details in 02-auth-and-signin.md) + Task ExchangeCodeAsync(string code, CancellationToken ct); + Task RefreshAsync(TokenSet current, CancellationToken ct); + Task GetIdentityAsync(TokenSet tokens, CancellationToken ct); // sub + email + + // Sync (pull-based, incremental) + Task SyncAsync(SyncCursor cursor, TokenSet tokens, CancellationToken ct); + Task FetchMessageAsync(string providerMessageId, TokenSet tokens, CancellationToken ct); + + // Mutations (only if Capabilities allow; mirrors current gmail.modify scope) + Task ApplyFlagAsync(string providerMessageId, MailFlagChange change, TokenSet tokens, CancellationToken ct); +} +``` +- `SyncPage` = `{ IReadOnlyList upserts, IReadOnlyList deletes, SyncCursor next, bool hasMore }`. +- `RawMessage` is the **provider-shaped** payload; a **normaliser** maps it to the domain + `EmailMessage`. The rest of the app never sees `RawMessage`. +- Capabilities let the UI/engine **degrade gracefully** (e.g., an IMAP server without + CONDSTORE falls back to full-scan sync; no `send` today for any provider). + +## Provider implementations +| Provider | API | Incremental cursor | Threading | Folders/Labels | Notes | +|----------|-----|--------------------|-----------|----------------|-------| +| **Gmail** | Gmail REST | `historyId` (History API) | `threadId` | labels | Reuses existing client; read + modify (no send), as today | +| **Outlook/365** | Microsoft Graph | **delta query** `@odata.deltaLink` | `conversationId` | mailFolders | OAuth via Microsoft identity platform | +| **IMAP** | IMAP4rev1 | `UIDVALIDITY`+`UIDNEXT`, `HIGHESTMODSEQ` (CONDSTORE/QRESYNC) | heuristic (References/In-Reply-To) | folders | Fallback = periodic UID scan if no CONDSTORE; MailKit already a dependency | + +## Normalisation (the unified model) +Each adapter maps provider fields → domain via a `IMessageNormaliser`: +| Domain field | Gmail | Graph | IMAP | +|--------------|-------|-------|------| +| `ProviderMessageId` | message id | message id | `UIDVALIDITY:UID` | +| `ProviderThreadId` | threadId | conversationId | derived (References) | +| flags (unread/star/important/trashed/inbox) | labelIds | isRead/flag/folder | `\Seen \Flagged`, folder | +| labels/folders | labels | mailFolders | folders | +| sent/received, from, subject, snippet, body, attachments, size | headers/parts | message resource | RFC822 parse (MailKit) | +- **Threads are per-account** (each provider defines its own). *Cross-provider* thread + linking is a later semantic/AI feature (see blueprint [07](../07-ai-feature-catalogue.md)), + not part of core normalisation. + +## Sync engine +- **`ISyncOrchestrator`** replaces the Gmail-specific worker: for each active `Account`, it + loads the `SyncCursor`, calls `provider.SyncAsync`, **upserts** normalised messages + (idempotent on `(AccountId, ProviderMessageId)`), applies deletes, and **persists the next + cursor** atomically. +- Runs as the existing hosted-worker pattern (`AccountSyncWorker`), one logical job per + account, bounded concurrency, Polly backoff, resumable. +- **Token refresh:** `SyncAsync`/mutations get a valid `TokenSet` from `ITokenStore`, which + refreshes on expiry/401 and **re-encrypts** at rest; a failed refresh flips the account to + `reauth_needed` (surfaced in UI, see [07](07-ux-flows.md)) — never crashes sync. +- **New mail** triggers AI enrichment + embedding jobs (blueprint [08](../08-technical-architecture.md)). + +## Search across providers +Because all providers normalise into **one `email_messages` store scoped by `UserId`**, +search (structured + FTS + semantic) **already spans every account a user has connected** — +no per-provider search code. An optional `AccountId` facet lets users scope to one mailbox. + +## Extensibility +Adding a provider = one `IEmailProvider` adapter + one normaliser + register in +`ProviderFactory` + a feature flag to enable it. **Zero changes to Domain, search, or AI.** diff --git a/docs/discovery/multi-provider/02-auth-and-signin.md b/docs/discovery/multi-provider/02-auth-and-signin.md new file mode 100644 index 0000000..060ed07 --- /dev/null +++ b/docs/discovery/multi-provider/02-auth-and-signin.md @@ -0,0 +1,70 @@ +# 02 — Auth & Sign-in (Part 2) + +**OAuth is the login.** No passwords. A user's identity is the set of provider accounts +linked to them; any one can authenticate the session. Identity key is **`(provider, sub)`** +— never email (emails change; `sub` is stable, and the same email can exist on Google *and* +Microsoft as distinct accounts). + +## Provider-selection sign-in (first-time) +``` +[ Choose how to sign in ] + ▸ Continue with Google ▸ Continue with Microsoft ( ▸ IMAP — future ) +``` +1. User picks a provider → redirect to provider OAuth (**PKCE**, `state`, `nonce`). +2. Callback → exchange code → `GetIdentityAsync` returns `(provider, sub, email, name)`. +3. **Resolve:** look up `accounts (provider, sub)`. + - **No match →** first-time. Create `user` (**first user ever = Admin**, otherwise `Member` + if `system_settings.registration_open`, else reject) + `account` (`is_login_identity=true`) + + encrypted `provider_tokens`. Start session. + - **Match →** existing user. Refresh tokens, start session. +4. Kick off the account's initial sync. + +## Adding another account later (linking) — the security-critical flow +The user is **already authenticated**. "Add account" → provider OAuth in **link mode**: +- Callback identity `(provider, sub)`: + - **Unlinked →** attach a new `account` (mailbox) to the **current** user. ✅ + - **Already linked to *this* user →** no-op / "already connected." + - **Already linked to *another* user →** **blocked** by the unique `(provider, provider_account_id)` + constraint + explicit check → error "This mailbox is connected to a different InboxIntel + user." **This is the anti-hijack guarantee** — you can only link an identity you can + authenticate *and* that no one else owns. +- A single user can hold **N accounts** across Google/Microsoft/IMAP; each can also serve as a + login identity (any of them signs you into the same user). + +## Switching +- **Switch mailbox (same user):** an **account switcher** changes the active mailbox context + (or "All accounts" unified view). No re-auth — it's all one user. Search can scope to one + account or span all. +- **Switch user (different person):** full sign-out → sign-in. Optional "fast switch" could + hold multiple sessions, but for a small team, explicit re-auth is simplest and safest. + +## Sessions +- **Opaque server-side session** (`sessions` table) referenced by an **HttpOnly · Secure · + SameSite=Lax** cookie. **Provider tokens are never exposed to the browser.** +- Rotate session id on login (anti-fixation); **idle (e.g., 7d) + absolute (e.g., 30d)** + expiry; revoke on logout; **"sign out everywhere"** and **admin revoke** delete session rows. +- CSRF: SameSite + anti-CSRF token on state-changing requests. + +## Token lifecycle +- Stored **encrypted at rest** (Data Protection); decrypted only in-memory for API calls. +- **Refresh** on expiry/401 via `ITokenStore` → re-encrypt + persist; failure flips account to + **`ReauthNeeded`** (banner + "Reconnect" CTA, see [07](07-ux-flows.md)) — sync/AI for that + account pause, the rest of the app is unaffected. + +## Provider OAuth specifics +| | Google | Microsoft (Graph) | +|--|--------|-------------------| +| Endpoint | accounts.google.com | login.microsoftonline.com (`common`) | +| Scopes | `openid email profile gmail.readonly gmail.modify` | `openid email profile offline_access Mail.Read Mail.ReadWrite` | +| Identity | `sub` (+ verified email) | `oid`/`sub` (+ email) | +| Refresh | refresh_token (offline) | refresh_token (`offline_access`) | +| Redirect | `/signin/google` | `/signin/microsoft` | +- **Least privilege:** request read/modify only (no send today — matches current posture). + Extra scopes are added per-feature with consent, never up-front. + +## Edge cases +- **Same email, two providers** → two distinct accounts (identity is `sub`), unless the user + links both to one InboxIntel user. +- **Provider disabled by admin flag** (`provider.microsoft=false`) → hide it on the picker; + existing accounts of that provider pause sync and show a notice. +- **Reused browser / stale cookie** → session validated server-side each request; revoked/expired → re-auth. diff --git a/docs/discovery/multi-provider/03-database-design.md b/docs/discovery/multi-provider/03-database-design.md new file mode 100644 index 0000000..82ed74f --- /dev/null +++ b/docs/discovery/multi-provider/03-database-design.md @@ -0,0 +1,79 @@ +# 03 — Database Design (Part 6) + +Schema for a **small-team, self-hosted, one-org** platform: multi-provider accounts per +user, a normalised email store scaling to **millions of messages**, settings, feature +flags, and audit. PostgreSQL + EF Core (+ pgvector for the blueprint's semantic tier). + +## Entity map +``` +users ─┬─< accounts ─┬─< provider_tokens (1:1, encrypted) + │ ├─< email_threads ─< email_messages ─┬─< attachments + │ │ └─< message_labels >─ labels + │ └─ sync_state (cursor) + ├─< user_settings (1:1) + └─< audit_logs (actor) +system_settings (singleton) feature_flags +``` + +## Tables + +### Identity & access +- **`users`** — the app identity (from OAuth). + `id (uuid pk) · primary_email (citext, unique) · display_name · avatar_url · role (enum: Admin|Member) · status (enum: Active|Suspended) · created_at · last_login_at` + *First-ever user is bootstrapped as **Admin** (see [05](05-admin-system.md)).* +- **`accounts`** — a connected mailbox **and** a login identity (OAuth-is-login). + `id (uuid pk) · user_id (fk) · provider (enum: Google|Microsoft|Imap) · provider_account_id (text, the OAuth 'sub' — immutable) · email (citext) · display_name · is_login_identity (bool) · status (enum: Active|ReauthNeeded|Disabled) · scopes (text[]) · added_at · last_sync_at` + **Unique:** `(provider, provider_account_id)` → resolves an OAuth login to exactly one account→user; prevents the same mailbox linking twice. +- **`provider_tokens`** — 1:1 with `accounts`, **encrypted at rest** (Data Protection API). + `account_id (pk/fk) · access_token_enc (bytea) · refresh_token_enc (bytea) · expires_at_utc · token_type · rotated_at` + *Never logged; see [06](06-security-model.md).* +- **`sessions`** — server-side app sessions (opaque cookie). + `id · user_id · created_at · expires_at · ip · user_agent · revoked_at` (supports "sign out everywhere" + admin revoke). + +### Email (normalised, provider-agnostic) +- **`email_threads`** — per account. + `id (uuid pk) · account_id (fk) · user_id (denorm) · provider_thread_id · subject · participants (jsonb) · message_count · last_message_at` + **Unique:** `(account_id, provider_thread_id)`. +- **`email_messages`** — the big table. + `id (uuid pk) · account_id (fk) · user_id (denorm) · thread_id (fk) · provider_message_id · sender_id (fk) · subject · snippet · body_text · sent_at_utc · received_at_utc · size_bytes · flags (unread/starred/important/in_inbox/trashed as bits or bools) · has_attachments · category (enum) · has_list_unsubscribe · supports_one_click_unsub · search_vector (tsvector, generated, weighted) · embedding (vector(768), nullable) · created_at` + **Unique:** `(account_id, provider_message_id)` (idempotent upsert). +- **`labels`** (`id · account_id · provider_label_id · name · type`) + **`message_labels`** (`message_id · label_id`, pk both). +- **`attachments`** (`id · message_id · filename · mime · size · provider_attachment_id · content_text nullable` for future OCR/search). +- **`senders`** / **`domains`** (existing) — kept, scoped per user (or global with per-user stats materialised in `sender_importance`). + +### Settings, flags, audit +- **`user_settings`** — 1:1 with `users`. + `user_id (pk) · theme (enum: system|light|dark) · inbox_layout (jsonb) · notifications (jsonb) · ai_prefs (jsonb) · provider_prefs (jsonb) · updated_at` +- **`system_settings`** — singleton (org-wide, admin-managed). + `id (const) · maintenance_mode (bool) · default_theme · registration_open (bool) · updated_by · updated_at` (+ arbitrary `values jsonb` for growth). +- **`feature_flags`** — the flag system (drives AI gating). + `key (pk text) · enabled (bool) · scope (enum: SystemOnly|UserOverridable) · description · rollout (jsonb, e.g. per-role) · updated_by · updated_at` + Seeded flags: `ai.enabled`, `ai.semantic_search`, `ai.ask_inbox`, `provider.google`, `provider.microsoft`, `provider.imap`, `maintenance.readonly`. +- **`audit_logs`** — admin + security events. + `id (bigserial) · actor_user_id (fk, nullable for system) · action (text) · target_type · target_id · metadata (jsonb) · ip · created_at` + Append-only; indexed on `(created_at)`, `(actor_user_id)`, `(action)`. + +## Indexing & performance +| Index | Column(s) | For | +|-------|-----------|-----| +| GIN | `email_messages.search_vector` | FTS | +| GIN `pg_trgm` | sender/subject | fuzzy (fixes non-sargable `.Contains`) | +| HNSW | `email_messages.embedding` | semantic k-NN | +| btree | `(user_id, sent_at_utc desc)`, `(account_id, sent_at_utc desc)`, `(thread_id)` | keyset pagination, scoping, threading | +| unique | `(account_id, provider_message_id)`, `(provider, provider_account_id)` | idempotency, login resolution | + +## Scalability (millions of emails, multi-account, incremental) +- **Multi-account** = first-class via `accounts`; every email carries `account_id` + denormalised `user_id` (so cross-account per-user search is a single indexed scan). +- **Millions of rows:** keyset (cursor) pagination, top-N-by-score ranking, `pg_trgm`/GIN/HNSW indexes. If a single user exceeds ~1–2M messages, **list-partition `email_messages` by `account_id`** (or hash by `user_id`). +- **Incremental sync:** per-account `sync_state` cursor (`historyId` / `deltaLink` / `uidvalidity+modseq`) → only deltas fetched; idempotent upserts on the unique key. +- **Idempotency & resumability:** all sync writes keyed on `(account_id, provider_message_id)`; cursor advanced atomically with the batch. + +## RBAC in the schema +Small-team model = a `role` column on `users` (`Admin|Member`) — no separate roles/permissions tables yet. **Extensible** to `roles`/`permissions`/`org_id` if this ever grows to multi-tenant, without reshaping the email tables. + +## Migration from today (see full [Migration Guide](12-migration-guide.md)) +1. Create `users` from existing OAuth identity; set first user = **Admin**. +2. Create one `Google` **`account`** per existing user; move current Gmail tokens → `provider_tokens`. +3. Backfill `email_messages.account_id`/`user_id`, rename/extend from the current `Email` table; widen `search_vector`; add nullable `embedding`. +4. Add `user_settings`, `system_settings`, `feature_flags` (seed `ai.enabled` from the current `Ai:Mode`), `audit_logs`. +5. All additive + backfill; no destructive step — safe to run behind a maintenance window. diff --git a/docs/discovery/multi-provider/04-settings-and-flags.md b/docs/discovery/multi-provider/04-settings-and-flags.md new file mode 100644 index 0000000..531b0b6 --- /dev/null +++ b/docs/discovery/multi-provider/04-settings-and-flags.md @@ -0,0 +1,68 @@ +# 04 — Settings & Feature Flags (Part 3) + +Two layers of configuration — **per-user preferences** and **org-wide system settings / +feature flags** — with a clear precedence. Critically: **AI is governed by a feature flag +(admin, global), not merely a user preference.** + +## User settings (per user) +Stored in `user_settings`; editable by the user. +| Setting | Values | +|---------|--------| +| `theme` | system · light · dark (dark-first default) | +| `inbox_layout` | density, pane layout, default view/lane, per-account or unified | +| `notifications` | channels, quiet hours, priority-only | +| `ai_prefs` | per-feature opt-in (summaries, replies, semantic, ask-inbox…) — **only effective if the flag allows** | +| `provider_prefs` | default account, sync frequency, signature per account | + +## System settings (admin, org-wide) +Stored in `system_settings` (singleton); Admin-only ([05](05-admin-system.md)). +- `maintenance_mode` (off · read-only · locked-except-admin) · `default_theme` · + `registration_open` · org display name · retention defaults. + +## Feature flag system +`feature_flags` rows: `key · enabled · scope · rollout · description · updated_by/at`. +- **`scope = SystemOnly`** — a hard org-wide switch; users cannot override (e.g., `provider.microsoft`, `maintenance.readonly`). +- **`scope = UserOverridable`** — a default that a user preference can turn *off* (never *on* beyond what the flag permits) — e.g., `ai.summaries`. +- **`rollout`** (jsonb) — optional per-role/percentage gating (e.g., enable a beta for Admins first). + +### Evaluation service +```csharp +public interface IFeatureFlags { + bool IsEnabled(string key, UserContext user); // system flag ∧ scope ∧ role rollout +} +public interface IAiGate { // the AI-specific resolver + bool IsAiEnabled(UserContext user); // ai.enabled (system) ∧ user.ai_prefs.master + bool IsAiFeatureEnabled(string feature, UserContext user); // ∧ ai. ∧ user opt-in +} +``` +- Flags are **cached** with change notification (hot-reload on admin edit); every read is cheap. +- All flag reads are **fail-closed**: unknown/errored flag ⇒ treated as **off**. + +## AI gating precedence (the key requirement) +Effective AI availability is an **AND** down a chain — the system flag is the master gate: +``` +AI feature X is available for user U ⇔ + feature_flags["ai.enabled"].enabled (admin master switch — SYSTEM) + ∧ feature_flags["ai." + X].enabled (per-feature flag — SYSTEM) + ∧ providerCapability(X) (Ollama/provider actually available) + ∧ user.ai_prefs.master_opt_in (user hasn't disabled AI for themselves) + ∧ user.ai_prefs[X] (user opted into this feature) +``` +- **Admin turns `ai.enabled` off ⇒ AI vanishes for everyone**, regardless of any user + preference. This is the behaviour the brief mandates. +- With AI on at the system level, users still choose per-feature. The **Null AI provider + + capability flags** (blueprint [06](../06-ai-strategy.md)) mean a disabled path **falls back + or hides** — never errors, never blocks core email. + +## Settings precedence (general) +``` +system default → feature flag (may hard-disable) → user override (only where UserOverridable) +``` + +## Maintenance mode +- `read-only`: mutations (send/cleanup/label) blocked with a banner; browsing/search stay up. +- `locked-except-admin`: only Admins can use the app (for migrations/upgrades). +- Enforced at the API via a middleware policy + surfaced as a global banner in the UI. + +## Auditing +Flag and system-setting changes are **admin actions → audit-logged** (who/what/old→new/when). diff --git a/docs/discovery/multi-provider/05-admin-system.md b/docs/discovery/multi-provider/05-admin-system.md new file mode 100644 index 0000000..a91b02b --- /dev/null +++ b/docs/discovery/multi-provider/05-admin-system.md @@ -0,0 +1,45 @@ +# 05 — Admin System (Part 4) + +An Admin-only panel to run the instance. **Admins manage the platform, not people's +inboxes** — no admin route can read another user's mail (see [06](06-security-model.md)). + +## Sections +| Section | Admin can | Notes | +|---------|-----------|-------| +| **Users** | List users; view role/status/last-login; **promote/demote** (Admin↔Member); **suspend/reactivate**; **revoke sessions**; remove user (with data-deletion policy) | **Never** view a user's email contents | +| **Feature flags** | List all flags; toggle `enabled`; set scope/rollout; per-role rollout | Includes AI + provider flags | +| **AI (global)** | Master `ai.enabled` toggle + per-feature (`ai.summaries`, `ai.semantic_search`, `ai.ask_inbox`…); see Ollama/model status | Off ⇒ AI hidden for everyone ([04](04-settings-and-flags.md)) | +| **Providers** | Enable/disable `provider.google` / `provider.microsoft` / `provider.imap` | Disabled ⇒ hidden on login picker; existing accounts pause | +| **System config** | Maintenance mode (off/read-only/locked); `registration_open`; org name; default theme; retention | Sensitive → step-up + audit | +| **Monitoring** | Basic health overview (below) | Read-only | +| **Audit log** | Search/filter admin + security events | Append-only | + +## Monitoring overview (basic) +- **Sync health:** per-account last-sync time, `ReauthNeeded` count, error rate; job-queue depth. +- **AI/Ollama:** reachable? loaded models, VRAM headroom, recent latency, failure rate. +- **Sessions:** active session count; recent logins. +- **System:** DB size / message count; background-job backlog; recent errors (from Serilog). +- Deliberately **overview-only** — deep observability is a future opportunity, not v1. + +## Access control & bootstrap +- Every admin route requires the **Admin policy**; sensitive mutations require **confirmation/ + step-up** + are **rate-limited** and **audited**. +- **Bootstrap:** the first user to sign in becomes **Admin** (one-time). Afterwards, admin is + granted only by an existing Admin (audited, forces target session refresh so new/removed + privileges take effect immediately). +- Guardrails: an Admin cannot demote/suspend the **last remaining Admin** (lock-out prevention). + +## Audit logging (what's recorded) +Actor · action · target (user/flag/setting/provider) · old→new · ip · timestamp — for **all** +admin mutations and security events (role change, flag toggle, provider disable, maintenance +on/off, session revoke, user suspend). Append-only `audit_logs`; visible in the Audit section; +exportable. + +## API surface (Admin-scoped, all audited) +``` +GET/PATCH /admin/users /admin/users/{id}/role /admin/users/{id}/status +GET/PATCH /admin/flags /admin/flags/{key} +GET/PATCH /admin/system-settings +GET /admin/monitoring /admin/audit +``` +All behind the Admin policy + maintenance-aware middleware. diff --git a/docs/discovery/multi-provider/06-security-model.md b/docs/discovery/multi-provider/06-security-model.md new file mode 100644 index 0000000..d37d79c --- /dev/null +++ b/docs/discovery/multi-provider/06-security-model.md @@ -0,0 +1,79 @@ +# 06 — Security Model (Part 5) + +Builds on the existing hardening (read-only scope, encrypted tokens, IDOR global query +filters, SSRF egress guard, non-root containers, confirmed destructive actions) and adds +what multi-user + admin + multi-provider require. + +## RBAC +- Roles: **Admin** · **Member** (small-team, one org). First user bootstraps as Admin. +- Enforced by **policy-based authorization** at the API (ASP.NET Core policies), not in the UI. + +| Capability | Member | Admin | +|------------|:------:|:-----:| +| Read/manage **own** mail & accounts | ✅ | ✅ | +| Own user settings | ✅ | ✅ | +| Link/unlink **own** provider accounts | ✅ | ✅ | +| View/manage **other users** | ❌ | ✅ | +| Toggle **feature flags** (incl. AI global) | ❌ | ✅ | +| Enable/disable **providers** | ❌ | ✅ | +| **Maintenance mode**, system settings | ❌ | ✅ | +| View **audit log** & monitoring | ❌ | ✅ | +- **No cross-user data access, ever** — Admin manages *accounts/flags/system*, **not** other + users' email contents (privacy). Admin power is over the *platform*, not people's inboxes. + +## OAuth token storage +- Refresh/access tokens **encrypted at rest** with the Data Protection API (AES); keys persist + to the mounted `/keys` volume (existing). Decrypted **only in-memory** for the moment of an + API call. **Never logged, never sent to the browser.** +- 1:1 `provider_tokens` per account; rotation timestamped; a compromised/rotated token is + replaced atomically. Token columns are `bytea` ciphertext, not readable in DB dumps. + +## Session handling +- Opaque **server-side sessions** (DB-backed) + HttpOnly/Secure/SameSite cookie; **id rotated + on login** (anti-fixation); idle + absolute expiry; server-side **revocation** (logout, + sign-out-everywhere, admin revoke, role change). CSRF via SameSite + token. + +## Admin access protection +- Admin routes require the **Admin policy**; sensitive mutations (toggle AI global, disable a + provider, suspend a user, enter maintenance) require a **confirmation / step-up** and are + **rate-limited**. +- **Every admin action is audit-logged** (`audit_logs`: actor, action, target, metadata, ip, + time) — append-only. +- First-admin bootstrap is one-time; afterwards admin is grant-only by an existing Admin + (logged). Guard against privilege escalation: role changes are Admin-only + audited + force + session refresh. + +## API security boundaries +- **Per-user isolation** via EF **global query filters** (extended to `account_id`/`user_id`) + so a query can *never* return another user's rows — the IDOR safeguard, now multi-account. +- **Input validation** (FluentValidation) on all DTOs; **mass-assignment safe** (explicit DTOs, + no entity binding). +- **Rate limiting** on auth, admin, search, and AI endpoints. +- **SSRF egress guard** (existing) constrains all outbound calls — provider APIs, IMAP hosts, + Ollama, and any opt-in cloud AI — to an allowlist; user-supplied IMAP hosts are validated. +- **Security headers** (CSP, HSTS, X-Frame-Options, etc.) via the reverse proxy/API; strict CORS. + +## Multi-provider & AI specifics +- **Least-privilege scopes** per provider; extra scopes added per-feature with consent. +- **Provider isolation:** disabling a provider flag revokes its use cleanly; per-account tokens + are independent (one reauth doesn't affect others). +- **Prompt injection:** email content is untrusted → LLM output is **advisory only, never + triggers actions**; a human/rule confirms. AI runs **local by default**; cloud AI is explicit + opt-in with per-feature consent + egress logging. +- **Attachments/vision:** sandboxed parsing, size/type limits, never executed. + +## Threat model (summary) +| Threat | Mitigation | +|--------|------------| +| Account hijack via linking | Must authenticate as target user; unique `(provider, sub)`; linking an owned identity blocked | +| Token theft / DB exposure | Encryption at rest; tokens never in logs/browser; rotation | +| Privilege escalation | Admin-only role changes, audited, session refresh; policy checks server-side | +| IDOR / cross-user leakage | Global query filters on user_id/account_id | +| CSRF / session fixation | SameSite + token; session id rotation; server-side revoke | +| SSRF (providers/IMAP/AI) | Egress allowlist guard; validate user-supplied hosts | +| Prompt injection | AI advisory-only; never acts; local-first | +| Mass admin abuse | Rate limit + step-up + full audit trail | + +## Non-negotiables +Admins manage the platform, **not** users' inboxes · tokens encrypted & browser-invisible · +every privileged action audited · AI never required and never acts autonomously. diff --git a/docs/discovery/multi-provider/07-ux-flows.md b/docs/discovery/multi-provider/07-ux-flows.md new file mode 100644 index 0000000..9c88a3c --- /dev/null +++ b/docs/discovery/multi-provider/07-ux-flows.md @@ -0,0 +1,75 @@ +# 07 — UX Flows (Part 7) + +Applies the [Design Brief](../00-design-brief.md) (Notion/Arc-professional, dark-first, +green accent, pointer-first, responsive). **Keep it simple** — these are utility surfaces, +not the daily driver. + +## 1. Provider selection on login +A calm, centred card — the *only* thing on screen. +``` + InboxIntel + ─────────────────────────── + Sign in to continue + [ ▸ Continue with Google ] + [ ▸ Continue with Microsoft ] + ( IMAP — coming soon, disabled ) + ─────────────────────────── + Your email stays on your machine. +``` +- Only **enabled** providers show (driven by `provider.*` flags). +- One click → provider OAuth → back into the app. First-timer lands on an empty, friendly + inbox with a "syncing your mail…" state. + +## 2. Connected accounts page +Reached from the account switcher or Settings → Accounts. +``` +Accounts [ + Add account ] +┌─────────────────────────────────────────────────────────┐ +│ 🟢 Google me@gmail.com Synced 2m ago ⋯ │ +│ 🟢 Microsoft me@outlook.com Synced 5m ago ⋯ │ +│ 🟠 Google old@gmail.com Reconnect needed → [Reconnect]│ +└─────────────────────────────────────────────────────────┘ +``` +- Status badges: 🟢 Active · 🟠 ReauthNeeded (with **Reconnect**) · ⚪ Disabled. +- Per-account row menu (⋯): set as default · sync now · rename · **remove account** (confirm + + explains local data deletion). +- **+ Add account** → provider picker → OAuth in link-mode → new row appears. +- **Account switcher** (top bar): "All accounts" (unified) or pick one to scope the inbox/search. + +## 3. Settings page (user) +Left sub-nav, one panel at a time — no overwhelm. +``` +Settings + Appearance ▸ Theme (System/Light/Dark) · density · layout + Accounts ▸ (the page above) + Notifications ▸ channels · quiet hours · priority-only + AI ▸ (visible only if ai.enabled; see below) + Privacy ▸ data, export, clear search history +``` + +## 4. AI toggle visibility +- If **`ai.enabled` (system) is OFF** → the **AI section is hidden entirely** (or shown as a + single disabled note: "AI features are turned off by your administrator"). No dead toggles. +- If **ON** → a master **"Use AI features"** switch (user opt-in) + per-feature toggles + (Summaries · Reply suggestions · Semantic search · Ask your inbox), each reflecting its + `ai.` flag. Toggling off a feature instantly falls back to the non-AI path. +- A small **status chip** ("Local · Ollama · ready") reassures it's on-device. + +## 5. Admin dashboard +Only visible to Admins (nav item appears for the Admin role). +``` +Admin + Overview ▸ sync health · Ollama/VRAM · sessions · errors (cards) + Users ▸ table: name · role · status · last login · [actions] + Flags ▸ toggles grouped: AI · Providers · Maintenance + System ▸ maintenance mode · registration · defaults + Audit ▸ searchable event log +``` +- Clean tables + toggles; destructive/sensitive actions show a **confirm dialog** (step-up). +- Maintenance mode shows a **global banner** to all users while active. + +## Cross-cutting +- **Reduced-motion & keyboard** reachability on all of the above (baseline a11y). +- **Empty/loading/error states** per the design system (skeletons, friendly empties, calm errors). +- Fully **responsive**: settings/admin sub-nav collapses to a top tab bar on mobile; the login + card is centred on all sizes. diff --git a/docs/discovery/multi-provider/08-ai-feature-flags.md b/docs/discovery/multi-provider/08-ai-feature-flags.md new file mode 100644 index 0000000..6e17307 --- /dev/null +++ b/docs/discovery/multi-provider/08-ai-feature-flags.md @@ -0,0 +1,68 @@ +# 08 — AI Feature-Flag Integration (Part 8) + +How the AI layer plugs into the flag system while staying **completely separable** from core +email logic. Extends the blueprint AI strategy ([../06](../06-ai-strategy.md)); the gate +math lives in [04](04-settings-and-flags.md). + +## Principle: AI is a guest, never a host +Core email (sync, search-lexical, cleanup, settings, admin) **never references an AI type**. +It calls domain services; those *optionally* consult AI through a single gate + facade. Remove +AI entirely and nothing in the core path breaks. + +``` +Core feature code + │ (never touches Ollama/IAiProvider directly) + ▼ +IAiGate.IsAiFeatureEnabled("summaries", user) ──► false ─► non-AI path / hide + │ true + ▼ +IInboxAi facade (Application) ──► model router ──► IAiProvider / IEmbeddingProvider + (Null | Ollama | future) +``` + +## Toggle behaviour (flag-driven) +- Before *any* AI call, code asks `IAiGate` (which folds in `ai.enabled` + `ai.` + + provider capability + user opt-in — the AND-chain from [04](04-settings-and-flags.md)). +- **Admin `ai.enabled` = off** ⇒ gate returns false everywhere ⇒ AI UI hidden, AI code paths + skipped. Flip on ⇒ features reappear (hot-reloaded flag cache) with **no redeploy**. +- Per-feature flags allow shipping AI features **dark** and enabling gradually (rollout). + +## Fallback when AI is disabled (per feature) +| AI feature | Fallback with AI off | +|------------|----------------------| +| Semantic / NL search | Lexical + structured + fuzzy search (still excellent) | +| Thread summary | Hidden; show first snippet + metadata | +| Reply suggestions | Hidden; normal compose | +| Follow-up detection | Heuristic-only (sent + question + no reply in N days) | +| Categorisation | `HeuristicClassifier` rules only | +| Ask-your-inbox | Feature hidden | +| Dedup | Exact-hash only (no near-dup) | +Every fallback is **first-class**, not a broken/greyed feature — this satisfies "AI must never +be required for core functionality." + +## Ollama integration layer (local models) +- `OllamaProvider` (`IAiProvider`) + `OllamaEmbeddingProvider` (`IEmbeddingProvider`) talk to a + local Ollama (own container, optional Compose `ai` profile). +- **Model router** maps logical task → model via config (`Ai:Models:{Chat,Embed,Vision}`); + swapping a model is a config change, not code. +- **VRAM guard** (RTX 3080 / 10 GB): embeddings hot, 7B warm, vision on-demand + ([../06](../06-ai-strategy.md)). +- **Health surfaced to admin** ([05](05-admin-system.md)): reachable? models loaded? VRAM? + latency? If Ollama is down, capability = false ⇒ gate falls back gracefully (no user errors). + +## Safe abstraction (`IAIProvider`) — separation guarantees +1. **Interface boundary:** only Infrastructure implements providers; Application depends on + `IInboxAi`/`IAiGate` abstractions. +2. **Null objects:** `NullAiProvider`/`NullEmbeddingProvider` return "unavailable" so the DI + graph is always valid, AI on or off. +3. **Analyzer pipeline:** AI enrichers (`IEmailAnalyzer`) declare required capabilities and are + **skipped** when unavailable — adding/removing AI features never touches core sync/search. +4. **Bounded:** every AI call has timeout + `CancellationToken` + Polly fallback to the Null + path; AI can never hang or crash the core. +5. **Provider-swap:** adding a future AI provider = one class + config + (optionally) a flag — + no feature-code changes. + +## Precedence recap (single source of truth) +The effective availability chain and admin master-switch semantics are defined once in +[04 — Settings & Feature Flags](04-settings-and-flags.md#ai-gating-precedence-the-key-requirement); +this document is the *architecture* of how features consume that decision. diff --git a/docs/discovery/multi-provider/09-implementation-plan.md b/docs/discovery/multi-provider/09-implementation-plan.md new file mode 100644 index 0000000..76665f3 --- /dev/null +++ b/docs/discovery/multi-provider/09-implementation-plan.md @@ -0,0 +1,57 @@ +# 09 — Implementation Plan (Part 9) + +Six phases (from the brief), each **shippable behind feature flags** so `main`/staging never +break and providers activate only when ready. Follows the established +[../../WORKFLOW.md](../../WORKFLOW.md) pipeline (PR → checks → staging → tag→prod). + +## Phase 1 — Provider abstraction + Google login refactor +- **Goal:** introduce the seam and move the *existing* Gmail behaviour behind it, with + OAuth-as-login and the multi-user identity foundation. +- **Deliverables:** `IEmailProvider` + `ProviderFactory`; **`GmailProvider`** adapter wrapping + today's Gmail code; `users` / `accounts` / `provider_tokens` / `sessions` tables; + OAuth-as-login for Google; first-user→Admin bootstrap. +- **Flags:** `provider.google` (on). **Exit:** existing Gmail users function unchanged through + the new abstraction; sign-in creates a `user`+`account`; all tests green. + +## Phase 2 — Microsoft Outlook integration +- **Goal:** prove the abstraction with a second provider. +- **Deliverables:** **`OutlookProvider`** (Microsoft Graph, delta query); Microsoft OAuth login + + link-mode; scope config; normaliser mappings. +- **Flags:** `provider.microsoft` (**off** until verified, then rollout). **Exit:** a user can + link an Outlook mailbox; it syncs and searches alongside Gmail; no core changes needed. + +## Phase 3 — Unified email model + sync engine +- **Goal:** formalise the normalised store and provider-agnostic sync. +- **Deliverables:** normalised `email_messages`/`email_threads` (widened `search_vector`, + nullable `embedding`); **`ISyncOrchestrator`** + `AccountSyncWorker` (per-account cursors, + idempotent upserts, incremental); **data migration** of existing Gmail rows → default account. +- **Flags:** none user-facing; migration behind a maintenance window. **Exit:** all providers + sync through one orchestrator into one store; cross-account search works. + +## Phase 4 — Settings system +- **Goal:** user + system settings + the flag engine. +- **Deliverables:** `user_settings`, `system_settings`, **`feature_flags`** + `IFeatureFlags`/ + `IAiGate` (cached, fail-closed); settings UI; maintenance-mode middleware. +- **Flags:** self-hosting (the engine that hosts the rest). **Exit:** users edit prefs; admins + can flip flags; AI gate resolves the AND-chain. + +## Phase 5 — Admin panel + feature flags +- **Goal:** the Admin surface + RBAC + audit. +- **Deliverables:** Admin API (policy-gated) + UI (Users/Flags/AI/Providers/System/Monitoring/ + Audit); `audit_logs`; role management; basic monitoring. +- **Flags:** admin nav shown by role. **Exit:** an Admin can manage users/flags/providers, + every action audited; step-up + rate-limit enforced. + +## Phase 6 — AI integration layer +- **Goal:** wire optional AI behind the gate. +- **Deliverables:** extend `IAiProvider` (+`CompleteStructuredAsync`, `IEmbeddingProvider`); + `IInboxAi` facade + model router + VRAM guard; analyzer pipeline; first AI features + (summaries, reply, follow-up-confirm) each **flag-gated + fallback**. +- **Flags:** `ai.enabled` + `ai.` (rollout). **Exit:** AI features work when enabled, + **vanish/fallback** when off; core unaffected; Ollama health in admin. + +## Sequencing notes +- Phases 1–3 are the platform spine; 4–5 the control plane; 6 the optional intelligence. +- **Nothing activates on merge** — flags gate everything, so partial phases are safe on `develop`/`main`. +- Aligns with the blueprint roadmap ([../09](../09-roadmap.md)): this multi-provider work is a + **v1.x platform epic** that the search/AI features then build on. diff --git a/docs/discovery/multi-provider/10-git-workflow.md b/docs/discovery/multi-provider/10-git-workflow.md new file mode 100644 index 0000000..a35f728 --- /dev/null +++ b/docs/discovery/multi-provider/10-git-workflow.md @@ -0,0 +1,50 @@ +# 10 — Git Workflow (Part 11) + +Extends [../../WORKFLOW.md](../../WORKFLOW.md) for multi-phase, multi-provider work. The +core idea: **feature flags decouple *merging* from *activating*, which makes every +integration safe to land and trivial to roll back.** + +## Branching per phase +| Phase | Milestone | Branches | +|-------|-----------|----------| +| 1 Provider abstraction + Google | `v1.x` | `epic/provider-platform` → `feature/email-provider-interface` · `feature/gmail-adapter` · `feature/oauth-login-users` | +| 2 Microsoft | `v1.x` | `feature/outlook-provider` · `feature/microsoft-oauth` | +| 3 Unified model + sync | `v1.x` | `feature/normalised-email-model` · `feature/sync-orchestrator` · `feature/data-migration` | +| 4 Settings | `v1.x` | `feature/settings-store` · `feature/feature-flags-engine` | +| 5 Admin | `v1.x` | `feature/admin-api` · `feature/admin-ui` · `feature/audit-log` | +| 6 AI layer | `v1.x` | `feature/ai-abstraction-ext` · `feature/ai-analyzers` · `feature/ai-flag-gating` | +- `feature/* → develop` (squash), `develop → main` (merge commit) — as established. Epics are + tracked by milestone/label; features integrate continuously (no long epic branch). + +## PR structure per provider integration +Each provider is a self-contained PR set that lands **dark**: +1. **Adapter PR** — `IEmailProvider` impl + normaliser + unit tests (mocked provider). +2. **Auth PR** — OAuth login/link for that provider. +3. **Enablement PR** — register in `ProviderFactory` + seed `provider.` flag **OFF**. +4. **Activation** — flip the flag on in staging → verify end-to-end → roll out in prod. +- **PR checklist adds:** provider behind a flag (off by default) · normaliser tests · token + encryption verified · no Domain leakage · docs updated. + +## Feature flags prevent breaking changes +- Merge = code present but **inert** until its flag is on. So half-finished providers/AI can + live on `main` safely; CI stays green; no long-lived divergence. +- AI ships behind `ai.*`; providers behind `provider.*`; risky changes behind their own flag. + +## Rollback strategy (per provider / per feature) +| Level | Action | Speed | +|-------|--------|-------| +| **Flag** (first resort) | Admin flips `provider.` / `ai.` **off** | **Instant, no deploy** — feature disappears, existing data untouched | +| **Deploy** | Redeploy the previous **tag** (`vX.Y.Z-1`) | Minutes (pipeline) | +| **Revert** | `git revert` the PR → PR → merge → deploy | Minutes–hours | +| **Data** | Provider accounts are isolated; disabling a provider **pauses** its sync — no destructive change to migrate back | Safe by design | +- Because providers are isolated and flag-gated, a bad integration **never blocks the others** + and never requires a risky data rollback. + +## Release milestones +- Cut a tag when a phase reaches its exit criteria (`deploy-prod.yml` fires on `v*`). +- Suggested: `v1.1` provider platform + Google · `v1.2` +Outlook · `v1.3` unified sync · + `v1.4` settings+admin · `v1.5` AI layer — folded into the blueprint roadmap ([../09](../09-roadmap.md)). + +## Docs alongside code +Every feature PR updates the relevant `multi-provider/*` doc + `CHANGELOG.md`; on approval the +design docs graduate into living `docs/` references (provider system, settings, admin, security). diff --git a/docs/discovery/multi-provider/11-risk-analysis.md b/docs/discovery/multi-provider/11-risk-analysis.md new file mode 100644 index 0000000..c23c3fc --- /dev/null +++ b/docs/discovery/multi-provider/11-risk-analysis.md @@ -0,0 +1,31 @@ +# 11 — Risk Analysis + +Risks specific to the multi-provider + multi-user + admin evolution (the blueprint-wide risks +are in [../11](../11-risks-and-future.md)). L/I = Likelihood/Impact (H/M/L). + +| # | Risk | L | I | Mitigation | +|---|------|---|---|------------| +| M1 | **Account-linking hijack** (attach someone's mailbox to your user) | L | H | Must authenticate as the target user; unique `(provider, sub)`; explicit "already owned" block; audited ([02](02-auth-and-signin.md)) | +| M2 | **Cross-user data leakage** (multi-user IDOR) | M | H | EF global query filters on `user_id`/`account_id`; policy-based authz; no admin route reads user mail; tests for isolation | +| M3 | **OAuth token theft / exposure** | L | H | Encrypted at rest (Data Protection); never logged or sent to browser; rotation; `bytea` ciphertext | +| M4 | **Provider quirks break sync** (Graph delta resets, IMAP `UIDVALIDITY` change, Gmail history gaps) | M | M | Per-provider cursor handling with **full-resync fallback**; idempotent upserts; capability flags; account flips to needs-attention, never crashes | +| M5 | **Migration corrupts existing Gmail data** | L | H | Additive-only schema; backfill is idempotent; maintenance window; tested on a staging copy; reversible ([12](12-migration-guide.md)) | +| M6 | **RBAC bug grants Member admin powers** | L | H | Server-side policies (not UI); admin-only role changes audited + force session refresh; can't demote last Admin; authz tests | +| M7 | **Feature-flag misconfiguration** (AI/provider on when not ready) | M | M | Flags default **off/fail-closed**; land dark; enable in staging first; audited toggles; rollout by role | +| M8 | **AI gate bypass** (feature runs when disabled) | L | M | Single `IAiGate` chokepoint before any AI call; Null providers; capability checks; no direct provider refs in core | +| M9 | **Session/CSRF weaknesses** across new surfaces | M | M | Server-side sessions, id rotation, SameSite + CSRF token, revoke-all, short idle expiry | +| M10 | **Admin abuse / mistake** (mass suspend, wrong flag) | M | M | Step-up confirm + rate-limit + full audit trail + reversible flags | +| M11 | **Scaling: millions of msgs × multiple accounts** | M | M | Denormalised `user_id`, keyset pagination, GIN/trgm/HNSW indexes, optional `account_id` partitioning, per-account sync throttling | +| M12 | **Provider OAuth app setup burden** (separate Google + Microsoft app registrations, redirect URIs, verification) | M | L | Documented setup per provider; providers flag-gated so an unconfigured one is simply hidden | +| M13 | **Scope/verification friction** (Google restricted scopes, MS admin consent) | M | M | Least-privilege scopes; document the consent/verification path; self-host uses the operator's own OAuth apps | + +## Top watch-items +- **M2 (isolation)** and **M6 (RBAC)** — the two ways multi-user can go wrong; both mitigated + by server-side authz + query filters + isolation tests as a release gate. +- **M4 (provider sync quirks)** — the most likely *operational* pain; the full-resync fallback + and per-account isolation contain it. + +## Overall +The flag-gated, additive, isolated design makes this evolution **low-blast-radius**: each +provider and the AI layer land dark and roll back by flag, migrations are additive, and the +security model closes the new multi-user gaps. Proceed **phase by phase behind flags**. diff --git a/docs/discovery/multi-provider/12-migration-guide.md b/docs/discovery/multi-provider/12-migration-guide.md new file mode 100644 index 0000000..3b56bbc --- /dev/null +++ b/docs/discovery/multi-provider/12-migration-guide.md @@ -0,0 +1,52 @@ +# 12 — Migration Guide (Part 10) + +Moving the current **single-account Gmail** app to the **multi-provider, multi-user** model +— **additive and reversible**, no destructive step. Executed as ordered EF Core migrations + +idempotent backfills, behind a short maintenance window. + +## Principles +- **Additive first:** create new tables/columns before moving data; keep old columns until parity is verified. +- **Idempotent backfills:** safe to re-run; keyed on stable ids. +- **Flag-gated cutover:** the new sign-in/model activates behind flags; the old path stays until removed. +- **Reversible:** each step has a documented rollback; no data is deleted during migration. + +## Step-by-step +1. **Schema (additive migration)** + - Create `users`, `accounts`, `provider_tokens`, `sessions`, `user_settings`, + `system_settings`, `feature_flags`, `audit_logs`. + - Add `account_id`, `user_id` (nullable) to the current email/thread tables; widen + `search_vector`; add nullable `embedding vector(768)` + `pgvector` extension. +2. **Identity backfill** + - For the existing operator/user, create a `users` row; mark the **first user = Admin**. + - Create one **`Google` `account`** per existing identity (`is_login_identity=true`); move + current encrypted Gmail tokens → `provider_tokens`. +3. **Email backfill** + - Set `account_id`/`user_id` on all existing `Email`/thread rows to the default Google account. + - Regenerate the widened `search_vector`; leave `embedding` null (backfilled later by the AI phase). + - Enforce the new unique keys `(account_id, provider_message_id)` / `(account_id, provider_thread_id)`. +4. **Config seed** + - Seed `feature_flags`: `provider.google=on`, `provider.microsoft=off`, `provider.imap=off`, + `ai.enabled` = derived from the current `Ai:Mode` (Disabled→off), `ai.*`=off, `maintenance.*`=off. + - Create `system_settings` singleton; create `user_settings` from any existing per-user prefs (else defaults). +5. **Cutover** + - Enable the new OAuth-as-login + unified sync behind their flags; verify on **staging** first + (the pipeline we built), then production via a tagged release. +6. **Cleanup (later, separate migration)** + - Once parity is confirmed in production, drop obsolete columns/paths. Not part of the cutover. + +## Verification checklist +- Existing user signs in via Google → lands on their mail unchanged. +- Email counts match pre/post; search returns identical results for sample queries. +- Tokens decrypt and refresh; sync resumes from the correct cursor. +- No cross-user rows visible (isolation test). + +## Rollback +- **Pre-cutover:** additive changes are inert → simply don't flip the flags; drop new tables if aborting. +- **Post-cutover issue:** flip flags off / redeploy previous **tag**; old columns still present → + the legacy path still works. No data was deleted, so no data rollback is needed. + +## Provider-app prerequisites (operator setup) +- **Google:** OAuth client (existing) + redirect `/(…)/signin/google`; scopes `gmail.readonly gmail.modify`. +- **Microsoft:** register an Entra app; redirect `/(…)/signin/microsoft`; scopes `Mail.Read Mail.ReadWrite offline_access`; admin consent if required. +- **IMAP (future):** per-account host/credentials; validated against the SSRF allowlist. +- Because providers are flag-gated, an unconfigured provider is simply hidden — configure, then enable. diff --git a/docs/discovery/multi-provider/README.md b/docs/discovery/multi-provider/README.md new file mode 100644 index 0000000..925b788 --- /dev/null +++ b/docs/discovery/multi-provider/README.md @@ -0,0 +1,48 @@ +# Multi-Provider Email Platform + Admin/Settings — Design + +**Design + architecture phase. No implementation until approved.** + +Evolves InboxIntel from a single-account Gmail tool into a **multi-provider platform** +(Gmail · Outlook/Graph · future IMAP) for a **small self-hosted team**, with settings, +feature flags, and an admin panel. Extends — does not discard — the +[discovery blueprint](../README.md). + +## Locked decisions (from interview) +1. **Tenancy:** **small team, self-hosted, one org.** Roles = **Admin / Member**. Shared + system settings + feature flags; each member's mail is private to them. No multi-tenant + org table (one implicit org); the model stays extensible to multi-org later. +2. **App identity = provider OAuth.** The first Google/Microsoft sign-in **creates/authenticates + the InboxIntel user**; additional mailboxes **link** to that same user. **No passwords stored.** + +## How this reshapes the blueprint (the "review" deltas) +| Blueprint assumption | New reality | +|----------------------|-------------| +| Single-user, local-first | **Multi-user (small team)** with Admin/Member RBAC + admin panel | +| Gmail-centric `Email`/`Sender` | **Account-scoped, provider-normalised** model (`IEmailProvider`) | +| AI gated by `Ai:Mode` + user pref | **AI gated by system feature flag → user pref → capability** (flag wins) | +| One implicit mailbox | **N provider accounts per user** (`accounts` table + per-account sync cursors) | +| Sync = `GmailSyncWorker` | **Provider-agnostic sync orchestrator** dispatching to provider adapters | + +These deltas will be back-ported into main-blueprint docs [08](../08-technical-architecture.md) +and [09](../09-roadmap.md) when this design is approved. + +## Documents +| # | Doc | Covers (brief part) | Status | +|---|-----|---------------------|--------| +| 01 | [Provider Abstraction](01-provider-abstraction.md) | Part 1 | ✅ draft | +| 02 | [Auth & Sign-in](02-auth-and-signin.md) | Part 2 | ✅ draft | +| 03 | [Database Design](03-database-design.md) | Part 6 | ✅ draft | +| 04 | [Settings & Feature Flags](04-settings-and-flags.md) | Part 3 | ✅ draft | +| 05 | [Admin System](05-admin-system.md) | Part 4 | ✅ draft | +| 06 | [Security Model](06-security-model.md) | Part 5 | ✅ draft | +| 07 | [UX Flows](07-ux-flows.md) | Part 7 | ✅ draft | +| 08 | [AI Feature-Flag Integration](08-ai-feature-flags.md) | Part 8 | ✅ draft | +| 09 | [Implementation Plan](09-implementation-plan.md) | Part 9 | ✅ draft | +| 10 | [Git Workflow](10-git-workflow.md) | Part 11 | ✅ draft | +| 11 | [Risk Analysis](11-risk-analysis.md) | output | ✅ draft | +| 12 | [Migration Guide](12-migration-guide.md) | Part 10 | ✅ draft | + +## Non-negotiables carried forward +Provider logic **never leaks into Domain** · search works **across all a user's accounts** +· emails stored in **one unified format** · **AI never required** for core function · +tokens **encrypted at rest** · admin actions **audit-logged**.