Inboxintel/docs/discovery/06-ai-strategy.md

# 06 — AI Strategy & Model Recommendations (Phase 5)

AI is **optional, modular, local-first, and explainable**. The app must be fully usable
with AI disabled. Primary hardware: **NVIDIA RTX 3080 (10 GB VRAM)**. All AI sits behind
an abstraction so models/providers swap without touching application logic.

---

## Part A — The AI abstraction (extends the existing seam)
Today there is one interface, `IAiProvider.CompleteAsync(system, user)`, with
`NullAiProvider` / `OllamaProvider` / `OpenAiProvider`. We extend it into a small set of
capability interfaces plus a task-oriented facade.

### Provider-level interfaces (Infrastructure)
```csharp
public interface IAiProvider {                    // chat / generation (exists)
    AiProviderMode Mode { get; }
    Task<string> CompleteAsync(string system, string user, CancellationToken ct = default);
    // NEW: schema-constrained JSON (Ollama `format`, OpenAI `response_format`)
    Task<T?> CompleteStructuredAsync<T>(string system, string user, CancellationToken ct = default);
}

public interface IEmbeddingProvider {             // NEW: vectors for semantic search/dedup
    Task<float[]> EmbedAsync(string text, CancellationToken ct = default);
    Task<IReadOnlyList<float[]>> EmbedBatchAsync(IReadOnlyList<string> texts, CancellationToken ct = default);
}

public interface IAiCapabilities {                // NEW: lets the UI hide what isn't available
    bool Chat { get; } bool Embeddings { get; } bool Vision { get; } bool StructuredJson { get; }
}
```
`NullAiProvider`/`NullEmbeddingProvider` return empty/"unavailable" so **every caller has a
graceful non-AI path** (fall back to lexical search, heuristics, or hide the feature).

### Task facade (Application) — the only thing feature code calls
```csharp
public interface IInboxAi {
    Task<AiResult<string>>        SummarizeThreadAsync(Guid threadId, CancellationToken ct);
    Task<AiResult<Extraction>>    ExtractAsync(string body, ExtractKinds kinds, CancellationToken ct);
    Task<AiResult<Classification>>ClassifyAsync(string subject, string body, CancellationToken ct);
    Task<AiResult<Answer>>        AskAsync(string question, SearchScope scope, CancellationToken ct); // RAG
    Task<float[]?>                EmbedAsync(string text, CancellationToken ct);
    Task<AiResult<RiskVerdict>>   AssessPhishingAsync(EmailContext ctx, CancellationToken ct);
}
```
- `AiResult<T>` carries `{ value, available, model, latencyMs, explanation }` → powers the
  **"explainability"** and graceful-degradation requirements everywhere.
- A **model router** maps *logical task → model name* from config, so swapping a model is a
  config change: `Ai:Models:{Chat,Summarize,Embed,Classify,Extract,Vision}`.
- Prompt templates are **versioned files**, separate from code.
- Cross-cutting: async, `CancellationToken`, per-call timeout + **Polly** fallback to the
  Null path, and a token/VRAM-aware concurrency limiter.

### Config shape
```
Ai:Mode = Disabled | LocalOllama | Cloud...
Ai:Ollama:BaseUrl, KeepAliveSeconds, VramBudgetMb
Ai:Models:Chat = qwen2.5:7b-instruct
Ai:Models:Embed = nomic-embed-text
Ai:Models:Vision = qwen2.5vl:7b        (load-on-demand)
```

---

## Part B — Model recommendations for the RTX 3080 (10 GB)

**Budget reality:** a 7–8B instruct model at Q4/Q5 (~5 GB + ~1 GB KV cache at 8k context)
runs **alongside** a small embedding model (~0.5 GB) inside 10 GB with headroom. 14B is
possible (~9 GB) but leaves no room for concurrency and is slower — **7–8B is the sweet spot.**

| Task | Recommended (primary) | Alt | ~VRAM (Q4) | Latency | License | Notes |
|------|----------------------|-----|-----------|---------|---------|-------|
| **Chat / reasoning / RAG answers** | **Qwen2.5-7B-Instruct** | Llama-3.1-8B, Gemma-2-9B | ~5.5 GB | ~30–60 tok/s | Apache-2.0 | Strong reasoning, multilingual, great JSON |
| **Summarisation** | reuse **Qwen2.5-7B** | Llama-3.1-8B | (shared) | fast | Apache-2.0 | No separate model needed |
| **Extraction (tasks/dates/entities)** | reuse **Qwen2.5-7B** + `format:json` | Phi-4 | (shared) | fast | Apache-2.0 | Schema-constrained JSON output |
| **Classification / tagging** | **Qwen2.5-3B** (fast) *or* embeddings-zero-shot | rules-first (exists) | ~2.2 GB | very fast | Apache-2.0/Qwen | High-volume → small model or embeddings, not 7B |
| **Embeddings (semantic search, dedup, zero-shot class.)** | **nomic-embed-text** (768-d, 8k ctx) | mxbai-embed-large, **bge-m3** (multilingual) | ~0.5 GB | very fast | Apache-2.0 | **Keep permanently loaded** |
| **OCR / attachment vision** | **Qwen2.5-VL-7B** *or* **MiniCPM-V** | llama3.2-vision-11B, moondream (tiny) | ~7 GB | slow | Apache/varied | **Load-on-demand**; evicts the text LLM |
| **Language detection** | **library, not an LLM** (fastText-lid / CLD3) | — | ~0 | instant | MIT/Apache | Don't waste VRAM on this |
| **Translation** | reuse **Qwen2.5-7B** (multilingual) | NLLB-200 (dedicated MT) | (shared) | med | Apache-2.0 | Good for common pairs; on-demand |
| **Spam detection** | **rules + small classifier/embeddings** | 3B LLM for edge cases | ~0–2 GB | fast | — | Traditional-first; LLM only for ambiguity |
| **Phishing detection** | **rules/URL analysis + embeddings**, LLM **reasoning** for suspicious | Qwen2.5-7B for explanation | shared | on-demand | — | LLM explains *why*; runs async on flagged mail |
| **Duplicate detection** | **hashing (exact) + embedding cosine (near)** | — | (uses embed) | fast | — | No chat model required |

**Selection rationale:** accuracy + **Apache-2.0 licensing** (commercial-safe) + fits 10 GB
+ strong **structured-JSON** and multilingual (mailboxes aren't English-only). Qwen2.5-7B is
a single versatile workhorse for chat/summarise/extract/translate; nomic-embed-text is a
tiny always-on retrieval engine; a vision model is a heavy, rare, on-demand guest.

### Load policy (VRAM management)
| State | Models | Rationale |
|-------|--------|-----------|
| **Hot (permanent)** | `nomic-embed-text` (~0.5 GB) | Used constantly (search, dedup, classify, backfill) — must be instant |
| **Warm (keep-alive)** | `qwen2.5:7b-instruct` (~5.5 GB) | Loaded on first chat/summarise, kept warm ~5–10 min idle via Ollama `keep_alive`, then unloaded |
| **Cold (on-demand)** | `qwen2.5vl:7b` vision, `nllb` translation | Loaded only for the specific job; unloaded after (would otherwise exceed budget alongside the 7B) |
- Ollama handles load/unload; we set `keep_alive` per task and a **VRAM budget guard** that
  serialises a vision job behind unloading the text LLM.
- Embedding **backfill** runs as a hosted background worker, batched, low priority, so it
  never starves interactive requests.

---

## Part C — Principles
1. **Traditional-first.** If rules/heuristics/library solve it well (language detection,
   exact dedup, basic spam, most categorisation), use them — AI only where it *clearly* wins.
2. **Always optional.** Null providers + capability flags → the app is whole without AI.
3. **Local & private by default.** Ollama on-device; data never leaves unless a cloud
   provider is explicitly chosen.
4. **Explainable.** Every AI output carries a short rationale + the model used.
5. **Swappable.** Logical-task→model routing via config; prompts versioned; no feature code
   references a model name.
6. **Bounded.** Timeouts, cancellation, Polly fallback, VRAM-aware concurrency — AI can never
   hang or crash the core experience.