Inboxintel/docs/discovery/06-ai-strategy.md

06 — AI Strategy & Model Recommendations (Phase 5)

AI is optional, modular, local-first, and explainable. The app must be fully usable with AI disabled. Primary hardware: NVIDIA RTX 3080 (10 GB VRAM). All AI sits behind an abstraction so models/providers swap without touching application logic.

---

Part A — The AI abstraction (extends the existing seam)

Today there is one interface, IAiProvider.CompleteAsync(system, user), with NullAiProvider / OllamaProvider / OpenAiProvider. We extend it into a small set of capability interfaces plus a task-oriented facade.

Provider-level interfaces (Infrastructure)

public interface IAiProvider {                    // chat / generation (exists)
    AiProviderMode Mode { get; }
    Task<string> CompleteAsync(string system, string user, CancellationToken ct = default);
    // NEW: schema-constrained JSON (Ollama `format`, OpenAI `response_format`)
    Task<T?> CompleteStructuredAsync<T>(string system, string user, CancellationToken ct = default);
}

public interface IEmbeddingProvider {             // NEW: vectors for semantic search/dedup
    Task<float[]> EmbedAsync(string text, CancellationToken ct = default);
    Task<IReadOnlyList<float[]>> EmbedBatchAsync(IReadOnlyList<string> texts, CancellationToken ct = default);
}

public interface IAiCapabilities {                // NEW: lets the UI hide what isn't available
    bool Chat { get; } bool Embeddings { get; } bool Vision { get; } bool StructuredJson { get; }
}

NullAiProvider/NullEmbeddingProvider return empty/"unavailable" so every caller has a graceful non-AI path (fall back to lexical search, heuristics, or hide the feature).

Task facade (Application) — the only thing feature code calls

public interface IInboxAi {
    Task<AiResult<string>>        SummarizeThreadAsync(Guid threadId, CancellationToken ct);
    Task<AiResult<Extraction>>    ExtractAsync(string body, ExtractKinds kinds, CancellationToken ct);
    Task<AiResult<Classification>>ClassifyAsync(string subject, string body, CancellationToken ct);
    Task<AiResult<Answer>>        AskAsync(string question, SearchScope scope, CancellationToken ct); // RAG
    Task<float[]?>                EmbedAsync(string text, CancellationToken ct);
    Task<AiResult<RiskVerdict>>   AssessPhishingAsync(EmailContext ctx, CancellationToken ct);
}

• AiResult<T> carries { value, available, model, latencyMs, explanation } → powers the "explainability" and graceful-degradation requirements everywhere.
• A model router maps logical task → model name from config, so swapping a model is a config change: Ai:Models:{Chat,Summarize,Embed,Classify,Extract,Vision}.
• Prompt templates are versioned files, separate from code.
• Cross-cutting: async, CancellationToken, per-call timeout + Polly fallback to the Null path, and a token/VRAM-aware concurrency limiter.

Config shape

Ai:Mode = Disabled | LocalOllama | Cloud...
Ai:Ollama:BaseUrl, KeepAliveSeconds, VramBudgetMb
Ai:Models:Chat = qwen2.5:7b-instruct
Ai:Models:Embed = nomic-embed-text
Ai:Models:Vision = qwen2.5vl:7b        (load-on-demand)

---

Part B — Model recommendations for the RTX 3080 (10 GB)

Budget reality: a 7–8B instruct model at Q4/Q5 (~5 GB + ~1 GB KV cache at 8k context) runs alongside a small embedding model (~0.5 GB) inside 10 GB with headroom. 14B is possible (~9 GB) but leaves no room for concurrency and is slower — 7–8B is the sweet spot.

Task	Recommended (primary)	Alt	~VRAM (Q4)	Latency	License	Notes
Chat / reasoning / RAG answers	Qwen2.5-7B-Instruct	Llama-3.1-8B, Gemma-2-9B	~5.5 GB	~30–60 tok/s	Apache-2.0	Strong reasoning, multilingual, great JSON
Summarisation	reuse Qwen2.5-7B	Llama-3.1-8B	(shared)	fast	Apache-2.0	No separate model needed
Extraction (tasks/dates/entities)	reuse Qwen2.5-7B + format:json	Phi-4	(shared)	fast	Apache-2.0	Schema-constrained JSON output
Classification / tagging	Qwen2.5-3B (fast) or embeddings-zero-shot	rules-first (exists)	~2.2 GB	very fast	Apache-2.0/Qwen	High-volume → small model or embeddings, not 7B
Embeddings (semantic search, dedup, zero-shot class.)	nomic-embed-text (768-d, 8k ctx)	mxbai-embed-large, bge-m3 (multilingual)	~0.5 GB	very fast	Apache-2.0	Keep permanently loaded
OCR / attachment vision	Qwen2.5-VL-7B or MiniCPM-V	llama3.2-vision-11B, moondream (tiny)	~7 GB	slow	Apache/varied	Load-on-demand; evicts the text LLM
Language detection	library, not an LLM (fastText-lid / CLD3)	—	~0	instant	MIT/Apache	Don't waste VRAM on this
Translation	reuse Qwen2.5-7B (multilingual)	NLLB-200 (dedicated MT)	(shared)	med	Apache-2.0	Good for common pairs; on-demand
Spam detection	rules + small classifier/embeddings	3B LLM for edge cases	~0–2 GB	fast	—	Traditional-first; LLM only for ambiguity
Phishing detection	rules/URL analysis + embeddings, LLM reasoning for suspicious	Qwen2.5-7B for explanation	shared	on-demand	—	LLM explains why; runs async on flagged mail
Duplicate detection	hashing (exact) + embedding cosine (near)	—	(uses embed)	fast	—	No chat model required

Selection rationale: accuracy + Apache-2.0 licensing (commercial-safe) + fits 10 GB

• strong structured-JSON and multilingual (mailboxes aren't English-only). Qwen2.5-7B is a single versatile workhorse for chat/summarise/extract/translate; nomic-embed-text is a tiny always-on retrieval engine; a vision model is a heavy, rare, on-demand guest.

Load policy (VRAM management)

State	Models	Rationale
Hot (permanent)	nomic-embed-text (~0.5 GB)	Used constantly (search, dedup, classify, backfill) — must be instant
Warm (keep-alive)	qwen2.5:7b-instruct (~5.5 GB)	Loaded on first chat/summarise, kept warm ~5–10 min idle via Ollama keep_alive, then unloaded
Cold (on-demand)	qwen2.5vl:7b vision, nllb translation	Loaded only for the specific job; unloaded after (would otherwise exceed budget alongside the 7B)

• Ollama handles load/unload; we set keep_alive per task and a VRAM budget guard that serialises a vision job behind unloading the text LLM.
• Embedding backfill runs as a hosted background worker, batched, low priority, so it never starves interactive requests.

---

Part C — Principles

1. Traditional-first. If rules/heuristics/library solve it well (language detection, exact dedup, basic spam, most categorisation), use them — AI only where it clearly wins.
2. Always optional. Null providers + capability flags → the app is whole without AI.
3. Local & private by default. Ollama on-device; data never leaves unless a cloud provider is explicitly chosen.
4. Explainable. Every AI output carries a short rationale + the model used.
5. Swappable. Logical-task→model routing via config; prompts versioned; no feature code references a model name.
6. Bounded. Timeouts, cancellation, Polly fallback, VRAM-aware concurrency — AI can never hang or crash the core experience.