cesnimda
2b6a0ea3f0
World-class discovery specification (design phase, no code): - Executive summary, architecture review, competitor + user research - UX/UI redesign + design system + themes, flagship search redesign - AI strategy (local Ollama, RTX 3080) + feature catalogue - Technical architecture, roadmap (MVP->v3), git plan, risks - Multi-provider platform: IEmailProvider abstraction, OAuth-as-login, unified DB schema, settings + feature flags, admin/RBAC, security model, 6-phase implementation plan, migration guide Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
80 lines
6.4 KiB
Markdown
80 lines
6.4 KiB
Markdown
# 03 — Database Design (Part 6)
|
||
|
||
Schema for a **small-team, self-hosted, one-org** platform: multi-provider accounts per
|
||
user, a normalised email store scaling to **millions of messages**, settings, feature
|
||
flags, and audit. PostgreSQL + EF Core (+ pgvector for the blueprint's semantic tier).
|
||
|
||
## Entity map
|
||
```
|
||
users ─┬─< accounts ─┬─< provider_tokens (1:1, encrypted)
|
||
│ ├─< email_threads ─< email_messages ─┬─< attachments
|
||
│ │ └─< message_labels >─ labels
|
||
│ └─ sync_state (cursor)
|
||
├─< user_settings (1:1)
|
||
└─< audit_logs (actor)
|
||
system_settings (singleton) feature_flags
|
||
```
|
||
|
||
## Tables
|
||
|
||
### Identity & access
|
||
- **`users`** — the app identity (from OAuth).
|
||
`id (uuid pk) · primary_email (citext, unique) · display_name · avatar_url · role (enum: Admin|Member) · status (enum: Active|Suspended) · created_at · last_login_at`
|
||
*First-ever user is bootstrapped as **Admin** (see [05](05-admin-system.md)).*
|
||
- **`accounts`** — a connected mailbox **and** a login identity (OAuth-is-login).
|
||
`id (uuid pk) · user_id (fk) · provider (enum: Google|Microsoft|Imap) · provider_account_id (text, the OAuth 'sub' — immutable) · email (citext) · display_name · is_login_identity (bool) · status (enum: Active|ReauthNeeded|Disabled) · scopes (text[]) · added_at · last_sync_at`
|
||
**Unique:** `(provider, provider_account_id)` → resolves an OAuth login to exactly one account→user; prevents the same mailbox linking twice.
|
||
- **`provider_tokens`** — 1:1 with `accounts`, **encrypted at rest** (Data Protection API).
|
||
`account_id (pk/fk) · access_token_enc (bytea) · refresh_token_enc (bytea) · expires_at_utc · token_type · rotated_at`
|
||
*Never logged; see [06](06-security-model.md).*
|
||
- **`sessions`** — server-side app sessions (opaque cookie).
|
||
`id · user_id · created_at · expires_at · ip · user_agent · revoked_at` (supports "sign out everywhere" + admin revoke).
|
||
|
||
### Email (normalised, provider-agnostic)
|
||
- **`email_threads`** — per account.
|
||
`id (uuid pk) · account_id (fk) · user_id (denorm) · provider_thread_id · subject · participants (jsonb) · message_count · last_message_at`
|
||
**Unique:** `(account_id, provider_thread_id)`.
|
||
- **`email_messages`** — the big table.
|
||
`id (uuid pk) · account_id (fk) · user_id (denorm) · thread_id (fk) · provider_message_id · sender_id (fk) · subject · snippet · body_text · sent_at_utc · received_at_utc · size_bytes · flags (unread/starred/important/in_inbox/trashed as bits or bools) · has_attachments · category (enum) · has_list_unsubscribe · supports_one_click_unsub · search_vector (tsvector, generated, weighted) · embedding (vector(768), nullable) · created_at`
|
||
**Unique:** `(account_id, provider_message_id)` (idempotent upsert).
|
||
- **`labels`** (`id · account_id · provider_label_id · name · type`) + **`message_labels`** (`message_id · label_id`, pk both).
|
||
- **`attachments`** (`id · message_id · filename · mime · size · provider_attachment_id · content_text nullable` for future OCR/search).
|
||
- **`senders`** / **`domains`** (existing) — kept, scoped per user (or global with per-user stats materialised in `sender_importance`).
|
||
|
||
### Settings, flags, audit
|
||
- **`user_settings`** — 1:1 with `users`.
|
||
`user_id (pk) · theme (enum: system|light|dark) · inbox_layout (jsonb) · notifications (jsonb) · ai_prefs (jsonb) · provider_prefs (jsonb) · updated_at`
|
||
- **`system_settings`** — singleton (org-wide, admin-managed).
|
||
`id (const) · maintenance_mode (bool) · default_theme · registration_open (bool) · updated_by · updated_at` (+ arbitrary `values jsonb` for growth).
|
||
- **`feature_flags`** — the flag system (drives AI gating).
|
||
`key (pk text) · enabled (bool) · scope (enum: SystemOnly|UserOverridable) · description · rollout (jsonb, e.g. per-role) · updated_by · updated_at`
|
||
Seeded flags: `ai.enabled`, `ai.semantic_search`, `ai.ask_inbox`, `provider.google`, `provider.microsoft`, `provider.imap`, `maintenance.readonly`.
|
||
- **`audit_logs`** — admin + security events.
|
||
`id (bigserial) · actor_user_id (fk, nullable for system) · action (text) · target_type · target_id · metadata (jsonb) · ip · created_at`
|
||
Append-only; indexed on `(created_at)`, `(actor_user_id)`, `(action)`.
|
||
|
||
## Indexing & performance
|
||
| Index | Column(s) | For |
|
||
|-------|-----------|-----|
|
||
| GIN | `email_messages.search_vector` | FTS |
|
||
| GIN `pg_trgm` | sender/subject | fuzzy (fixes non-sargable `.Contains`) |
|
||
| HNSW | `email_messages.embedding` | semantic k-NN |
|
||
| btree | `(user_id, sent_at_utc desc)`, `(account_id, sent_at_utc desc)`, `(thread_id)` | keyset pagination, scoping, threading |
|
||
| unique | `(account_id, provider_message_id)`, `(provider, provider_account_id)` | idempotency, login resolution |
|
||
|
||
## Scalability (millions of emails, multi-account, incremental)
|
||
- **Multi-account** = first-class via `accounts`; every email carries `account_id` + denormalised `user_id` (so cross-account per-user search is a single indexed scan).
|
||
- **Millions of rows:** keyset (cursor) pagination, top-N-by-score ranking, `pg_trgm`/GIN/HNSW indexes. If a single user exceeds ~1–2M messages, **list-partition `email_messages` by `account_id`** (or hash by `user_id`).
|
||
- **Incremental sync:** per-account `sync_state` cursor (`historyId` / `deltaLink` / `uidvalidity+modseq`) → only deltas fetched; idempotent upserts on the unique key.
|
||
- **Idempotency & resumability:** all sync writes keyed on `(account_id, provider_message_id)`; cursor advanced atomically with the batch.
|
||
|
||
## RBAC in the schema
|
||
Small-team model = a `role` column on `users` (`Admin|Member`) — no separate roles/permissions tables yet. **Extensible** to `roles`/`permissions`/`org_id` if this ever grows to multi-tenant, without reshaping the email tables.
|
||
|
||
## Migration from today (see full [Migration Guide](12-migration-guide.md))
|
||
1. Create `users` from existing OAuth identity; set first user = **Admin**.
|
||
2. Create one `Google` **`account`** per existing user; move current Gmail tokens → `provider_tokens`.
|
||
3. Backfill `email_messages.account_id`/`user_id`, rename/extend from the current `Email` table; widen `search_vector`; add nullable `embedding`.
|
||
4. Add `user_settings`, `system_settings`, `feature_flags` (seed `ai.enabled` from the current `Ai:Mode`), `audit_logs`.
|
||
5. All additive + backfill; no destructive step — safe to run behind a maintenance window.
|