jobtrackingapp/docs/infrastructure/database-ownership.md

# Database ownership and startup order

> Updated 2026-08-15. Which component creates which table, in what order, and why a clean MariaDB install used
> to fail. Read this before adding a table or touching `StartupInitializationExtensions`.

## The problem this document exists to prevent

EF Core bakes **provider-specific type names into a migration at scaffold time**. Every migration in
this repo was scaffolded against SQLite, so run against MariaDB it emits:

- `TEXT` for `DateTimeOffset` and every unbounded string
- `INTEGER` for `bool` and `int`
- a `PRIMARY KEY` with **no** `AUTO_INCREMENT`

A composite index over one of those `TEXT`/`longtext` columns then exceeds MySQL's 3072-byte key
limit and startup dies with `Specified key was too long`. This is not theoretical — it crashed
production once (Phase 4 `CvVariants`) and made every clean MariaDB install fail until 2026-07-19.

**Rule: a table whose migration was scaffolded against SQLite must not be created by that migration
on MariaDB.** Empty the migration and give the table to the reconciler, which carries correct DDL per
provider.

## Startup order

`InitializeJobTrackerAsync` runs this provider-aware sequence:

```
1. Connect
2. ReconcileSchema()      ← repair legacy schema/create prerequisites
3a. SQLite: apply one migration, reconcile, repeat
3b. MariaDB: apply the complete migration chain
4. ReconcileSchema()      ← create/repair everything skipped before migrations
5. Seed admin, start services
```

### Why migration sequencing differs by provider

Neither a single reconciliation position nor one shared provider sequence works:

- **Pass 1 must come first.** A legacy database has hand-added columns and Identity tables that
  predate the migrations; without repairing them (and stamping the legacy migration id into
  `__EFMigrationsHistory`) `Migrate()` collides with them. `AddCareerProfileRelationalChildren` also
  adds children that reference `CareerProfiles`, a **reconciler-owned** table — so it must exist
  before migrations run.
- **The final pass must come after.** On a brand-new database the migration-owned tables do not exist during
  pass 1, so every reconciler table that references one (FK into `JobApplications`) is skipped, as
  are the index and `AUTO_INCREMENT` repairs.
- **SQLite reconciles between migrations.** Historical SQLite table rebuilds read the current model
  shape, including columns that were originally supplied by reconciliation. The per-migration pass
  establishes that shape before a later rebuild reads it.
- **MariaDB does not reconcile between migrations.** Its ALTER operations do not use SQLite table
  rebuilds, and an intermediate pass could create a later migration's column early and cause a
  duplicate-column failure. It applies the chain first and uses the shared final repair pass.

Every statement in `ReconcileSchema` is existence-guarded, so the second pass is a no-op scan on an
already-correct database. Two consequences worth knowing:

- The `DbConnection` is **not** wrapped in `using` — it belongs to the `DbContext`, and disposing it
  in pass 1 made pass 2 throw `ObjectDisposedException`.
- `conn.Open()` is guarded on `ConnectionState`, because pass 2 may inherit an open connection.

## Ownership

### Migration-owned

Created by EF migrations, never by the reconciler:

`Companies`, `JobApplications`, `Jobs`, `Correspondences`, `Attachments`, `JobEvents`,
`RuleSettings`, and the ASP.NET Identity tables. Two compatibility migrations use guarded
`CREATE TABLE IF NOT EXISTS` bootstraps for `AspNetUsers` and `AiInteractions` so standalone EF
tooling can traverse the historical chain; normal application startup makes those statements no-ops.

The reconciler may **repair** these (add a missing column, add an index, fix a non-`AUTO_INCREMENT`
primary key) and may seed the default `RuleSettings` row — but it must never `CREATE TABLE` them.
It used to create `RuleSettings`, which is precisely why a clean install failed with
`Table 'RuleSettings' already exists` once `Migrate()` reached the initial migration.

### Reconciler-owned

Created by `StartupInitializationExtensions`, with a **no-op migration** holding the model snapshot:

`UserRuleSettings`, `SystemEmailSettings`, `CvUploadArtifacts`, `CvExtractionRuns`,
`GmailConnections`, `MicrosoftGraphConnections`, `ImapConnections`, `TailoredCvDrafts`,
`CareerProfiles`, `CareerProfileVersions`, the six CareerProfile children (`CareerExperiences`,
`CareerEducations`, `CareerSkills`, `CareerProjects`, `CareerCertifications`, `CareerLanguages`),
`InterviewPrepNotes`, `AiWorkspaceNotes`, `CvVariants`, `CvVariantVersions`, `AiInteractions`,
`ApplicationChecklistItems`, `CoverLetterVersions`, `InterviewPrepItems`, `TwoFactorRecoveryCodes`,
`TrustedDevices`, `UserSessions`.

No-op migrations, each with a comment explaining why:

| Migration | Tables |
|---|---|
| `20260717222917_AddCareerProfileRelationalChildren` | the six CareerProfile children |
| `20260718074509_AddCvVariants` | `CvVariants`, `CvVariantVersions` |
| `20260718131138_AddAiInteractions` | `AiInteractions` |
| `20260719085904_AddApplicationChecklistItems` | `ApplicationChecklistItems` |
| `20260719094728_SyncCareerChildKeyLengths` | snapshot sync only |
| `20260719120954_AddCoverLetterVersions` | `CoverLetterVersions` |
| `20260719145044_AddInterviewPrepItems` | `InterviewPrepItems` |

### Dependency guards

A reconciler table that references another table is guarded on **its parent existing**, so pass 1
skips it on a fresh database and pass 2 creates it:

| Table | Waits for |
|---|---|
| `TailoredCvDrafts`, `InterviewPrepNotes`, `AiWorkspaceNotes`, `CvVariants`, `AiInteractions`, `ApplicationChecklistItems`, `CoverLetterVersions`, `InterviewPrepItems` | `JobApplications` (migration-owned) |
| `CvVariantVersions` | `CvVariants` |
| `CareerProfileVersions`, the six CareerProfile children | `CareerProfiles` |
| `CvExtractionRuns` | `CvUploadArtifacts` |

Index creation goes through one helper, `EnsureMySqlIndex`, which is guarded on **table** existence
as well as index existence — repairing an absent table is not pass 1's job.

## Adding a new table

1. Add the entity and its `DbSet`, and **bound every indexed string** with `HasMaxLength` — an
   unbounded string becomes `longtext`, which MariaDB cannot index without a prefix length. This is
   what broke the CareerProfile children.
2. `dotnet ef migrations add …`, then **empty the `Up`/`Down`** and say why in a comment.
3. Add SQLite DDL (`CREATE TABLE IF NOT EXISTS`) and MySQL DDL (`int AUTO_INCREMENT`, `varchar(n)`,
   `datetime(6)`, `tinyint(1)`) to the reconciler. Guard the MySQL create on any parent table.
4. Create indexes via `EnsureMySqlIndex` / `CREATE INDEX IF NOT EXISTS`.
5. Verify on a real MariaDB container — see below. EF InMemory will not catch any of this.

## Fresh install

No manual database preparation. Point the app at an **empty** database and start it:

```bash
# MariaDB
Database__Provider=mysql \
ConnectionStrings__JobTracker="Server=…;Database=jobtracker;User=…;Password=…;" \
dotnet run --project JobTrackerApi/JobTrackerApi.csproj

# SQLite (default) — creates Data__Root/jobtracker.db
dotnet run --project JobTrackerApi/JobTrackerApi.csproj
```

Create the empty schema/database itself (`CREATE DATABASE jobtracker;`); the application builds
everything inside it.

Standalone EF tooling is also supported for a blank SQLite database. The historical initial
migration now supplies the stable JobApplication columns required by later SQLite rebuilds, and
guarded compatibility bootstraps provide the reconciler-owned source tables used by later additive
migrations. Application startup may subsequently reconcile the remaining Identity and auxiliary
tables without losing rows.

## Production upgrade

Deploy and restart. The reconciler is idempotent and additive:

- it never drops a table that holds rows (`DropMalformedMySqlTable` checks the row count first)
- it only adds missing columns, tables and indexes
- migrations already recorded in `__EFMigrationsHistory` are not re-run, so emptying a migration's
  `Up` changes nothing for an existing database

No downtime step, no manual SQL, no data migration.

## Verified

All four scenarios, 2026-07-19, against MariaDB 11 and SQLite:

| Scenario | Result |
|---|---|
| Empty MariaDB | 42 tables created, app starts |
| Populated MariaDB, restart | idempotent — still 42 tables, rows preserved |
| Partially-migrated MariaDB (Phase 4/5 tables dropped) | healed 35 → 42, surviving rows preserved |
| Empty MariaDB via the Docker image | 42 tables created, app starts |
| Fresh SQLite | 42 tables created, app starts |
| Existing partially-migrated SQLite dev DB (34 tables) | upgraded to 44 tables, 13 applications and 8 companies preserved |

Column types on MariaDB spot-checked: `int AUTO_INCREMENT` primary keys, `varchar(255)` owner keys,
`datetime(6)` timestamps, `tinyint(1)` booleans, and every composite index inside the key limit.

On 2026-08-15 the current 29-migration chain was additionally verified against a blank standalone
SQLite database, an older populated SQLite checkpoint, and a disposable MariaDB 11.8 database.
Standalone SQLite migration and retry both reached the latest migration; populated title/date and
reconciler-owned owner/summary data survived. Starting the application over that EF-only database
served `/health` successfully. Fresh MariaDB startup and restart both served `/health` with 29
migrations and 49 tables; provider-sensitive ID, owner, decimal and timestamp column types were
spot-checked. No production database was changed.