cesnimda
5c5a572cfc
Adds the two issues found and fixed during the release-candidate audit to release-candidate-review.md: the follow-up reminder index that never created on MariaDB (fix in the preceding commit), and the nondeterministic timeline day-grouping test. Corrects database-ownership.md drift: the MariaDB startup scenarios now report 42 tables (measured in every scenario this audit), not the stale 40 from before the last Phase 5 tables were added, and adds the partially-migrated heal scenario (35 -> 42) that was verified. All claims reflect behaviour verified in this audit: 420 tests on Windows and Linux in both ICU modes, all three Docker images built, four database startup scenarios against live MariaDB 11 and SQLite, and a full backup -> restore -> app-start cycle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
162 lines
7.5 KiB
Markdown
162 lines
7.5 KiB
Markdown
# Database ownership and startup order
|
|
|
|
> 2026-07-19. Which component creates which table, in what order, and why a clean MariaDB install used
|
|
> to fail. Read this before adding a table or touching `StartupInitializationExtensions`.
|
|
|
|
## The problem this document exists to prevent
|
|
|
|
EF Core bakes **provider-specific type names into a migration at scaffold time**. Every migration in
|
|
this repo was scaffolded against SQLite, so run against MariaDB it emits:
|
|
|
|
- `TEXT` for `DateTimeOffset` and every unbounded string
|
|
- `INTEGER` for `bool` and `int`
|
|
- a `PRIMARY KEY` with **no** `AUTO_INCREMENT`
|
|
|
|
A composite index over one of those `TEXT`/`longtext` columns then exceeds MySQL's 3072-byte key
|
|
limit and startup dies with `Specified key was too long`. This is not theoretical — it crashed
|
|
production once (Phase 4 `CvVariants`) and made every clean MariaDB install fail until 2026-07-19.
|
|
|
|
**Rule: a table whose migration was scaffolded against SQLite must not be created by that migration
|
|
on MariaDB.** Empty the migration and give the table to the reconciler, which carries correct DDL per
|
|
provider.
|
|
|
|
## Startup order
|
|
|
|
`InitializeJobTrackerAsync` runs exactly this sequence:
|
|
|
|
```
|
|
1. Connect
|
|
2. ReconcileSchema() ← pass 1: repair existing schema, create reconciler-owned tables
|
|
3. Database.Migrate() ← create every migration-owned table
|
|
4. ReconcileSchema() ← pass 2: everything pass 1 had to skip
|
|
5. Seed admin, start services
|
|
```
|
|
|
|
### Why the reconciler runs twice
|
|
|
|
Neither position alone works:
|
|
|
|
- **Pass 1 must come first.** A legacy database has hand-added columns and Identity tables that
|
|
predate the migrations; without repairing them (and stamping the legacy migration id into
|
|
`__EFMigrationsHistory`) `Migrate()` collides with them. `AddCareerProfileRelationalChildren` also
|
|
adds children that reference `CareerProfiles`, a **reconciler-owned** table — so it must exist
|
|
before migrations run.
|
|
- **Pass 2 must come after.** On a brand-new database the migration-owned tables do not exist during
|
|
pass 1, so every reconciler table that references one (FK into `JobApplications`) is skipped, as
|
|
are the index and `AUTO_INCREMENT` repairs.
|
|
|
|
Every statement in `ReconcileSchema` is existence-guarded, so the second pass is a no-op scan on an
|
|
already-correct database. Two consequences worth knowing:
|
|
|
|
- The `DbConnection` is **not** wrapped in `using` — it belongs to the `DbContext`, and disposing it
|
|
in pass 1 made pass 2 throw `ObjectDisposedException`.
|
|
- `conn.Open()` is guarded on `ConnectionState`, because pass 2 may inherit an open connection.
|
|
|
|
## Ownership
|
|
|
|
### Migration-owned
|
|
|
|
Created by EF migrations, never by the reconciler:
|
|
|
|
`Companies`, `JobApplications`, `Jobs`, `Correspondences`, `Attachments`, `JobEvents`,
|
|
`RuleSettings`, and the ASP.NET Identity tables.
|
|
|
|
The reconciler may **repair** these (add a missing column, add an index, fix a non-`AUTO_INCREMENT`
|
|
primary key) and may seed the default `RuleSettings` row — but it must never `CREATE TABLE` them.
|
|
It used to create `RuleSettings`, which is precisely why a clean install failed with
|
|
`Table 'RuleSettings' already exists` once `Migrate()` reached the initial migration.
|
|
|
|
### Reconciler-owned
|
|
|
|
Created by `StartupInitializationExtensions`, with a **no-op migration** holding the model snapshot:
|
|
|
|
`UserRuleSettings`, `SystemEmailSettings`, `CvUploadArtifacts`, `CvExtractionRuns`,
|
|
`GmailConnections`, `MicrosoftGraphConnections`, `ImapConnections`, `TailoredCvDrafts`,
|
|
`CareerProfiles`, `CareerProfileVersions`, the six CareerProfile children (`CareerExperiences`,
|
|
`CareerEducations`, `CareerSkills`, `CareerProjects`, `CareerCertifications`, `CareerLanguages`),
|
|
`InterviewPrepNotes`, `AiWorkspaceNotes`, `CvVariants`, `CvVariantVersions`, `AiInteractions`,
|
|
`ApplicationChecklistItems`, `CoverLetterVersions`, `InterviewPrepItems`, `TwoFactorRecoveryCodes`,
|
|
`TrustedDevices`, `UserSessions`.
|
|
|
|
No-op migrations, each with a comment explaining why:
|
|
|
|
| Migration | Tables |
|
|
|---|---|
|
|
| `20260717222917_AddCareerProfileRelationalChildren` | the six CareerProfile children |
|
|
| `20260718074509_AddCvVariants` | `CvVariants`, `CvVariantVersions` |
|
|
| `20260718131138_AddAiInteractions` | `AiInteractions` |
|
|
| `20260719085904_AddApplicationChecklistItems` | `ApplicationChecklistItems` |
|
|
| `20260719094728_SyncCareerChildKeyLengths` | snapshot sync only |
|
|
| `20260719120954_AddCoverLetterVersions` | `CoverLetterVersions` |
|
|
| `20260719145044_AddInterviewPrepItems` | `InterviewPrepItems` |
|
|
|
|
### Dependency guards
|
|
|
|
A reconciler table that references another table is guarded on **its parent existing**, so pass 1
|
|
skips it on a fresh database and pass 2 creates it:
|
|
|
|
| Table | Waits for |
|
|
|---|---|
|
|
| `TailoredCvDrafts`, `InterviewPrepNotes`, `AiWorkspaceNotes`, `CvVariants`, `AiInteractions`, `ApplicationChecklistItems`, `CoverLetterVersions`, `InterviewPrepItems` | `JobApplications` (migration-owned) |
|
|
| `CvVariantVersions` | `CvVariants` |
|
|
| `CareerProfileVersions`, the six CareerProfile children | `CareerProfiles` |
|
|
| `CvExtractionRuns` | `CvUploadArtifacts` |
|
|
|
|
Index creation goes through one helper, `EnsureMySqlIndex`, which is guarded on **table** existence
|
|
as well as index existence — repairing an absent table is not pass 1's job.
|
|
|
|
## Adding a new table
|
|
|
|
1. Add the entity and its `DbSet`, and **bound every indexed string** with `HasMaxLength` — an
|
|
unbounded string becomes `longtext`, which MariaDB cannot index without a prefix length. This is
|
|
what broke the CareerProfile children.
|
|
2. `dotnet ef migrations add …`, then **empty the `Up`/`Down`** and say why in a comment.
|
|
3. Add SQLite DDL (`CREATE TABLE IF NOT EXISTS`) and MySQL DDL (`int AUTO_INCREMENT`, `varchar(n)`,
|
|
`datetime(6)`, `tinyint(1)`) to the reconciler. Guard the MySQL create on any parent table.
|
|
4. Create indexes via `EnsureMySqlIndex` / `CREATE INDEX IF NOT EXISTS`.
|
|
5. Verify on a real MariaDB container — see below. EF InMemory will not catch any of this.
|
|
|
|
## Fresh install
|
|
|
|
No manual database preparation. Point the app at an **empty** database and start it:
|
|
|
|
```bash
|
|
# MariaDB
|
|
Database__Provider=mysql \
|
|
ConnectionStrings__JobTracker="Server=…;Database=jobtracker;User=…;Password=…;" \
|
|
dotnet run --project JobTrackerApi/JobTrackerApi.csproj
|
|
|
|
# SQLite (default) — creates Data__Root/jobtracker.db
|
|
dotnet run --project JobTrackerApi/JobTrackerApi.csproj
|
|
```
|
|
|
|
Create the empty schema/database itself (`CREATE DATABASE jobtracker;`); the application builds
|
|
everything inside it.
|
|
|
|
## Production upgrade
|
|
|
|
Deploy and restart. The reconciler is idempotent and additive:
|
|
|
|
- it never drops a table that holds rows (`DropMalformedMySqlTable` checks the row count first)
|
|
- it only adds missing columns, tables and indexes
|
|
- migrations already recorded in `__EFMigrationsHistory` are not re-run, so emptying a migration's
|
|
`Up` changes nothing for an existing database
|
|
|
|
No downtime step, no manual SQL, no data migration.
|
|
|
|
## Verified
|
|
|
|
All four scenarios, 2026-07-19, against MariaDB 11 and SQLite:
|
|
|
|
| Scenario | Result |
|
|
|---|---|
|
|
| Empty MariaDB | 42 tables created, app starts |
|
|
| Populated MariaDB, restart | idempotent — still 42 tables, rows preserved |
|
|
| Partially-migrated MariaDB (Phase 4/5 tables dropped) | healed 35 → 42, surviving rows preserved |
|
|
| Empty MariaDB via the Docker image | 42 tables created, app starts |
|
|
| Fresh SQLite | 42 tables created, app starts |
|
|
| Existing partially-migrated SQLite dev DB (34 tables) | upgraded to 44 tables, 13 applications and 8 companies preserved |
|
|
|
|
Column types on MariaDB spot-checked: `int AUTO_INCREMENT` primary keys, `varchar(255)` owner keys,
|
|
`datetime(6)` timestamps, `tinyint(1)` booleans, and every composite index inside the key limit.
|