cesnimda
e933409698
Move workflow checklist state into an additive provider-aware migration. Preserve system-key idempotency, manual progress, ordering, and application cascade semantics.
256 lines
14 KiB
Markdown
256 lines
14 KiB
Markdown
# Database ownership and startup order
|
|
|
|
> Updated 2026-08-30. Which component creates which table, in what order, and why a clean MariaDB install used
|
|
> to fail. Read this before adding a table or touching `StartupInitializationExtensions`.
|
|
|
|
## The problem this document exists to prevent
|
|
|
|
EF Core bakes **provider-specific type names into a migration at scaffold time**. Every migration in
|
|
this repo was scaffolded against SQLite, so run against MariaDB it emits:
|
|
|
|
- `TEXT` for `DateTimeOffset` and every unbounded string
|
|
- `INTEGER` for `bool` and `int`
|
|
- a `PRIMARY KEY` with **no** `AUTO_INCREMENT`
|
|
|
|
A composite index over one of those `TEXT`/`longtext` columns then exceeds MySQL's 3072-byte key
|
|
limit and startup dies with `Specified key was too long`. This is not theoretical — it crashed
|
|
production once (Phase 4 `CvVariants`) and made every clean MariaDB install fail until 2026-07-19.
|
|
|
|
That historical workaround created two schema owners and is now being retired under JT-019. New
|
|
tables belong to migrations. When the generated operation is not provider-safe, hand-author a
|
|
provider-aware migration (using `ActiveProvider`) instead of adding more startup DDL. The reconciler
|
|
is compatibility code for tables and columns that already shipped under its ownership.
|
|
|
|
## Startup order
|
|
|
|
`InitializeJobTrackerAsync` runs this provider-aware sequence:
|
|
|
|
```
|
|
1. Connect
|
|
2. ReconcileSchema() ← repair legacy schema/create prerequisites
|
|
3a. SQLite: apply one migration, reconcile, repeat
|
|
3b. MariaDB: apply the complete migration chain
|
|
4. ReconcileSchema() ← create/repair everything skipped before migrations
|
|
5. Seed admin, start services
|
|
```
|
|
|
|
### Why migration sequencing differs by provider
|
|
|
|
Neither a single reconciliation position nor one shared provider sequence works:
|
|
|
|
- **Pass 1 must come first.** A legacy database has hand-added columns and Identity tables that
|
|
predate the migrations; without repairing them (and stamping the legacy migration id into
|
|
`__EFMigrationsHistory`) `Migrate()` collides with them. `AddCareerProfileRelationalChildren` also
|
|
adds children that reference `CareerProfiles`, a **reconciler-owned** table — so it must exist
|
|
before migrations run.
|
|
- **The final pass must come after.** On a brand-new database the migration-owned tables do not exist during
|
|
pass 1, so every reconciler table that references one (FK into `JobApplications`) is skipped, as
|
|
are the index and `AUTO_INCREMENT` repairs.
|
|
- **SQLite reconciles between migrations.** Historical SQLite table rebuilds read the current model
|
|
shape, including columns that were originally supplied by reconciliation. The per-migration pass
|
|
establishes that shape before a later rebuild reads it.
|
|
- **MariaDB does not reconcile between migrations.** Its ALTER operations do not use SQLite table
|
|
rebuilds, and an intermediate pass could create a later migration's column early and cause a
|
|
duplicate-column failure. It applies the chain first and uses the shared final repair pass.
|
|
|
|
Every statement in `ReconcileSchema` is existence-guarded, so the second pass is a no-op scan on an
|
|
already-correct database. Two consequences worth knowing:
|
|
|
|
- The `DbConnection` is **not** wrapped in `using` — it belongs to the `DbContext`, and disposing it
|
|
in pass 1 made pass 2 throw `ObjectDisposedException`.
|
|
- `conn.Open()` is guarded on `ConnectionState`, because pass 2 may inherit an open connection.
|
|
|
|
## Ownership
|
|
|
|
### Migration-owned
|
|
|
|
Created by EF migrations, never by the reconciler:
|
|
|
|
`AccountDeletionFiles`, `AccountDeletionRequests`, `AiInteractions`, `AiUsageRecords`, `AiWorkspaceNotes`,
|
|
`ApplicationChecklistItems`, `Attachments`, `Companies`,
|
|
`Correspondences`, `CvExtractionRuns`, `CvUploadArtifacts`, `CvVariants`, `CvVariantVersions`,
|
|
`EmailDrafts`, `EmailSendAttempts`, `GmailConnections`,
|
|
`GmailReviewDecisions`, `ImapConnections`, `InterviewPrepNotes`, `JobApplications`, `JobEvents`, `Jobs`,
|
|
`MicrosoftGraphConnections`, `RuleSettings`, `SystemEmailSettings`, `TailoredCvDrafts`,
|
|
`TrustedDevices`, `TwoFactorRecoveryCodes`, `UserNotifications`, `UserOperations`, `UserRuleSettings`,
|
|
and `UserSessions`.
|
|
|
|
`SystemEmailSettings` is the first completed ownership transfer: migration
|
|
`20260830120000_AdoptSystemEmailSettingsSchema` creates it for both supported providers and preserves
|
|
an existing reconciler-created MariaDB table. Startup no longer creates it.
|
|
|
|
`UserRuleSettings` followed in `20260830121000_AdoptUserRuleSettingsSchema`; its owner-keyed rows are
|
|
preserved across adoption, downgrade and retry, and startup no longer creates it.
|
|
|
|
`GmailReviewDecisions` moved in `20260830122000_AdoptGmailReviewDecisionsSchema`. This also supplies
|
|
the table on MariaDB, where the old reconciler had no creation path.
|
|
|
|
The authentication support group (`TwoFactorRecoveryCodes`, `TrustedDevices`, and `UserSessions`)
|
|
moved in `20260830123000_AdoptAuthenticationSupportSchema`. These tables deliberately have no
|
|
database foreign key to `AspNetUsers` because they are queried during authentication before a
|
|
current-user scope exists.
|
|
|
|
The email-provider connection group (`GmailConnections`, `MicrosoftGraphConnections`, and
|
|
`ImapConnections`) moved in `20260830124000_AdoptEmailConnectionSchema`. Startup retains only
|
|
additive Gmail sync-column repair plus guarded MariaDB identity/index repair; it does not create
|
|
these tables.
|
|
|
|
The CV import persistence group (`CvUploadArtifacts` and `CvExtractionRuns`) moved in
|
|
`20260830125000_AdoptCvExtractionSchema`. The extraction run's optional artifact foreign key remains
|
|
`ON DELETE SET NULL`, so retention cleanup cannot erase the extraction/review record.
|
|
|
|
`TailoredCvDrafts` moved in `20260830126000_AdoptTailoredCvDraftSchema`; its generated and edited
|
|
content is retained during adoption while the one-to-one job-application relationship remains
|
|
`ON DELETE CASCADE`.
|
|
|
|
The persisted job-workspace note group (`InterviewPrepNotes` and `AiWorkspaceNotes`) moved in
|
|
`20260830127000_AdoptJobWorkspaceNotesSchema`. Both caches retain their unique owner/job keys and
|
|
remain application-owned through `ON DELETE CASCADE`.
|
|
|
|
The CV builder aggregate (`CvVariants` and `CvVariantVersions`) moved in
|
|
`20260830128000_AdoptCvVariantSchema`. Public slugs remain unique, deleting a linked application
|
|
sets the CV link to null, and deleting a CV cascades through its append-only revision history.
|
|
|
|
`AiInteractions` moved in `20260830129000_AdoptAiInteractionSchema`. The older cross-feature usage
|
|
migration retains its compatibility bootstrap for historical chain traversal, but startup no longer
|
|
creates the table; only additive counter and guarded MariaDB shape/index repairs remain.
|
|
|
|
`ApplicationChecklistItems` moved in `20260830130000_AdoptApplicationChecklistSchema`. Its stable
|
|
system-key uniqueness and owner/job/sort index preserve idempotent seeding alongside freely ordered
|
|
manual tasks; application deletion remains cascading.
|
|
|
|
The reconciler may **repair** these (add a missing column, add an index, fix a non-`AUTO_INCREMENT`
|
|
primary key) and may seed the default `RuleSettings` row — but it must never `CREATE TABLE` them.
|
|
It used to create `RuleSettings`, which is precisely why a clean install failed with
|
|
`Table 'RuleSettings' already exists` once `Migrate()` reached the initial migration.
|
|
|
|
### Reconciler-owned
|
|
|
|
Created by `StartupInitializationExtensions`, with a **no-op migration** holding the model snapshot:
|
|
|
|
`CareerProfiles`, `CareerProfileVersions`, the six CareerProfile children (`CareerExperiences`,
|
|
`CareerEducations`, `CareerSkills`, `CareerProjects`, `CareerCertifications`, `CareerLanguages`),
|
|
`CoverLetterVersions` and `InterviewPrepItems`.
|
|
|
|
The seven ASP.NET Identity tables are also currently reconciler-owned, despite older wording that
|
|
called them migration-owned: `AspNetRoles`, `AspNetUsers`, `AspNetRoleClaims`, `AspNetUserClaims`,
|
|
`AspNetUserLogins`, `AspNetUserRoles`, and `AspNetUserTokens`. Guarded migration bootstraps for
|
|
`AspNetUsers` supports standalone traversal but does not yet transfer ownership. The historical
|
|
`AiInteractions` bootstrap remains in the migration chain, but its current owner is the later
|
|
provider-aware adoption migration.
|
|
|
|
`StartupSchemaOwnership` is the executable inventory. Its tests require every EF model table to
|
|
have exactly one creation owner and keep compatibility bootstraps out of the migration-owned set.
|
|
|
|
No-op migrations, each with a comment explaining why:
|
|
|
|
| Migration | Tables |
|
|
|---|---|
|
|
| `20260717222917_AddCareerProfileRelationalChildren` | the six CareerProfile children |
|
|
| `20260718074509_AddCvVariants` | historical no-op; ownership transferred by `20260830128000_AdoptCvVariantSchema` |
|
|
| `20260718131138_AddAiInteractions` | historical no-op; ownership transferred by `20260830129000_AdoptAiInteractionSchema` |
|
|
| `20260719085904_AddApplicationChecklistItems` | historical no-op; ownership transferred by `20260830130000_AdoptApplicationChecklistSchema` |
|
|
| `20260719094728_SyncCareerChildKeyLengths` | snapshot sync only |
|
|
| `20260719120954_AddCoverLetterVersions` | `CoverLetterVersions` |
|
|
| `20260719145044_AddInterviewPrepItems` | `InterviewPrepItems` |
|
|
|
|
### Dependency guards
|
|
|
|
A reconciler table that references another table is guarded on **its parent existing**, so pass 1
|
|
skips it on a fresh database and pass 2 creates it:
|
|
|
|
| Table | Waits for |
|
|
|---|---|
|
|
| `TailoredCvDrafts`, `InterviewPrepNotes`, `AiWorkspaceNotes`, `CvVariants`, `AiInteractions`, `ApplicationChecklistItems`, `CoverLetterVersions`, `InterviewPrepItems` | `JobApplications` (migration-owned) |
|
|
| `CvVariantVersions` | `CvVariants` |
|
|
| `CareerProfileVersions`, the six CareerProfile children | `CareerProfiles` |
|
|
| `CvExtractionRuns` | `CvUploadArtifacts` |
|
|
|
|
Index creation goes through one helper, `EnsureMySqlIndex`, which is guarded on **table** existence
|
|
as well as index existence — repairing an absent table is not pass 1's job.
|
|
|
|
## Adding a new table
|
|
|
|
1. Add the entity and its `DbSet`, and **bound every indexed string** with `HasMaxLength` — an
|
|
unbounded string becomes `longtext`, which MariaDB cannot index without a prefix length. This is
|
|
what broke the CareerProfile children.
|
|
2. Add an EF migration. Review its generated SQL for both SQLite and MariaDB; do not assume a
|
|
SQLite-scaffolded type is valid on MariaDB.
|
|
3. If needed, replace the generated operation with provider-aware migration SQL (`int
|
|
AUTO_INCREMENT`, bounded `varchar(n)`, `datetime(6)`, `tinyint(1)` on MariaDB). Do not add the
|
|
table to `StartupInitializationExtensions`.
|
|
4. Classify the table in `StartupSchemaOwnership.MigrationOwnedTables`; the ownership test will fail
|
|
if it is omitted or duplicated.
|
|
5. Verify blank, populated-upgrade and restart paths on SQLite and a real MariaDB container. EF
|
|
InMemory will not catch provider DDL defects.
|
|
|
|
## Retiring a reconciler-owned table
|
|
|
|
Move one leaf/dependency group at a time:
|
|
|
|
1. Add an idempotent provider-aware migration that creates the legacy shape when absent.
|
|
2. Make downgrade preserve a table that may predate migration ownership; never drop ambiguous data.
|
|
3. Remove only that table's startup `CREATE TABLE` block and move it between the executable ownership sets.
|
|
4. Prove a blank database, a pre-migration database with a representative row, migration retry, and
|
|
provider SQL/runtime behaviour.
|
|
5. Leave column/index repairs in place until historical upgrade fixtures prove they are redundant;
|
|
table creation and legacy repair are separate ownership decisions.
|
|
|
|
## Fresh install
|
|
|
|
No manual database preparation. Point the app at an **empty** database and start it:
|
|
|
|
```bash
|
|
# MariaDB
|
|
Database__Provider=mysql \
|
|
ConnectionStrings__JobTracker="Server=…;Database=jobtracker;User=…;Password=…;" \
|
|
dotnet run --project JobTrackerApi/JobTrackerApi.csproj
|
|
|
|
# SQLite (default) — creates Data__Root/jobtracker.db
|
|
dotnet run --project JobTrackerApi/JobTrackerApi.csproj
|
|
```
|
|
|
|
Create the empty schema/database itself (`CREATE DATABASE jobtracker;`); the application builds
|
|
everything inside it.
|
|
|
|
Standalone EF tooling is also supported for a blank SQLite database. The historical initial
|
|
migration now supplies the stable JobApplication columns required by later SQLite rebuilds, and
|
|
guarded compatibility bootstraps provide the reconciler-owned source tables used by later additive
|
|
migrations. Application startup may subsequently reconcile the remaining Identity and auxiliary
|
|
tables without losing rows.
|
|
|
|
## Production upgrade
|
|
|
|
Deploy and restart. The reconciler is idempotent and additive:
|
|
|
|
- it never drops a table that holds rows (`DropMalformedMySqlTable` checks the row count first)
|
|
- it only adds missing columns, tables and indexes
|
|
- migrations already recorded in `__EFMigrationsHistory` are not re-run, so emptying a migration's
|
|
`Up` changes nothing for an existing database
|
|
|
|
No downtime step, no manual SQL, no data migration.
|
|
|
|
## Verified
|
|
|
|
All four scenarios, 2026-07-19, against MariaDB 11 and SQLite:
|
|
|
|
| Scenario | Result |
|
|
|---|---|
|
|
| Empty MariaDB | 42 tables created, app starts |
|
|
| Populated MariaDB, restart | idempotent — still 42 tables, rows preserved |
|
|
| Partially-migrated MariaDB (Phase 4/5 tables dropped) | healed 35 → 42, surviving rows preserved |
|
|
| Empty MariaDB via the Docker image | 42 tables created, app starts |
|
|
| Fresh SQLite | 42 tables created, app starts |
|
|
| Existing partially-migrated SQLite dev DB (34 tables) | upgraded to 44 tables, 13 applications and 8 companies preserved |
|
|
|
|
Column types on MariaDB spot-checked: `int AUTO_INCREMENT` primary keys, `varchar(255)` owner keys,
|
|
`datetime(6)` timestamps, `tinyint(1)` booleans, and every composite index inside the key limit.
|
|
|
|
On 2026-08-15 the current 29-migration chain was additionally verified against a blank standalone
|
|
SQLite database, an older populated SQLite checkpoint, and a disposable MariaDB 11.8 database.
|
|
Standalone SQLite migration and retry both reached the latest migration; populated title/date and
|
|
reconciler-owned owner/summary data survived. Starting the application over that EF-only database
|
|
served `/health` successfully. Fresh MariaDB startup and restart both served `/health` with 29
|
|
migrations and 49 tables; provider-sensitive ID, owner, decimal and timestamp column types were
|
|
spot-checked. No production database was changed.
|