jobtrackingapp/docs/infrastructure/database-ownership.md

Database ownership and startup order

Updated 2026-08-30. Which component creates which table, in what order, and why a clean MariaDB install used to fail. Read this before adding a table or touching StartupInitializationExtensions.

The problem this document exists to prevent

EF Core bakes provider-specific type names into a migration at scaffold time. Every migration in this repo was scaffolded against SQLite, so run against MariaDB it emits:

• TEXT for DateTimeOffset and every unbounded string
• INTEGER for bool and int
• a PRIMARY KEY with no AUTO_INCREMENT

A composite index over one of those TEXT/longtext columns then exceeds MySQL's 3072-byte key limit and startup dies with Specified key was too long. This is not theoretical — it crashed production once (Phase 4 CvVariants) and made every clean MariaDB install fail until 2026-07-19.

That historical workaround created two schema owners and is now being retired under JT-019. New tables belong to migrations. When the generated operation is not provider-safe, hand-author a provider-aware migration (using ActiveProvider) instead of adding more startup DDL. The reconciler is compatibility code for tables and columns that already shipped under its ownership.

Startup order

InitializeJobTrackerAsync runs this provider-aware sequence:

1. Connect
2. ReconcileSchema()      ← repair legacy schema/create prerequisites
3a. SQLite: apply one migration, reconcile, repeat
3b. MariaDB: apply the complete migration chain
4. ReconcileSchema()      ← create/repair everything skipped before migrations
5. Seed admin, start services

Why migration sequencing differs by provider

Neither a single reconciliation position nor one shared provider sequence works:

• Pass 1 must come first. A legacy database has hand-added columns and Identity tables that predate the migrations; without repairing them (and stamping the legacy migration id into __EFMigrationsHistory) Migrate() collides with them. AddCareerProfileRelationalChildren also adds children that reference CareerProfiles, a reconciler-owned table — so it must exist before migrations run.
• The final pass must come after. On a brand-new database the migration-owned tables do not exist during pass 1, so every reconciler table that references one (FK into JobApplications) is skipped, as are the index and AUTO_INCREMENT repairs.
• SQLite reconciles between migrations. Historical SQLite table rebuilds read the current model shape, including columns that were originally supplied by reconciliation. The per-migration pass establishes that shape before a later rebuild reads it.
• MariaDB does not reconcile between migrations. Its ALTER operations do not use SQLite table rebuilds, and an intermediate pass could create a later migration's column early and cause a duplicate-column failure. It applies the chain first and uses the shared final repair pass.

Every statement in ReconcileSchema is existence-guarded, so the second pass is a no-op scan on an already-correct database. Two consequences worth knowing:

• The DbConnection is not wrapped in using — it belongs to the DbContext, and disposing it in pass 1 made pass 2 throw ObjectDisposedException.
• conn.Open() is guarded on ConnectionState, because pass 2 may inherit an open connection.

Ownership

Migration-owned

Created by EF migrations, never by the reconciler:

AccountDeletionFiles, AccountDeletionRequests, AiUsageRecords, AiWorkspaceNotes, Attachments, Companies, Correspondences, CvExtractionRuns, CvUploadArtifacts, EmailDrafts, EmailSendAttempts, GmailConnections, GmailReviewDecisions, ImapConnections, InterviewPrepNotes, JobApplications, JobEvents, Jobs, MicrosoftGraphConnections, RuleSettings, SystemEmailSettings, TailoredCvDrafts, TrustedDevices, TwoFactorRecoveryCodes, UserNotifications, UserOperations, UserRuleSettings, and UserSessions.

SystemEmailSettings is the first completed ownership transfer: migration 20260830120000_AdoptSystemEmailSettingsSchema creates it for both supported providers and preserves an existing reconciler-created MariaDB table. Startup no longer creates it.

UserRuleSettings followed in 20260830121000_AdoptUserRuleSettingsSchema; its owner-keyed rows are preserved across adoption, downgrade and retry, and startup no longer creates it.

GmailReviewDecisions moved in 20260830122000_AdoptGmailReviewDecisionsSchema. This also supplies the table on MariaDB, where the old reconciler had no creation path.

The authentication support group (TwoFactorRecoveryCodes, TrustedDevices, and UserSessions) moved in 20260830123000_AdoptAuthenticationSupportSchema. These tables deliberately have no database foreign key to AspNetUsers because they are queried during authentication before a current-user scope exists.

The email-provider connection group (GmailConnections, MicrosoftGraphConnections, and ImapConnections) moved in 20260830124000_AdoptEmailConnectionSchema. Startup retains only additive Gmail sync-column repair plus guarded MariaDB identity/index repair; it does not create these tables.

The CV import persistence group (CvUploadArtifacts and CvExtractionRuns) moved in 20260830125000_AdoptCvExtractionSchema. The extraction run's optional artifact foreign key remains ON DELETE SET NULL, so retention cleanup cannot erase the extraction/review record.

TailoredCvDrafts moved in 20260830126000_AdoptTailoredCvDraftSchema; its generated and edited content is retained during adoption while the one-to-one job-application relationship remains ON DELETE CASCADE.

The persisted job-workspace note group (InterviewPrepNotes and AiWorkspaceNotes) moved in 20260830127000_AdoptJobWorkspaceNotesSchema. Both caches retain their unique owner/job keys and remain application-owned through ON DELETE CASCADE.

The reconciler may repair these (add a missing column, add an index, fix a non-AUTO_INCREMENT primary key) and may seed the default RuleSettings row — but it must never CREATE TABLE them. It used to create RuleSettings, which is precisely why a clean install failed with Table 'RuleSettings' already exists once Migrate() reached the initial migration.

Reconciler-owned

Created by StartupInitializationExtensions, with a no-op migration holding the model snapshot:

CareerProfiles, CareerProfileVersions, the six CareerProfile children (CareerExperiences, CareerEducations, CareerSkills, CareerProjects, CareerCertifications, CareerLanguages), CvVariants, CvVariantVersions, AiInteractions, ApplicationChecklistItems, CoverLetterVersions, and InterviewPrepItems.

The seven ASP.NET Identity tables are also currently reconciler-owned, despite older wording that called them migration-owned: AspNetRoles, AspNetUsers, AspNetRoleClaims, AspNetUserClaims, AspNetUserLogins, AspNetUserRoles, and AspNetUserTokens. Guarded migration bootstraps for AspNetUsers and AiInteractions support standalone traversal but do not yet transfer ownership.

StartupSchemaOwnership is the executable inventory. Its tests require every EF model table to have exactly one creation owner and keep compatibility bootstraps out of the migration-owned set.

No-op migrations, each with a comment explaining why:

Migration	Tables
20260717222917_AddCareerProfileRelationalChildren	the six CareerProfile children
20260718074509_AddCvVariants	CvVariants, CvVariantVersions
20260718131138_AddAiInteractions	AiInteractions
20260719085904_AddApplicationChecklistItems	ApplicationChecklistItems
20260719094728_SyncCareerChildKeyLengths	snapshot sync only
20260719120954_AddCoverLetterVersions	CoverLetterVersions
20260719145044_AddInterviewPrepItems	InterviewPrepItems

Dependency guards

A reconciler table that references another table is guarded on its parent existing, so pass 1 skips it on a fresh database and pass 2 creates it:

Table	Waits for
TailoredCvDrafts, InterviewPrepNotes, AiWorkspaceNotes, CvVariants, AiInteractions, ApplicationChecklistItems, CoverLetterVersions, InterviewPrepItems	JobApplications (migration-owned)
CvVariantVersions	CvVariants
CareerProfileVersions, the six CareerProfile children	CareerProfiles
CvExtractionRuns	CvUploadArtifacts

Index creation goes through one helper, EnsureMySqlIndex, which is guarded on table existence as well as index existence — repairing an absent table is not pass 1's job.

Adding a new table

1. Add the entity and its DbSet, and bound every indexed string with HasMaxLength — an unbounded string becomes longtext, which MariaDB cannot index without a prefix length. This is what broke the CareerProfile children.
2. Add an EF migration. Review its generated SQL for both SQLite and MariaDB; do not assume a SQLite-scaffolded type is valid on MariaDB.
3. If needed, replace the generated operation with provider-aware migration SQL (int AUTO_INCREMENT, bounded varchar(n), datetime(6), tinyint(1) on MariaDB). Do not add the table to StartupInitializationExtensions.
4. Classify the table in StartupSchemaOwnership.MigrationOwnedTables; the ownership test will fail if it is omitted or duplicated.
5. Verify blank, populated-upgrade and restart paths on SQLite and a real MariaDB container. EF InMemory will not catch provider DDL defects.

Retiring a reconciler-owned table

Move one leaf/dependency group at a time:

1. Add an idempotent provider-aware migration that creates the legacy shape when absent.
2. Make downgrade preserve a table that may predate migration ownership; never drop ambiguous data.
3. Remove only that table's startup CREATE TABLE block and move it between the executable ownership sets.
4. Prove a blank database, a pre-migration database with a representative row, migration retry, and provider SQL/runtime behaviour.
5. Leave column/index repairs in place until historical upgrade fixtures prove they are redundant; table creation and legacy repair are separate ownership decisions.

Fresh install

No manual database preparation. Point the app at an empty database and start it:

# MariaDB
Database__Provider=mysql \
ConnectionStrings__JobTracker="Server=…;Database=jobtracker;User=…;Password=…;" \
dotnet run --project JobTrackerApi/JobTrackerApi.csproj

# SQLite (default) — creates Data__Root/jobtracker.db
dotnet run --project JobTrackerApi/JobTrackerApi.csproj

Create the empty schema/database itself (CREATE DATABASE jobtracker;); the application builds everything inside it.

Standalone EF tooling is also supported for a blank SQLite database. The historical initial migration now supplies the stable JobApplication columns required by later SQLite rebuilds, and guarded compatibility bootstraps provide the reconciler-owned source tables used by later additive migrations. Application startup may subsequently reconcile the remaining Identity and auxiliary tables without losing rows.

Production upgrade

Deploy and restart. The reconciler is idempotent and additive:

• it never drops a table that holds rows (DropMalformedMySqlTable checks the row count first)
• it only adds missing columns, tables and indexes
• migrations already recorded in __EFMigrationsHistory are not re-run, so emptying a migration's Up changes nothing for an existing database

No downtime step, no manual SQL, no data migration.

Verified

All four scenarios, 2026-07-19, against MariaDB 11 and SQLite:

Scenario	Result
Empty MariaDB	42 tables created, app starts
Populated MariaDB, restart	idempotent — still 42 tables, rows preserved
Partially-migrated MariaDB (Phase 4/5 tables dropped)	healed 35 → 42, surviving rows preserved
Empty MariaDB via the Docker image	42 tables created, app starts
Fresh SQLite	42 tables created, app starts
Existing partially-migrated SQLite dev DB (34 tables)	upgraded to 44 tables, 13 applications and 8 companies preserved

Column types on MariaDB spot-checked: int AUTO_INCREMENT primary keys, varchar(255) owner keys, datetime(6) timestamps, tinyint(1) booleans, and every composite index inside the key limit.

On 2026-08-15 the current 29-migration chain was additionally verified against a blank standalone SQLite database, an older populated SQLite checkpoint, and a disposable MariaDB 11.8 database. Standalone SQLite migration and retry both reached the latest migration; populated title/date and reconciler-owned owner/summary data survived. Starting the application over that EF-only database served /health successfully. Fresh MariaDB startup and restart both served /health with 29 migrations and 49 tables; provider-sensitive ID, owner, decimal and timestamp column types were spot-checked. No production database was changed.