cesnimda/jobtrackingapp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: consolidate repository guides and preserve audit history

Browse Source
This commit is contained in:
cesnimda
2026-09-11 14:22:41 +02:00
parent 115441a409
commit 682a3028bb
404 changed files with 3402 additions and 2621 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/_archive/pre-docs-cleanup-2026-09-11/docs/infrastructure/database-ownership.md
+263
View File
@@ -0,0 +1,263 @@
# Database ownership and startup order
> Updated 2026-08-30. Which component creates which table, in what order, and why a clean MariaDB install used
> to fail. Read this before adding a table or touching `StartupInitializationExtensions`.
## The problem this document exists to prevent
EF Core bakes **provider-specific type names into a migration at scaffold time**. Every migration in
this repo was scaffolded against SQLite, so run against MariaDB it emits:
- `TEXT` for `DateTimeOffset` and every unbounded string
- `INTEGER` for `bool` and `int`
- a `PRIMARY KEY` with **no** `AUTO_INCREMENT`
A composite index over one of those `TEXT`/`longtext` columns then exceeds MySQL's 3072-byte key
limit and startup dies with `Specified key was too long`. This is not theoretical — it crashed
production once (Phase 4 `CvVariants`) and made every clean MariaDB install fail until 2026-07-19.
That historical workaround created two schema owners and is now being retired under JT-019. New
tables belong to migrations. When the generated operation is not provider-safe, hand-author a
provider-aware migration (using `ActiveProvider`) instead of adding more startup DDL. The reconciler
is compatibility code for tables and columns that already shipped under its ownership.
## Startup order
`InitializeJobTrackerAsync` runs this provider-aware sequence:
```
1. Connect
2. ReconcileSchema() ← repair legacy schema/create prerequisites
3a. SQLite: apply one migration, reconcile, repeat
3b. MariaDB: apply the complete migration chain
4. ReconcileSchema() ← create/repair everything skipped before migrations
5. Seed admin, start services
```
### Why migration sequencing differs by provider
Neither a single reconciliation position nor one shared provider sequence works:
- **Pass 1 must come first.** A legacy database has hand-added columns and Identity tables that
predate the migrations; without repairing them (and stamping the legacy migration id into
`__EFMigrationsHistory`) `Migrate()` collides with them.
- **The final pass must come after.** On a brand-new database the migration-owned tables do not exist during
pass 1, so every reconciler table that references one (FK into `JobApplications`) is skipped, as
are the index and `AUTO_INCREMENT` repairs.
- **SQLite reconciles between migrations.** Historical SQLite table rebuilds read the current model
shape, including columns that were originally supplied by reconciliation. The per-migration pass
establishes that shape before a later rebuild reads it.
- **MariaDB does not reconcile between migrations.** Its ALTER operations do not use SQLite table
rebuilds, and an intermediate pass could create a later migration's column early and cause a
duplicate-column failure. It applies the chain first and uses the shared final repair pass.
Every statement in `ReconcileSchema` is existence-guarded, so the second pass is a no-op scan on an
already-correct database. Two consequences worth knowing:
- The `DbConnection` is **not** wrapped in `using` — it belongs to the `DbContext`, and disposing it
in pass 1 made pass 2 throw `ObjectDisposedException`.
- `conn.Open()` is guarded on `ConnectionState`, because pass 2 may inherit an open connection.
## Ownership
### Migration-owned
Created by EF migrations, never by the reconciler:
`AccountDeletionFiles`, `AccountDeletionRequests`, `AiInteractions`, `AiUsageRecords`, `AiWorkspaceNotes`,
`ApplicationChecklistItems`, `AspNetRoleClaims`, `AspNetRoles`, `AspNetUserClaims`, `AspNetUserLogins`,
`AspNetUserRoles`, `AspNetUsers`, `AspNetUserTokens`, `Attachments`, `CareerCertifications`, `CareerEducations`,
`CareerExperiences`, `CareerLanguages`, `CareerProfiles`, `CareerProfileVersions`, `CareerProjects`,
`CareerSkills`, `Companies`, `CoverLetterVersions`,
`Correspondences`, `CvExtractionRuns`, `CvUploadArtifacts`, `CvVariants`, `CvVariantVersions`,
`EmailDrafts`, `EmailSendAttempts`, `GmailConnections`,
`GmailReviewDecisions`, `ImapConnections`, `InterviewPrepItems`, `InterviewPrepNotes`, `JobApplications`, `JobEvents`, `Jobs`,
`MicrosoftGraphConnections`, `RuleSettings`, `SystemEmailSettings`, `TailoredCvDrafts`,
`TrustedDevices`, `TwoFactorRecoveryCodes`, `UserNotifications`, `UserOperations`, `UserRuleSettings`,
and `UserSessions`.
`SystemEmailSettings` is the first completed ownership transfer: migration
`20260830120000_AdoptSystemEmailSettingsSchema` creates it for both supported providers and preserves
an existing reconciler-created MariaDB table. Startup no longer creates it.
`UserRuleSettings` followed in `20260830121000_AdoptUserRuleSettingsSchema`; its owner-keyed rows are
preserved across adoption, downgrade and retry, and startup no longer creates it.
`GmailReviewDecisions` moved in `20260830122000_AdoptGmailReviewDecisionsSchema`. This also supplies
the table on MariaDB, where the old reconciler had no creation path.
The authentication support group (`TwoFactorRecoveryCodes`, `TrustedDevices`, and `UserSessions`)
moved in `20260830123000_AdoptAuthenticationSupportSchema`. These tables deliberately have no
database foreign key to `AspNetUsers` because they are queried during authentication before a
current-user scope exists.
The email-provider connection group (`GmailConnections`, `MicrosoftGraphConnections`, and
`ImapConnections`) moved in `20260830124000_AdoptEmailConnectionSchema`. Startup retains only
additive Gmail sync-column repair plus guarded MariaDB identity/index repair; it does not create
these tables.
The CV import persistence group (`CvUploadArtifacts` and `CvExtractionRuns`) moved in
`20260830125000_AdoptCvExtractionSchema`. The extraction run's optional artifact foreign key remains
`ON DELETE SET NULL`, so retention cleanup cannot erase the extraction/review record.
`TailoredCvDrafts` moved in `20260830126000_AdoptTailoredCvDraftSchema`; its generated and edited
content is retained during adoption while the one-to-one job-application relationship remains
`ON DELETE CASCADE`.
The persisted job-workspace note group (`InterviewPrepNotes` and `AiWorkspaceNotes`) moved in
`20260830127000_AdoptJobWorkspaceNotesSchema`. Both caches retain their unique owner/job keys and
remain application-owned through `ON DELETE CASCADE`.
The CV builder aggregate (`CvVariants` and `CvVariantVersions`) moved in
`20260830128000_AdoptCvVariantSchema`. Public slugs remain unique, deleting a linked application
sets the CV link to null, and deleting a CV cascades through its append-only revision history.
`AiInteractions` moved in `20260830129000_AdoptAiInteractionSchema`. The older cross-feature usage
migration retains its compatibility bootstrap for historical chain traversal, but startup no longer
creates the table; only additive counter and guarded MariaDB shape/index repairs remain.
`ApplicationChecklistItems` moved in `20260830130000_AdoptApplicationChecklistSchema`. Its stable
system-key uniqueness and owner/job/sort index preserve idempotent seeding alongside freely ordered
manual tasks; application deletion remains cascading.
`CoverLetterVersions` moved in `20260830131000_AdoptCoverLetterVersionSchema`. Manual and AI-approved
revisions retain their source/action metadata and owner/job/version ordering; application deletion
remains cascading.
`InterviewPrepItems` moved in `20260830132000_AdoptInterviewPrepItemSchema`. User-authored and
AI-generated questions, answers, preparation state, sources, and ordering are retained; application
deletion remains cascading.
The Career Profile aggregate (`CareerProfiles`, `CareerProfileVersions`, and all six relational
child tables) moved in `20260830133000_AdoptCareerProfileSchema`. Canonical/long-tail JSON,
append-only history, stable child keys and ordering are retained; bounded MariaDB owner/item keys
keep all aggregate indexes provider-safe.
The ASP.NET Identity aggregate moved in `20260830134000_AdoptIdentitySchema`. Accounts, password
hashes, security/2FA state, UI preferences, roles, claims, external logins, tokens, indexes, and
cascades are retained. The earlier guarded `AspNetUsers` bootstrap remains only so historical
standalone migration traversal can reach later additive user-column migrations.
The reconciler may **repair** these (add a missing column, add an index, fix a non-`AUTO_INCREMENT`
primary key) and may seed the default `RuleSettings` row — but it must never `CREATE TABLE` them.
It used to create `RuleSettings`, which is precisely why a clean install failed with
`Table 'RuleSettings' already exists` once `Migrate()` reached the initial migration.
### Reconciler-owned
None. The reconciler contains guarded historical repair logic, but it no longer creates EF model
tables. Compatibility bootstraps inside old migrations are chain prerequisites, not current owners.
`StartupSchemaOwnership` is the executable inventory. Its tests require every EF model table to
have exactly one creation owner and keep compatibility bootstraps out of the migration-owned set.
No-op migrations, each with a comment explaining why:
| Migration | Tables |
|---|---|
| `20260717222917_AddCareerProfileRelationalChildren` | historical no-op; ownership transferred by `20260830133000_AdoptCareerProfileSchema` |
| `20260718074509_AddCvVariants` | historical no-op; ownership transferred by `20260830128000_AdoptCvVariantSchema` |
| `20260718131138_AddAiInteractions` | historical no-op; ownership transferred by `20260830129000_AdoptAiInteractionSchema` |
| `20260719085904_AddApplicationChecklistItems` | historical no-op; ownership transferred by `20260830130000_AdoptApplicationChecklistSchema` |
| `20260719094728_SyncCareerChildKeyLengths` | historical snapshot sync; ownership transferred by `20260830133000_AdoptCareerProfileSchema` |
| `20260719120954_AddCoverLetterVersions` | historical no-op; ownership transferred by `20260830131000_AdoptCoverLetterVersionSchema` |
| `20260719145044_AddInterviewPrepItems` | historical no-op; ownership transferred by `20260830132000_AdoptInterviewPrepItemSchema` |
### Dependency guards
A reconciler table that references another table is guarded on **its parent existing**, so pass 1
skips it on a fresh database and pass 2 creates it:
| Table | Waits for |
|---|---|
| `TailoredCvDrafts`, `InterviewPrepNotes`, `AiWorkspaceNotes`, `CvVariants`, `AiInteractions`, `ApplicationChecklistItems`, `CoverLetterVersions`, `InterviewPrepItems` | `JobApplications` (migration-owned) |
| `CvVariantVersions` | `CvVariants` |
| `CareerProfileVersions`, the six CareerProfile children | `CareerProfiles` (migration-owned) |
| `CvExtractionRuns` | `CvUploadArtifacts` |
Index creation goes through one helper, `EnsureMySqlIndex`, which is guarded on **table** existence
as well as index existence — repairing an absent table is not pass 1's job.
## Adding a new table
1. Add the entity and its `DbSet`, and **bound every indexed string** with `HasMaxLength` — an
unbounded string becomes `longtext`, which MariaDB cannot index without a prefix length. This is
what broke the CareerProfile children.
2. Add an EF migration. Review its generated SQL for both SQLite and MariaDB; do not assume a
SQLite-scaffolded type is valid on MariaDB.
3. If needed, replace the generated operation with provider-aware migration SQL (`int
AUTO_INCREMENT`, bounded `varchar(n)`, `datetime(6)`, `tinyint(1)` on MariaDB). Do not add the
table to `StartupInitializationExtensions`.
4. Classify the table in `StartupSchemaOwnership.MigrationOwnedTables`; the ownership test will fail
if it is omitted or duplicated.
5. Verify blank, populated-upgrade and restart paths on SQLite and a real MariaDB container. EF
InMemory will not catch provider DDL defects.
## Retiring a reconciler-owned table
Move one leaf/dependency group at a time:
1. Add an idempotent provider-aware migration that creates the legacy shape when absent.
2. Make downgrade preserve a table that may predate migration ownership; never drop ambiguous data.
3. Remove only that table's startup `CREATE TABLE` block and move it between the executable ownership sets.
4. Prove a blank database, a pre-migration database with a representative row, migration retry, and
provider SQL/runtime behaviour.
5. Leave column/index repairs in place until historical upgrade fixtures prove they are redundant;
table creation and legacy repair are separate ownership decisions.
## Fresh install
No manual database preparation. Point the app at an **empty** database and start it:
```bash
# MariaDB
Database__Provider=mysql \
ConnectionStrings__JobTracker="Server=…;Database=jobtracker;User=…;Password=…;" \
dotnet run --project JobTrackerApi/JobTrackerApi.csproj
# SQLite (default) — creates Data__Root/jobtracker.db
dotnet run --project JobTrackerApi/JobTrackerApi.csproj
```
Create the empty schema/database itself (`CREATE DATABASE jobtracker;`); the application builds
everything inside it.
Standalone EF tooling is also supported for a blank SQLite database. The historical initial
migration now supplies the stable JobApplication columns required by later SQLite rebuilds, and
guarded compatibility bootstraps provide source tables used by later additive migrations.
Application startup may subsequently apply guarded historical column/index repairs without losing rows.
## Production upgrade
Deploy and restart. The reconciler is idempotent and additive:
- it never drops a table that holds rows (`DropMalformedMySqlTable` checks the row count first)
- it only adds missing columns, tables and indexes
- migrations already recorded in `__EFMigrationsHistory` are not re-run, so emptying a migration's
`Up` changes nothing for an existing database
No downtime step, no manual SQL, no data migration.
## Verified
All four scenarios, 2026-07-19, against MariaDB 11 and SQLite:
| Scenario | Result |
|---|---|
| Empty MariaDB | 42 tables created, app starts |
| Populated MariaDB, restart | idempotent — still 42 tables, rows preserved |
| Partially-migrated MariaDB (Phase 4/5 tables dropped) | healed 35 → 42, surviving rows preserved |
| Empty MariaDB via the Docker image | 42 tables created, app starts |
| Fresh SQLite | 42 tables created, app starts |
| Existing partially-migrated SQLite dev DB (34 tables) | upgraded to 44 tables, 13 applications and 8 companies preserved |
Column types on MariaDB spot-checked: `int AUTO_INCREMENT` primary keys, `varchar(255)` owner keys,
`datetime(6)` timestamps, `tinyint(1)` booleans, and every composite index inside the key limit.
On 2026-08-15 the current 29-migration chain was additionally verified against a blank standalone
SQLite database, an older populated SQLite checkpoint, and a disposable MariaDB 11.8 database.
Standalone SQLite migration and retry both reached the latest migration; populated title/date and
reconciler-owned owner/summary data survived. Starting the application over that EF-only database
served `/health` successfully. Fresh MariaDB startup and restart both served `/health` with 29
migrations and 49 tables; provider-sensitive ID, owner, decimal and timestamp column types were
spot-checked. No production database was changed.
docs/_archive/pre-docs-cleanup-2026-09-11/docs/infrastructure/runner-investigation.md
+343
View File
@@ -0,0 +1,343 @@
# Infrastructure Investigation — CI runner and deploy failures
> **Historical investigation.** The current workflow retains the full test gate and defensive setup
> retries; current runner verification remains tracked in `BLOCKERS.md` until a reviewed tree passes CI.
> 2026-07-18. Supersedes `docs/ci-runner-investigation.md`.
> **Conclusion (2026-07-18): failures A and B are outside the repository.** Application code was
> eliminated as a cause for those two by direct experiment. Confirmation and repair require host
> access — the exact asks are at the end.
>
> **Update 2026-07-19 — a THIRD, unrelated failure was a real code bug, and the runner caught it
> correctly.** After the Phase 5 work added `HumanLanguageCatalogTests`, the `test` job failed on
> `Norwegian_aliases_resolve_to_the_canonical_name("nynorsk")`. This was **not** runner instability:
> it was a genuine host-ICU-dependence defect that reproduces deterministically on the runner's ICU
> version and is now fixed (`fba858e`, commit "force language alias precedence over host culture data").
> Full evidence in *Finding C* below. The lesson: do not assume every red on this runner is
> environmental — this one was the runner doing its job.
There are **two independent infrastructure failures**:
- **A — CI `test` job**: the backend suite fails only on the self-hosted `Live-Runner`.
- **B — CI `deploy` job**: the SSH step fails ~3 s in, before doing any work.
They are unrelated to each other and to the application code.
---
## A — Backend suite fails only on the runner
### Evidence gathered
- The suite had **never actually run in CI**. The workflow built only `JobTrackerApi`, then ran
`dotnet test --no-build`, so the test project was never compiled and the step was a ~1 s no-op.
Fixed in `cfba7fb`. **The failure is newly surfaced, not a regression** — it may be long-standing.
- Job logs are unreadable: `GET /api/v1/.../actions/jobs/{id}/logs` → `401 token is required`.
Step boundaries were therefore the only telemetry, so the suite was bisected across CI runs.
- Failure is localised to the **`AiWorkspace` classes** — 10 tests across `AiWorkspaceTests`
(Phase 5) and `AiWorkspaceNotePersistenceTests` (pre-existing). Locally these run in **1 s**.
| Run | Steps observed | Reading |
|---|---|---|
| 524 | `Test backend` ✗ 8 s | one combined step, no detail |
| 525 | restore ✓ 4 s, build ✓ 4 s, **test ✗ 3 s** | not a restore or compile error |
| 526 | host smoke ✓ **1 s**, full suite (serial) ✗ 3 s | test host starts fine; not parallelism |
| 527 | quarters: **A–C ✗ 3 s**, rest never ran | offender is alphabetically early |
| 528 | per class: **`T AiWorkspace` ✗ 3 s**, others never ran | offender named |
### Experiments performed
Every experiment ran the same commit. All pass unless stated.
| # | Experiment | Result |
|---|---|---|
| 1 | Windows host, full suite | 306 pass |
| 2 | Clean `mcr.microsoft.com/dotnet/sdk:9.0` container (Linux, case-sensitive FS) | 306 pass |
| 3 | CI's exact order: build `JobTrackerApi` → then build/test the test project | 306 pass |
| 4 | Memory cap `--memory=1g --memory-swap=1g` | 306 pass |
| 5 | Bare `ubuntu:22.04`, SDK via `dotnet-install.sh` into `$HOME/.dotnet`, `PATH` only, **`DOTNET_ROOT` unset** — mirrors the runner's SDK setup | 306 pass |
| 6 | Collection parallelism disabled (`parallelizeTestCollections=false`, `maxParallelThreads=1`) | passes locally; **still fails on runner** |
| 7 | `LC_ALL=LANG=tr_TR.UTF-8` (Turkish-I culture trap) | 10/10 pass, 1 s |
| 8 | `TZ=Pacific/Kiritimati` (UTC+14) | 10/10 pass, 1 s |
| 9 | **Clean `git archive HEAD` tree** — byte-identical to CI's checkout, with none of the gitignored runtime dirs (`jobtracker.db`, `keys/`, `CvArtifacts/`, `backups/`) present locally | 10/10 pass, 1 s |
| 10 | Shared-state audit: `TestHostFactory.CreateInMemoryDb` uses `Guid.NewGuid()` per test | no shared store |
Experiment 9 is the decisive one: it removes the last difference between the local tree and the
runner's checkout. The exact source CI compiles produces a passing suite.
### Root cause hypothesis
The runner host kills the test process. The workflow already documents **three separate failure modes
on this same runner, all with the signature of a process dying with no usable error output**:
- `actions/setup-dotnet` "intermittently leaves a partial extraction in the shared tool-cache
(`tar: Cannot open: File exists`) or corrupts the SDK download" — hence the hand-rolled installer.
- `npm ci` "occasionally segfaults on the runner (SIGSEGV/139, a memory/native flake)".
- The frontend build "has repeatedly died silently on this runner with no error output
(OOM/SIGSEGV signature — same resource-starved-runner class)".
A .NET test host exiting ~3 s into a 10-test run belongs to that same family. Two candidate mechanisms,
in order of likelihood:
1. **Resource exhaustion — memory or PID/thread limits.** The runner appears to share the host with
the production Docker stack. A 1 GB cap did not reproduce it, so either available memory at that
moment is lower, or the binding limit is `pids`/threads rather than RAM (the .NET test host spawns
more threads than `npm ci`, so it would hit a low `pids.max` first).
2. **Disk exhaustion.** This fits the documented symptoms better than memory does: partial tar
extraction, corrupted downloads, and silent process deaths are all classic disk-full signatures.
`testhost` writes `TestResults/` and may write dumps.
### Host telemetry (2026-07-18, post-reboot) — resource exhaustion ruled out
Collected from the server:
```
/ 217G 146G 62G 71% (inodes 14% used)
/dev/shm 16G 0 16G 0% (=> ~32 GB RAM)
ulimit -u 127749 ulimit -n 1024 ulimit -m unlimited
/sys/fs/cgroup/pids.max: No such file or directory
fail2ban-client: command not found
journalctl -u sshd: No entries (Ubuntu's unit is `ssh`, not `sshd`)
dmesg: read kernel buffer failed: Operation not permitted (needs sudo; cleared by reboot anyway)
```
**This falsifies both resource hypotheses at the host level**: there is no disk pressure, no inode
pressure, no memory pressure and no restrictive PID limit. It also falsifies the fail2ban explanation
for the deploy failure — fail2ban is not installed.
**Caveat that matters:** Gitea `act_runner` normally executes a `runs-on: ubuntu-latest` job **inside
a Docker container**, so the figures above describe the *host*, not the environment the tests
actually ran in. A job container has its own cgroup memory/PID limits and, by Docker default, a
**64 MB `/dev/shm`** regardless of the host's 16 GB. The relevant limits have therefore not been
measured yet — see the asks below.
The one host-level value worth noting is `ulimit -n 1024` (file descriptors), which is low by modern
standards, though it is the interactive shell's value and not necessarily the runner service's.
### Post-reboot runs — the failure moves between stages
Two further runs after the machine was restarted:
| Run | Result |
|---|---|
| `8f73548` (post-reboot) | **Identical** to pre-reboot: host smoke ✓ 1 s, full suite ✗ 3 s |
| `c4c0cd4` | Failure **moved earlier**: `Restore backend tests` ✗ **0 s** — a step that took 3–4 s and succeeded in all previous runs. None of the queued diagnostic steps executed. |
Two conclusions:
1. **The reboot changed nothing**, so this is not stuck state, a leaked process, or a corrupted
workspace that a restart would clear.
2. **The failing stage is not stable across runs.** `dotnet restore` failing in 0 s on one run and
succeeding in 3–4 s on the next, with the same commit and the same runner, is nondeterministic
infrastructure behaviour. It also argues *against* a deterministic explanation such as a seccomp /
W^X policy blocking runtime IL emission (which would fail identically every time), and against any
single-test explanation.
Taken with the three failure modes the workflow already documents on this runner (SDK tar corruption,
`npm ci` SIGSEGV, silent CRA build death), the pattern is a runner that intermittently kills or fails
child processes at arbitrary stages, with no diagnostic surfaced.
### Confidence level
- **Application code is not the cause — high confidence (~95 %).** Ten independent environments,
including a byte-exact clean checkout, all pass. Every axis raised (Linux behaviour, case
sensitivity, path separators, locale, time zone, environment variables, parallel execution, test
ordering, shared state, memory) has been experimentally eliminated.
- **Specific mechanism — low confidence (~25 %), and lower than before.** Host telemetry ruled out
disk, inode, memory and PID exhaustion; the reboot ruled out stuck state; the moving failure stage
ruled out a deterministic sandbox policy. What remains is an intermittent fault in the runner's
execution environment (most likely the job container / `act_runner` configuration rather than the
host), which cannot be identified without the job log or runner config. I am not asserting a
mechanism.
### Why application code is no longer suspected
1. The identical commit passes in nine environments, including one built from `git archive HEAD` —
exactly what CI checks out, with no local-only files.
2. The 10 failing tests use EF **InMemory** with a per-test GUID database, `Moq`, and a fake
summarizer. They open no file, no socket, no process, and assert on no clock or culture value.
3. The test host demonstrably starts and passes a test **on the runner itself** (host smoke, 1 s), so
this is not a toolchain or assembly-load problem.
4. The failure survives disabling parallelism and is unaffected by execution order — the tests are
mutually isolated.
5. Three pre-existing, code-unrelated failure modes with the same "silently killed process" signature
are already documented on this exact runner and worked around with retries.
---
## B — Deploy job fails before doing any work
### Evidence
- Step `Run remote deploy` (the `appleboy/ssh-action`) failed in **3 s** (`18:44:07 → 18:44:10`).
That is before `git fetch`, before `deploy.sh`, before any Docker build.
- The **first** deploy attempt (run 522, commit `7a74311`) failed after **37 s** — long enough, with a
warm Docker cache, to have run `deploy.sh` and failed its post-deploy backend health check. That is
consistent with the MariaDB migration crash fixed in `1430313`.
- Every attempt since fails at **3–7 s**, i.e. at connection time. The failure mode changed.
- **Production is up and healthy**: `https://jobs.cesnimda.uk/` → 200 HTML,
`/api/auth/config` → 200 JSON (`{"requireAuth":true,...}`). The host is reachable from the internet,
so this is not an outage.
- **Production is stale — Phase 4 and Phase 5 have never deployed.** Probe:
`/api/public-cv/{unknown}` returns **404 locally** (the route exists and is `AllowAnonymous`) but
**401 on prod**, identical to prod's response for a nonsense path such as
`/api/definitely-not-a-route-xyz`. `PublicCvController` is absent from production.
### Consequence worth recording
Because no Phase 4/5 deploy ever succeeded, **production never executed the faulty migration**. There
are no half-built `CvVariants`/`AiInteractions` tables in production, and no data cleanup is required.
The reconciler will create all three tables correctly on the first successful deploy;
`DropMalformedMySqlTable` remains as harmless, row-count-guarded insurance.
### Hypothesis
The prod host is refusing the runner's SSH connection rather than failing inside the script. Most
likely `fail2ban`/`sshd` blocking the runner's IP after the repeated failed deploy attempts, or a
changed host key / rotated `PROD_SSH_KEY`. Confidence: **medium (~50 %)** — the timing and the 37 s →
3 s transition support it, but it cannot be confirmed without the host.
---
## C — Backend test fails on the runner's ICU version (real code bug, FIXED 2026-07-19)
Unlike A and B, this failure **was** in the application code. The runner reported it correctly.
### Symptom
```
JobTrackerApi.Tests.HumanLanguageCatalogTests.Norwegian_aliases_resolve_to_the_canonical_name(alias: "nynorsk") [FAIL]
Assert.Equal() Failure: Strings differ
Expected: "Norwegian"
Actual: "Norwegian Nynorsk"
Failed! - Failed: 1, Passed: 419
```
Passed on the author's Windows machine and in Debian/Ubuntu-22.04+ containers; failed only on the
runner. That pattern *looks* like instability — but it is deterministic, and the cause is the runner's
**ICU (libicu) version**, not its stability.
### Root cause — ruled in, not assumed
`HumanLanguageCatalog.BuildLanguageLookup` seeds explicit aliases (`nynorsk`, `bokmål`, `norsk` →
`Norwegian`) using `Dictionary.TryAdd`, *after* enumerating `CultureInfo.GetCultures`. `TryAdd` keeps
the first value written, so if culture enumeration already claimed a key, the explicit alias silently
loses.
The `nn` (Norwegian Nynorsk) culture's **NativeName differs by libicu version**:
| ICU | `nn` NativeName (cleaned) | key `nynorsk` populated by enumeration? | old code result |
|---|---|---|---|
| libicu66 (Ubuntu 20.04) | `nynorsk` | **yes** → `Norwegian Nynorsk` | `nynorsk` → **`Norwegian Nynorsk`** ❌ |
| libicu70 (Ubuntu 22.04) | `norsk nynorsk` | no | `nynorsk` → `Norwegian` ✓ |
| libicu72 (Debian 12) | `norsk nynorsk` | no | `nynorsk` → `Norwegian` ✓ |
| libicu74 (Ubuntu 24.04) | `norsk nynorsk` | no | `nynorsk` → `Norwegian` ✓ |
So the runner is on **old ICU (libicu66 / Ubuntu 20.04-class)**. This is the same host-ICU-dependence
class as the earlier language-drop bug (`9681618`).
### Evidence (reproduced end to end)
- **Probe on `runtime:6.0-focal` (libicu66):** the `nn` culture's `NativeName` is the bare word
`nynorsk`; the old `TryAdd` logic resolves `nynorsk` → `Norwegian Nynorsk` — exactly the CI failure.
- **Same probe with the fix logic on the same libicu66:** `nynorsk` → `Norwegian`.
- **The real net9 test DLL on Ubuntu 20.04 / libicu66, .NET 9.0.316 installed via `dotnet-install.sh`
(identical to CI):** 420/420 pass with the fix.
- Also 420/420 on libicu72 (Debian) and libicu74 (Ubuntu 24.04), and locally on Windows.
### Fix
`fba858e` — an `Override` helper (`map[key] = value`) applied to the alias block so the explicit
mappings win regardless of what culture enumeration inserted. Correct for any ICU version by
construction; no test weakened.
### Why this matters for A and B
It does not exonerate the runner — A (test host dies at ~3 s) and B (deploy SSH fails at ~3 s) remain
separately evidenced as environmental. But it is a caution: **not every red on this runner is
infrastructure.** This one was a real defect the runner surfaced because its ICU is older than any
developer machine. Worth keeping the runner's OS/ICU in mind as a legitimate signal, not just noise.
---
## Required infrastructure changes
**Blocking — cannot proceed without one of these:**
1. **The job log for step `T AiWorkspace` (run 528)** — roughly 20 lines settles issue A outright. Or
a **read-scoped Gitea API token**, so CI failures can be diagnosed without a human relay. This is
the single highest-value item.
2. **The deploy job log (run 523+)** — the `ssh-action` error line settles issue B.
**Host checks (issue A):**
3. `dmesg -T | grep -iE 'oom|killed process'` around the run time — a killed `dotnet`/`testhost`
confirms the OOM hypothesis.
4. `df -h` and `df -i` on the runner's work and Docker volumes — tests the disk-exhaustion hypothesis.
5. `ulimit -a` and `cat /sys/fs/cgroup/pids.max` for the runner user — tests the PID-limit hypothesis.
6. `journalctl -u <gitea-runner-service> --since '2 hours ago'`.
**Host checks (issue B):**
7. `fail2ban-client status sshd` on the prod host, and `journalctl -u sshd --since '2 hours ago' | grep -i <runner-ip>`.
8. Confirm the `PROD_HOST`/`PROD_USER`/`PROD_SSH_KEY` secrets still match the host's
`authorized_keys`, and that the host key has not changed.
**Recommended remediation regardless of which hypothesis lands:**
9. **Give the runner its own resource allocation, or move it off the production host.** It currently
appears to share a box with the prod Docker stack. This is the common root of the documented
`npm ci` segfaults, silent CRA build deaths, SDK cache corruption, and now the test host death —
all of which are currently papered over with retries.
---
## Decisive evidence: a docs-only commit fails identically (2026-07-19)
The strongest single data point, found while pushing Phase 5 Milestone 2:
| Run | Commit | What it changed | `test` job |
|---|---|---|---|
| 531 | `8f73548` | **one markdown file** — `docs/infrastructure/runner-investigation.md` | failure, 1m18s |
| 532 | `c4c0cd4` | CI workflow experiment | failure, 52s |
| 533 | `3b59152` | CI workflow + markdown | failure, 1m0s |
| 534 | `e55a6e8` | Phase 5 Milestone 1 (app code) | failure, 1m13s |
| 535 | `3a906b8` | Phase 5 Milestone 2 (app code) | failure, 1m8s |
`8f73548` changed **no application code, no test, no dependency, no workflow file** — a single
documentation paragraph — and the pipeline failed anyway, in the same duration band as every other
run. A change that cannot affect compilation or test behaviour cannot cause a test job to fail.
This closes the question the investigation was asked to answer: **the failure is not in the
repository.** It raises confidence that application code is not the cause from ~95% to effectively
certain. The specific runner mechanism remains unidentified (still ~25%) and still requires the
access listed above.
Local verification of `3a906b8` used CI's exact commands, in Release, with CI's parallelism flags:
329 backend tests, 94 frontend tests (32 suites, `--runInBand`), `tsc --noEmit`, the Next.js
production build, and both Docker images — all green.
### Separately discovered: bare-MariaDB bootstrap fails (pre-existing, not this failure)
Booting the API against a **completely empty** MariaDB crashes with
`Table 'jobtracker.CareerProfiles' doesn't exist` — the MySQL reconciler assumes the migration-owned
tables already exist. Reproduced on clean `HEAD` (`git stash`) *before* the Milestone 2 changes, so
it is pre-existing and unrelated. It has never affected prod, whose database is long since populated,
and it does not affect CI, which does not start the backend. Worth fixing on its own, but out of
scope here and **not** the runner failure.
---
## Repository state
Kept — all correct independent of the outcome, none reverted:
- `1430313` — CV builder / AI workspace tables provisioned by the MySQL-safe reconciler
(reproduced and verified against a real MariaDB 11 container).
- `cfba7fb` — CI actually runs the backend suite.
- `45725ac` — restore / build / test split, restore retries once.
- `2bdc4a9` — one-test host smoke; collection parallelism disabled for determinism.
- `7fa3080`, `0f62dc4` — bisection scaffolding, since removed.
**No test was weakened, skipped, filtered, or disabled at any point.** CI is red on purpose: the
failure is real and must stay visible until the runner is fixed.