jobtrackingapp/docs/release-candidate-review.md

# Release candidate review

> **Historical snapshot (2026-07-19).** For the reconciled 2026-07-31 implementation status and
> remaining external work, use `docs/implementation-roadmap.md` and `BLOCKERS.md`.

> 2026-07-19. Final validation pass before the first production deployment. **No application code was
> changed by this review** — findings only. Every claim below was checked against the implementation,
> not against the other documents. Where a document and the code disagreed, the code is reported.
>
> Companions: `deploy/first-production-deployment.md` (how to deploy),
> `docs/production-readiness-review.md` (what was audited),
> `docs/infrastructure/database-ownership.md` (who creates which table),
> `docs/release-checklist.md` (state of the build).

**Updated 2026-07-19 (final pass).** Three findings are now fixed and verified:

- **B1** — the pre-deploy backup silently backed up the wrong thing (`66b02bc`)
- **N2** — `/health` always reported `version: unknown` (`8f6f2ba`)
- **CV languages** — human languages were dropped depending on the host's ICU data (`9681618`)

The first two are recorded under *Closed since the first pass* with their original text, because the
failure modes are worth understanding. Also since the previous pass: the deployment sequence was
re-verified by line number rather than by prose, and a full backup → restore → start-the-app rehearsal
was completed and recorded in
[`docs/operations/production-backup-verification.md`](operations/production-backup-verification.md).

**Release-candidate audit, 2026-07-19.** A full verification pass — 420 backend tests (Windows + Linux,
both ICU modes), frontend tests/typecheck/build, all three Docker images, and all four database startup
scenarios run against live MariaDB 11 and SQLite — found and fixed two more issues: a follow-up reminder
index that never created on MariaDB (and logged a false rollback signal on every boot), and a
nondeterministic timeline test. Both are recorded below. Backup → restore → app-start was re-run
end to end. No open blocker remains in the code or deployment path.

**Correction 2026-07-19 (later).** The next CI run then failed on a *third*, real code bug — a
language-alias lookup that depended on the runner's older ICU version (`nynorsk` → `Norwegian Nynorsk`
instead of `Norwegian`). This was **not** runner instability; the runner caught a genuine defect. Fixed
in `fba858e` and verified on the runner's exact ICU (Ubuntu 20.04 / libicu66). So B2 below is no longer
"purely external": the most recent red was ours. See `docs/infrastructure/runner-investigation.md`
Finding C. The current status is carried by `docs/release-final-report.md`.

**Verdict: one blocker remains, and it is external — CI (B2).** Nothing in the application or the
deployment path is now known to be blocking. Everything verified here is *local* verification; CI has
proven none of it.

| Section | Meaning |
|---|---|
| [READY](#ready--completed-technical-checks) | Verified. Nothing further needed before deploying |
| [BLOCKED](#blocked--requires-external-action) | Requires action outside this repository. Deployment is gated on it |
| [MANUAL VERIFICATION](#manual-verification--only-the-owner-can-do-these) | Only the owner can confirm — sign-in, real data, production smoke tests |
| [Accepted](#accepted-for-the-first-release-non-blocking) | Known limitations, deliberately shipped as-is |

---

## READY — completed technical checks

### Database

| Item | Verified against |
|---|---|
| Startup order is connect → reconcile → `Migrate()` → reconcile → seed | `StartupInitializationExtensions.InitializeJobTrackerAsync`; matches `database-ownership.md` exactly |
| Migration ownership | `Companies`, `JobApplications`, `Jobs`, `Correspondences`, `Attachments`, `JobEvents`, `RuleSettings`, Identity — created only by migrations; reconciler repairs but never `CREATE TABLE`s them |
| Reconciler ownership | Every Phase 4/5 table is reconciler-owned with a paired no-op migration holding the snapshot. The seven no-op migrations named in `database-ownership.md` all exist on disk |
| Dependency guards | Reconciler tables with an FK are guarded on the parent existing, so pass 1 skips and pass 2 creates. `EnsureMySqlIndex` guards on table existence, not just index existence |
| Reconciler is non-destructive | `DropMalformedMySqlTable` checks row count first and skips any table holding rows |
| Provider selection accepts both spellings | `Program.cs:68` — `provider is "mysql" or "mariadb"`. `deploy/README.md` documents `DATABASE_PROVIDER=mariadb`; that value works |
| Rollback reasoning is sound | Phase 4/5 migrations are no-ops, so reverting code never leaves migration history ahead of the schema. Older code ignores the extra tables |

### Security

| Item | Verified against |
|---|---|
| Explicit authorization | Every user-owned controller carries a class-level `[Authorize(AuthenticationSchemes = "local")]`. Confirmed directly on `JobApplicationsController`, `CompaniesController`, `CorrespondenceController`, `RulesController`, `AttachmentsController`, `BackupController`, `ExportController` |
| Independent of `Auth:Require` | The `FallbackPolicy` at `Program.cs:380` is *additional*, not the only defence. `/api/jobapplications` returns 401 with `Auth:Require` unset |
| Anonymous surface is a deliberate allow-list | `PublicCvController` (class-level `[AllowAnonymous]`), `ClientErrorsController`, `/health`, and per-method anonymity on `AuthController` / `TwoFactorController` / the two OAuth callbacks |
| OAuth callbacks are not an open door | `GmailController.Callback` and `MicrosoftGraphController.Callback` are anonymous by necessity but gated on `ConsumeState(state)`; an unknown or replayed state is rejected before any token exchange |
| Tenant isolation | 25 global query filters, all deny-on-null (`CurrentUserId != null && OwnerUserId == CurrentUserId`), covering every owner-scoped root entity |
| Attachments are tenant-scoped and path-safe | Resolved through the parent `JobApplication` query (so the job-level filter applies); stored names go through `Path.GetFileName` + `BuildStoredFileName`, so a crafted upload name cannot escape the attachments root |
| AI service is not reachable from outside | `ai-service` is on `ai_internal` only — not on `default`, not on the external `shared_services`, no published port. Backend is the only member that can route to it |
| AI service authenticates its caller | `X-Ai-Service-Token` middleware with `hmac.compare_digest`; only `/health` is open. Compose declares the token with `:?` so a deploy that forgets it fails loudly |
| Public CV is opt-in | Off by default, per-variant, unguessable slug, `noindex` |
| Secrets not committed | `.env` is gitignored (`.gitignore:8`); nothing sensitive tracked |
| Backup handles the password safely | `MYSQL_PWD`, never on the command line, so it cannot reach the process list or the deploy log. Dump stderr is scrubbed before it is echoed |

### Application integrity — architecture rules

| Rule | Verified |
|---|---|
| **CareerProfile is the only editable career source** | The two Phase 5 services that touch `CareerProfiles` — `ApplicationChecklistService:343` and `ApplicationIntelligenceService:199` — both read `AsNoTracking()`. Nothing downstream writes to it |
| **CvVariant is a derived lens** | `ApplicationAssetsService.AttachVariantAsync` writes only `JobApplicationId` and `UpdatedAtUtc`. Attaching a CV to an application never touches variant content |
| **Application Workspace aggregates only** | `ApplicationIntelligenceService` and `ApplicationTimelineService` contain **zero** `SaveChanges` calls. They are pure projections |
| **JobEvent is the timeline source of truth** | `ApplicationTimelineService` interprets `JobEvent` rows and stores nothing. Emission is centralised in `JobLifecycleEvents` |
| **AI is suggestion-only** | Generation appends to `AiInteraction`; nothing is written to a profile, variant, cover letter or prep item without an explicit user save |

### Deployment sequence — verified against the code, not the prose

Checked by line number in `deploy/deploy.sh` and `StartupInitializationExtensions.cs` on 2026-07-19.

**Before anything is replaced** — the order is what matters, and it holds:

| Order | Step | Where |
|---|---|---|
| 1 | Load `/opt/job-tracker/shared/.env` into the deploy shell | `deploy.sh` (before any decision) |
| 2 | **Validate configuration** — provider, connection string, `AI_SERVICE_TOKEN`, `AUTH_JWT_KEY` | `deploy.sh:343` |
| 3 | **Take and verify the database backup**; abort the deploy if it fails | `deploy.sh:347` |
| 4 | Build images | `deploy.sh:375` |
| 5 | Replace containers (`up -d --force-recreate`) | `deploy.sh:382` |

Nothing is built, stopped or replaced before validation and backup. Confirmed: 343 and 347 both
precede 375 and 382.

**During deployment** — backend startup, `InitializeJobTrackerAsync`:

| Order | Step | Where |
|---|---|---|
| 1 | `ReconcileSchema()` — pass 1, repair and reconciler-owned tables | `StartupInitializationExtensions.cs:1972` |
| 2 | `Database.Migrate()` — migration-owned tables | `:1980` |
| 3 | `ReconcileSchema()` — pass 2, everything pass 1 had to skip | `:1991` |

**Migrations expected to run: none that create anything.** All seven Phase 4/5 migrations were
confirmed to have a **literally empty `Up` body** (0 statements each): `AddCareerProfileRelationalChildren`,
`AddCvVariants`, `AddAiInteractions`, `AddApplicationChecklistItems`, `SyncCareerChildKeyLengths`,
`AddCoverLetterVersions`, `AddInterviewPrepItems`. The reconciler owns their DDL with correct
per-provider types. This is what makes a code rollback safe — migration history never runs ahead of
the schema.

**Health checks and failure behaviour:**

| Item | Verified |
|---|---|
| Backend health check | `curl` against `/health`, 90s start period for first-boot reconciliation |
| Frontend health check | `wget` against nginx |
| Dependency gate | `frontend` declares `depends_on: backend: condition: service_healthy` — a backend that never reports healthy makes `compose up` fail and `set -e` aborts the deploy |
| Rollback procedure documented | `deploy/first-production-deployment.md` and `deploy/README.md`, both distinguishing code rollback from database restore |

### Deployment

| Item | Verified |
|---|---|
| CI gates deployment | `deploy` job declares `needs: test` and `if: push && ref == refs/heads/main` |
| Deploy is pinned to the tested commit | The remote script `git reset --hard ${{ github.sha }}` and aborts if that commit is not fetchable |
| Container dependency ordering | `frontend` declares `depends_on: backend: condition: service_healthy`. A backend that never reports healthy makes `compose up` fail, and `set -e` aborts the deploy — a broken backend cannot present as a running stack |
| Health checks exist on both services | `curl` against `/health` for backend (90s start period, covering first-boot reconciliation), `wget` against nginx for frontend |
| `/health` does not touch the database | Deliberate: a DB-querying probe would restart a healthy backend on any database blip, and would hand out an unauthenticated way to probe database availability |
| Missing `AI_SERVICE_TOKEN` fails the stack | `${AI_SERVICE_TOKEN:?…}` in compose, on both `backend` and `ai-service` |
| Missing `AUTH_JWT_KEY` fails the backend | With `Auth__Require=true`, `Program.cs:241` throws `InvalidOperationException`. It does not silently generate an ephemeral key |
| Backup rejects a useless dump | `verify_backup` fails the deploy on an empty file, and on a MariaDB dump lacking `CREATE TABLE` |
| Backups never overwrite | UTC-timestamped filenames; retention is explicitly manual and the script says so |
| **Deploy script loads its own environment** | `deploy.sh` parses the shared `.env` into its own shell before any decision. Values already in the environment (CI's `APP_VERSION`) still win. No value is echoed |
| **`DATABASE_PROVIDER` is required, not defaulted** | Missing or unrecognised aborts the deploy. Verified: exits 1, names the variable, leaves no backup file |
| **Configuration validated before the stack is touched** | Connection string, `AI_SERVICE_TOKEN` and `AUTH_JWT_KEY` are checked before the backup, and therefore before any build, stop or replace |
| **Backups verified per provider** | A dump needs valid gzip, `CREATE TABLE` and the `Dump completed` trailer; an archive needs `jobtracker.db`. Truncated and trailer-stripped dumps both rejected |
| **SQLite volume resolved by real name** | Project-prefixed, and fails if absent — the bare name would have created an empty volume and backed that up |
| **No secret leakage in deploy output** | Zero occurrences of any password, token or key across every failure-path test |
| **`/health` reports the deployed version** | Reads `App:Version`; verified end to end that `App__Version=9.9.9-test` surfaces as `9.9.9-test`, and that an unset value falls back to the assembly version |
| Test suite | 420 backend tests pass in Release, and also pass under `DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1` and on Linux — see the ICU finding below |

---

## Closed since the first pass

Both were found by this review, fixed in their own commits, and re-verified. Original findings kept.

### B1. ~~The pre-deploy database backup does not back up the database~~ — **CLOSED 2026-07-19**

*Original finding, kept because the failure mode is worth understanding:*

**Severity: critical. This invalidates the restore point that the entire deployment plan depends on.**

`deploy/deploy.sh` symlinks `/opt/job-tracker/shared/.env` into the checkout so that **docker compose**
can read it. It never *sources* it. There is no `set -a`, no `. .env`, no `export` of the database
variables anywhere in the script.

So the script's own shell evaluates:

```sh
local provider="${DATABASE_PROVIDER:-sqlite}"
```

`DATABASE_PROVIDER` is not set in that shell. The CI deploy step exports only `APP_VERSION`,
`APP_COMMIT_SHA` and `APP_BUILD_STAMP`, and a non-interactive `ssh` session does not read a profile.
**`provider` resolves to `sqlite` on a MariaDB production host.**

The consequence is not a loud failure, which is what makes this serious:

1. The SQLite branch runs and tars the `jobtracker_data` volume.
2. That volume exists in production (the backend mounts it for `/data`, exports and attachments), so
   the tar succeeds and produces a non-empty file.
3. `verify_backup` is called **without** the `CREATE TABLE` expectation on this path — it only checks
   the file is non-empty.
4. The script prints `Backup verified: …` and the deploy proceeds.

The operator sees a green backup line and a new file in `/opt/job-tracker/backups`. There is no
MariaDB dump. If the deploy then damages the schema, there is nothing to restore.

**Same root cause, two further silent effects:**

- `APP_PUBLIC_BASE_URL` is likewise unset, so the public smoke check at the end of `deploy.sh` never
  runs. `deploy/first-production-deployment.md` states it does.
- `OLLAMA_MODEL` is unset, so the post-deploy Ollama warmup never runs.

**How to confirm before trusting any fix:** run the deploy and check that the newest file in
`/opt/job-tracker/backups` is named `jobtracker-<dbname>-<stamp>.sql.gz`, not
`jobtracker-sqlite-<stamp>.tar.gz`. The filename alone distinguishes the two paths.

**Closed.** `deploy/deploy.sh` now:

1. **Loads `/opt/job-tracker/shared/.env` into its own shell** before any decision. Parsed line by
   line rather than sourced, because a compose `.env` is not a shell script. Variables already set in
   the environment win, so CI-provided `APP_VERSION` and friends still override the file. No value is
   echoed.
2. **Requires `DATABASE_PROVIDER` explicitly.** The `:-sqlite` default is gone. Missing means stop and
   say which variable is missing; an unrecognised value means stop.
3. **Validates the rest of the deployment configuration before the backup**, and therefore before
   anything is built, stopped or replaced: the connection string when the provider needs one,
   `AI_SERVICE_TOKEN` (compose declares it with `:?`, so missing it would otherwise kill the stack
   after the images are built) and `AUTH_JWT_KEY` (the backend throws at startup on a blank key, after
   the containers have been replaced). Names only in the output, never values.
4. **Verifies each backup against its own format.** A MariaDB dump must be valid gzip, contain
   `CREATE TABLE`, and carry the `Dump completed` trailer that `mariadb-dump` writes last — so a dump
   that died partway through is rejected. A SQLite archive must be valid gzip and actually contain
   `jobtracker.db`. A tar can no longer pass the dump check, which is precisely what went wrong.
5. **Resolves the SQLite volume by its real, project-prefixed name** and fails if it does not exist.
   The old code named the bare `jobtracker_data`, which on a real deploy would have silently *created*
   an empty volume and backed that up — a second instance of the same class of bug, found while
   testing the fix.

Two side effects of the same root cause are also resolved: `APP_PUBLIC_BASE_URL` now reaches the
script, so the post-deploy public smoke check actually runs (and the script says so explicitly when it
is unset rather than skipping in silence), as does `OLLAMA_MODEL` for the warmup.

*Verified against a seeded MariaDB 11 container and real Docker volumes:*

| Scenario | Result |
|---|---|
| MariaDB production-style `.env` | ✅ env loaded, MariaDB path selected, `.sql.gz` written containing `CREATE TABLE`, the seeded row, and the `Dump completed` trailer |
| `DATABASE_PROVIDER` missing | ✅ exits 1 naming the variable; nothing built, stopped or replaced; no backup file |
| `DATABASE_PROVIDER` unrecognised | ✅ exits 1 |
| `AI_SERVICE_TOKEN` / `AUTH_JWT_KEY` missing | ✅ both reported in one pass, exits 1 |
| SQLite with a populated volume | ✅ `.tar.gz` written and verified |
| SQLite volume containing no `jobtracker.db` | ✅ rejected, file deleted |
| SQLite volume name not present | ✅ rejected before any container ran; no stray empty volume created |
| Broken database credentials | ✅ exits 1, no partial file left behind |
| Truncated dump (no `CREATE TABLE`) | ✅ rejected |
| Dump with the trailer stripped | ✅ rejected as truncated |
| Secret leakage across every test's output | ✅ zero occurrences of any password, token or key |

### N2. ~~`/health` always reports `version: unknown` under Docker~~ — **CLOSED 2026-07-19**

*Original finding:* the endpoint read the `APP_VERSION` **environment variable**, but compose passes
`App__Version`, which binds to the `App:Version` **configuration key**. No variable by that name
existed in the container, so the version was always `unknown`.

**Closed.** `/health` now reads `App:Version` through `IConfiguration` — the approach
`AdminSystemController` already used for the same value. The resolution rule (configured version, else
assembly version) moved to a shared `BuildMetadata` helper rather than being written twice, so the
admin page and `/health` cannot drift apart.

*Verified against a running backend:* `App__Version=9.9.9-test` reports `9.9.9-test`; unset reports the
assembly version rather than `unknown`. Tests pin the configuration **key**, including that an
`App__Version` environment variable binds to `App:Version` — the original bug failed silently, so a
behavioural test alone would not have caught it.

---

### CV languages depended on the host's ICU data — **CLOSED 2026-07-19**

Found while investigating a reported CI failure. `HumanLanguageCatalog` built its lookup table solely
from `CultureInfo.GetCultures`, so which languages counted as human languages depended on the
machine's culture data rather than on the CV. Measured: **806 cultures** on a normal Windows or Linux
box, **exactly 1** under globalization-invariant mode.

Consequences, all silent — no error, no log line:

| Environment | Behaviour |
|---|---|
| Full ICU | Correct |
| Trimmed ICU data | Names present in the reduced set survive, the rest are dropped — a CV keeps English and loses Norwegian |
| Invariant mode | **Every** language dropped; a CV import loses its Languages section entirely |

**The tests were right and were not modified.** The catalog is now seeded explicitly with the languages
a CV realistically lists, before the culture enumeration, which still runs and still adds breadth.
Nothing in the seed collides with a technical skill — `Go`, `Java`, `Swift`, `Rust` and `Basic` are
deliberately absent, and `Basic` is also a proficiency level.

*Verified:* 420 tests pass in four environments — Windows and Linux, each with full ICU and with
`DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1`. Before the fix the invariant runs failed 5 tests. A new
`HumanLanguageCatalogTests` pins the seeded catalog, confirmed non-vacuous by removing the seed and
watching 15 tests fail.

**Why this matters beyond the bug:** the defect was invisible to a normal local test run. It is the
clearest evidence in this review that *passing locally* and *correct in the deployed container* are
different claims — which is exactly what B2 below leaves unresolved.

---

### Follow-up reminder index never created on MariaDB — **CLOSED 2026-07-19**

Found during the release-candidate audit, watching a real backend boot against an empty MariaDB.

`IX_JobApplications_OwnerUserId_FollowUpAt` was declared `(OwnerUserId(191), FollowUpAt)` with **no
prefix length on `FollowUpAt`**. But `FollowUpAt` is `text` on MariaDB — `JobApplications` is
migration-owned and the migration was scaffolded against SQLite, which stores `DateTimeOffset` as
`TEXT`. A text column cannot be indexed without a prefix length, so the index failed the 3072-byte key
check on **every** MariaDB boot, was caught by `TryCreateIndex`, and was silently skipped.

Two consequences, both real:

- The follow-up reminder query (`OwnerUserId + FollowUpAt`) ran unindexed — the index the code says it
  creates never existed.
- Every healthy boot logged `Specified key was too long` — the exact string
  `deploy/first-production-deployment.md` lists as a **stop-and-roll-back** signal. An operator
  following the runbook could abort a perfectly good deploy on a false alarm.

**Fixed** by prefixing `FollowUpAt(20)`, matching the `Status(50)` fix the author already applied one
line below for the identical longtext problem. ISO-8601 date strings sort lexicographically, so a
20-char prefix stays useful for the reminder scan.

*Verified* on a fresh empty MariaDB 11 container: index now created with both key parts, **zero**
"too long" lines, **zero** skipped indexes, **zero** unhandled exceptions, 42 tables, app healthy.
Audited the whole class — `FollowUpAt` was the only unprefixed text column in any reconciler composite
index; the datetime columns on reconciler-owned tables are `datetime(6)`.

### Timeline day-grouping test was nondeterministic — **CLOSED 2026-07-19**

`Timeline_groups_by_day_newest_first` seeded two "same day" events with
`DateTime.Now.AddDays(-3).AddHours(2)`, which crossed midnight whenever the wall clock was within two
hours of it — failing the test ~2 hours out of every 24, including in CI. The service is correct
(groups by `.Date`); the test was fixed to anchor the older events to `DateTime.Today` plus fixed
hours. Verified passing at 22:35 local (inside the failing window) and on Linux, both ICU modes.

---

## BLOCKED — requires external action

One item. Nothing in this repository can clear it.

### B2. CI is red — deployment is gated on it

Unchanged and still external to the repository. Commit `8f73548` changed one markdown file and its
test job failed in the same duration band as every other run; a change that cannot affect compilation
cannot fail a test job. The `deploy` job declares `needs: test`, so nothing promotes until this clears.

Everything in this review is therefore **local verification**. Blocked on, and needing you:

- **Job logs** — the Gitea API returns 401 unauthenticated. A read-scoped token would unblock this
- **`journalctl -u act_runner --since '2 hours ago'`** on the runner host
- **The act_runner container/service configuration**

Evidence in `docs/infrastructure/runner-investigation.md`.

**The decision this forces.** Either fix the runner, or deploy deliberately from a locally verified
commit with CI knowingly red. The second is a defensible choice for a self-hosted single-node
deployment — but it should be a decision rather than a default, and it means the 420 backend tests and
128 frontend tests have only ever run on one machine.

---

## Accepted for the first release (non-blocking)

Known limitations, deliberately shipped as-is. None should stop a deploy; all are worth knowing.

| # | Finding | Why it is not blocking |
|---|---|---|
| N1 | ~~**`.env.example` omits `DATABASE_PROVIDER` and `JOBTRACKER_CONNECTION_STRING`**~~ — **CLOSED 2026-07-19** | Both added to the template with the connection-string host caveat. `deploy.sh` now hard-fails without `DATABASE_PROVIDER`, so the template had to name it |
| N2 | ~~**`/health` always reports `"version":"unknown"` under Docker**~~ — **CLOSED 2026-07-19** | Fixed; see *Closed since the first pass* above |
| N3 | **The post-deploy gate in `deploy.sh` checks `.State == running`, not health** — a container can be `running` while `starting` or `unhealthy` | Harmless in practice: `compose up` already blocks on `service_healthy` for the frontend's dependency, so an unhealthy backend aborts the deploy before this check is reached. The check is weaker than it looks, not wrong |
| N4 | **Table-count discrepancy across documents** — `database-ownership.md` records 40 tables on MariaDB; `release-checklist.md` and the runbook say ~42 | Both were measured, at different points in Phase 5. Use "the tables listed in `database-ownership.md` all exist" as the check, not a number |
| N5 | **`ai-service` opens completely if `AI_SERVICE_TOKEN` is the empty string** — the middleware is `if AI_SERVICE_TOKEN and …`, so a blank token disables the check rather than failing closed | Compose declares the variable with `:?` on both services, so the stack refuses to start without it. No defence in depth behind that, though |
| N6 | **Logging is console-only** | Captured by `docker logs`; adequate for a single-node self-hosted deployment. No retention, structure or aggregation |
| N7 | **No global exception handler** | Unhandled errors return a bare 500 with no correlation id. No stack traces leak outside Development, so this is a supportability cost, not a security one |
| N8 | **`ClientErrorsController` is anonymous** | By design — browser error reports must work on pages reached before sign-in. Size-limited to 32 KB and field-truncated; stores nothing |
| N9 | **A code rollback does not recover rows written to the new tables** | Correct and documented. A *code* rollback loses nothing; only a *database restore* discards checklist items, cover letter versions, interview prep and AI history created since the dump |
| N10 | **Prompt quality is unmeasured** | Interview generation now receives analysis and match context. Whether the output is *better* is a judgement no test makes |

---

## MANUAL VERIFICATION — only the owner can do these

Nothing in this review, and nothing in the test suite, covers any of these — every automated check
stops at the authentication boundary, and none of it has touched production data.

### Post-deployment checklist — walk this in order

Each item names what "wrong" looks like, because "it loaded" is not a check.

| # | Area | Check | Wrong looks like |
|---|---|---|---|
| 1 | **Login** | Sign in with a real existing account | Password rejected, or the session drops on refresh — the latter means `AUTH_JWT_KEY` changed |
| 2 | **Existing applications** | The list loads and the **count matches the pre-deploy number** | Any drop. This is the single most important check on the page |
| 3 | **Application workspace** | Open one application: Overview, Checklist, Timeline, Analysis, Match | A section erroring. Empty is **correct** for applications that predate the feature |
| 4 | **Career profile** | Opens with your real experience, education, skills — and **languages** | Languages missing or reduced to English only. That is the bug fixed in `9681618`; if it reappears the container's ICU data differs from what was tested |
| 5 | **CV builder** | Lists existing variants; open one; it renders with its theme | A variant that opens blank, or loses its theme |
| 6 | **Public CV** | Open `/cv/<slug>` for an already-public variant, then **refresh it** | A 404 on refresh — that is SPA deep-link routing, not the CV |
| 7 | **AI features** | Run one generation (interview prep or cover letter) | A 5xx, or a hang. If `ai-service` is down the deploy still succeeds — it is not a deploy gate |
| 8 | **Attachments** | Download an existing attachment from an old application | A 404. Attachments live in the `jobtracker_data` volume; a missing file means the volume did not survive |

Then confirm `/health` reports the version you deployed rather than `1.0.0.0`.

### Backup and restore readiness

A full backup → verify → restore → start-the-app rehearsal was completed on 2026-07-19 and is recorded
in [`docs/operations/production-backup-verification.md`](operations/production-backup-verification.md):
42 tables dumped and restored into a clean MariaDB 11 container, every table's row count identical,
content and foreign keys intact, and the application started healthy against the restored database.

**That rehearsal used seeded data, not production data.** No production host was contacted. What
remains unproven is the one thing a rehearsal cannot prove: that it works **on your database**.

- [ ] **Take a MariaDB dump by hand from production and restore it into a scratch database.** Not a
      formality. Container verification says the mechanism is sound; it says nothing about your data
      volume, your disk space, or your MariaDB version's dump quirks.
- [ ] **Confirm `/opt/job-tracker/backups` exists and has room.** `deploy.sh` creates it, but a
      full disk fails the backup and therefore the deploy.
- [ ] **Know which backup is the restore point** before you start. After the deploy, confirm the newest
      file is `jobtracker-<database>-<stamp>.sql.gz` — a `jobtracker-sqlite-*.tar.gz` would mean the
      environment is wrong, though the provider check should now stop that first.
- [ ] **Check non-ASCII text survived the round trip.** Open a restored CV or career profile containing
      `æ`, `ø` or `å` and confirm it is not mangled. The rehearsal data was ASCII-heavy, so this is the
      most likely silent failure and the least likely to be noticed.

### Before deploying

- [ ] **Confirm `/opt/job-tracker/shared/.env` contains `DATABASE_PROVIDER=mariadb` and
      `JOBTRACKER_CONNECTION_STRING`.** `deploy.sh` now aborts without them, so a missing value costs
      an aborted deploy rather than a bad backup — but check first and skip the round trip.
- [ ] **Confirm the connection-string host resolves *from inside the backend container*** —
      `127.0.0.1` there means the container, not the Docker host.
- [ ] **Record the current commit** (`git rev-parse HEAD`) and the current row counts for
      `JobApplications` and `Companies`. The row counts are the check that matters most afterwards.

### After deploying — beyond the eight-point checklist above

- [ ] **Email and calendar integrations still connect**, if you use them — the OAuth callbacks were
      reviewed statically but never exercised against live Google or Microsoft.
- [ ] **The full journey once, deliberately** — empty profile through to a recorded outcome. No one has
      walked it. Friction in that path is currently unmapped.

### Why this section cannot shrink

Authenticated smoke testing is not an automation gap that more work would close. Signing in requires a
password, and no automated step in this repository should ever handle one. The 420 backend tests
verify that the authorization boundary *exists* and holds; only you can verify what is behind it, with
your data.