Compare commits

..

4 Commits

Author SHA1 Message Date
cesnimda 83206df9d7 docs(release): final release report
CI and Deploy / test (push) Successful in 2m47s
CI and Deploy / deploy (push) Failing after 7s
Consolidates release-candidate verification: backend 420 (incl. the CI
runner's exact Ubuntu 20.04/libicu66 environment), frontend 128, all
four DB scenarios, backup/restore with a byte-exact æøå round trip,
health and auth checks.

Status: READY WITH DOCUMENTED RISKS. No open code blocker. Remaining
risks separated into code (none), infrastructure (runner instability
A/B, still unconfirmed-fixed; old runner ICU), and manual owner
verification (sign-in, production backup, production scale).

Recommendation: deploy with documented risks -- re-run CI with the ICU
fix, owner runs a real backup + scratch restore, deploy, then run the
manual smoke test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 23:15:41 +02:00
cesnimda 57fabe9a97 docs(deployment): backup-restore, smoke test, runner finding C
- docs/deployment/backup-restore.md: production backup checklist; documents
  that deploy.sh loads the env, validates before backup, and validates the
  dump. Adds VERIFIED UTF-8/Norwegian-character round trip (æ ø å survive a
  real deploy.sh backup -> restore byte-exact; HEX compared). States plainly
  that no production database was reached and the owner must run one real
  backup + scratch restore.
- docs/deployment/manual-smoke-test.md: owner-run post-deploy checklist
  (auth, applications, career profile, CV builder, AI, files). Each item
  names what "wrong" looks like. Documents that login requires the owner.
- runner-investigation.md: Finding C -- the latest CI red was a real ICU
  code bug the runner caught correctly, not instability. Amends the blanket
  "outside the repository" conclusion. A and B stand as separate env issues.
- release-candidate-review.md: corrected drifted line refs after the index
  fix; noted the CI ICU finding so the "purely external" verdict is honest.

All claims reflect behaviour verified this session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 23:14:41 +02:00
cesnimda b705cbaf60 chore: pin shell scripts to LF via .gitattributes
deploy.sh and the other scripts run on the Linux deploy host and in
Docker. A CRLF checkout breaks them with "bad interpreter: bash\r".
The committed blobs are already LF, but nothing guaranteed it against a
host with core.autocrlf=true. `*.sh text eol=lf` makes it explicit.

Verified: git ls-files --eol shows attr/text eol=lf on all .sh files;
renormalize produced no index churn (already LF).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 23:14:27 +02:00
cesnimda fba858e8eb fix(cv): force language alias precedence over host culture data
The CI backend test job failed on HumanLanguageCatalogTests:
  nynorsk -> expected "Norwegian", actual "Norwegian Nynorsk"

This was a real bug, correctly caught by the runner -- not runner
instability. Reproduced on Ubuntu 20.04 / libicu66 (the CI runner's ICU)
with .NET 9 installed via dotnet-install.sh exactly as CI does.

Root cause: BuildLanguageLookup's explicit normalization aliases
(nynorsk/bokmål/norsk -> Norwegian) were added with map.TryAdd, which
loses to any key the culture enumeration already inserted. On libicu66
the "nn" culture's NativeName is the bare word "nynorsk", so enumeration
claimed key "nynorsk" -> "Norwegian Nynorsk" first and the explicit alias
silently lost. On libicu70+ (Debian/Ubuntu 22.04+, my earlier local
runs) the native name is "norsk nynorsk", so the key was free and the
alias won -- which is why it passed locally and only failed on the
runner's older ICU. Same host-ICU dependence class as 9681618.

Fix: add an Override helper (map[key] = value) and apply it to the
alias block so these mappings win regardless of insertion order. Also
collapse the full "Norwegian Nynorsk"/"Norwegian Bokmål" phrases to
"Norwegian" for consistency. Correct by construction for any ICU version.

Verified:
- reproduced the exact failure on libicu66 with the old code (probe)
- fix logic yields nynorsk/bokmål -> Norwegian on that same libicu66
- real net9 test DLL: 420/420 on focal libicu66 (CI mirror, dotnet
  9.0.316 via dotnet-install.sh), and 420/420 on libicu72 (Debian) and
  libicu74 (Ubuntu 24.04), plus locally
No test weakened, skipped, or relaxed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 23:08:37 +02:00
7 changed files with 394 additions and 14 deletions
+4
View File
@@ -0,0 +1,4 @@
# Shell scripts must be LF: they run on the Linux deploy host and in Docker.
# A CRLF deploy.sh fails with "bad interpreter: /usr/bin/env bash\r" or silently
# mis-parses. This guarantees LF in every checkout regardless of core.autocrlf.
*.sh text eol=lf
+28 -8
View File
@@ -103,6 +103,20 @@ public static class HumanLanguageCatalog
map.TryAdd(normalizedAlias, normalizedCanonical);
}
// Force an alias to a canonical, overriding whatever culture enumeration inserted for that
// key. TryAdd is not enough here: on a host whose ICU data carries a "Norwegian Nynorsk"
// culture, the enumeration below claims the key "nynorsk" -> "Norwegian Nynorsk" first, and a
// later TryAdd("nynorsk", "Norwegian") silently loses. That made "nynorsk" resolve to
// "Norwegian Nynorsk" on the CI runner but "Norwegian" locally — the same host-ICU dependence
// this seeding exists to remove. Overrides must win regardless of insertion order.
void Override(string alias, string canonical)
{
var normalizedAlias = NormalizeKey(alias);
var normalizedCanonical = NormalizeDisplayName(canonical);
if (string.IsNullOrWhiteSpace(normalizedAlias) || string.IsNullOrWhiteSpace(normalizedCanonical)) return;
map[normalizedAlias] = normalizedCanonical;
}
// Seeded FIRST, and deliberately not derived from the host.
//
// This table used to come only from CultureInfo.GetCultures, which returns whatever
@@ -145,14 +159,20 @@ public static class HumanLanguageCatalog
Add(native, english);
}
Add("norsk", "Norwegian");
Add("bokmal", "Norwegian");
Add("bokmål", "Norwegian");
Add("nynorsk", "Norwegian");
Add("mandarin", "Chinese");
Add("cantonese", "Chinese");
Add("farsi", "Persian");
Add("persian", "Persian");
// These collapse regional/script variants and common exonyms to the umbrella language a CV
// means. They must beat culture enumeration (see Override), because ICU carries "Norwegian
// Bokmål"/"Norwegian Nynorsk" and "Chinese (Simplified/Traditional)" as their own cultures.
Override("norsk", "Norwegian");
Override("bokmal", "Norwegian");
Override("bokmål", "Norwegian");
Override("nynorsk", "Norwegian");
Override("norwegian bokmal", "Norwegian");
Override("norwegian bokmål", "Norwegian");
Override("norwegian nynorsk", "Norwegian");
Override("mandarin", "Chinese");
Override("cantonese", "Chinese");
Override("farsi", "Persian");
Override("persian", "Persian");
return map;
}
+102
View File
@@ -0,0 +1,102 @@
# Production backup & restore
> 2026-07-19. The backup **mechanism** is verified end to end against MariaDB 11 containers, including a
> byte-exact Norwegian-character round trip. **No production database has been backed up from this
> environment** — see *Production access*. This document is the checklist to run against production,
> plus the evidence for what is already proven.
Companion to `deploy/deploy.sh` (the implementation), `deploy/README.md` (backup location/retention),
and `docs/operations/production-backup-verification.md` (the earlier container rehearsal).
## Production backup checklist
Run top to bottom. Every item is checked automatically by `deploy.sh` **before** it builds or replaces
anything — this list is for a manual pre-flight and for an out-of-band backup.
- [ ] **`DATABASE_PROVIDER=mariadb`** in `/opt/job-tracker/shared/.env`. No default — `deploy.sh`
aborts if it is missing. If it were wrong, the SQLite path would run and back up the wrong thing.
- [ ] **`JOBTRACKER_CONNECTION_STRING` present** and pointing at the production database. The host
resolves from the deploy shell, not from inside a container, for the backup step.
- [ ] **`deploy.sh` loads the environment.** It parses `/opt/job-tracker/shared/.env` into its own
shell before deciding anything (verified: the "Loaded deployment environment" line prints first).
- [ ] **Backup runs before replacement.** `validate_deploy_config``backup_database` → build →
`up -d --force-recreate`. Confirmed by line order in `deploy.sh` (validate/backup precede
build/replace). A failed backup aborts the deploy with the running stack untouched.
- [ ] **Backup validation works.** The dump must be valid gzip, contain `CREATE TABLE`, and end with
the `-- Dump completed` trailer; otherwise the file is deleted and the deploy stops. A SQLite
archive must contain `jobtracker.db`.
- [ ] **Filename confirms the type.** `jobtracker-<db>-<UTC>.sql.gz` = MariaDB dump.
A `jobtracker-sqlite-*.tar.gz` on a MariaDB host means the environment is wrong.
- [ ] **Password never on the command line**`deploy.sh` passes it via `MYSQL_PWD`.
## Restore procedure
Restore is a **separate** operation from code rollback. A bad deploy usually needs only the rollback;
restore the database **only if the data itself is wrong**, because it discards everything written since
the dump.
```bash
# 1. Restore into a SEPARATE, empty database first — never straight over the live one.
gzip -dc /opt/job-tracker/backups/jobtracker-<db>-<UTC>.sql.gz \
| MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker_scratch
# 2. Sanity-check row counts and character fidelity (below), THEN, if replacing production:
docker compose stop backend
gzip -dc <backup> | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker
docker compose start backend
```
Always pass `--default-character-set=utf8mb4` on both dump and restore so multibyte text is not
mangled.
## UTF-8 / Norwegian character verification — **VERIFIED 2026-07-19**
The earlier rehearsal used ASCII-only seed data, so character fidelity was unproven. It has now been
tested explicitly through the real `deploy.sh` backup path.
Seeded into a MariaDB 11 database on the real 42-table schema:
| Field | Value |
|---|---|
| Company name | `Ærøskøbing Systemutvikling AS` |
| CV variant name | `Søknad Bjørn Håkonsen` (note the em-dash ``) |
| User email | `bjørn@dåg.no` |
| Career profile JSON | `Erfaren utvikler frå Tromsø … Språk: norsk … flåten … Morsmål` |
Backed up with the real `backup_database` function, restored into a **clean** MariaDB 11 container,
compared byte-for-byte:
```
source HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
restored HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
BYTE-EXACT MATCH
```
`C386` = `Æ`, `C3B8` = `ø`, `C3A5` = `å` — correct UTF-8, not double-encoded or stripped. **æ ø å**,
plus the em-dash, survived the full dump → gzip → restore cycle unchanged.
To repeat this against production after a restore:
```bash
MYSQL_PWD='<pw>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 -N -B jobtracker_scratch \
-e "SELECT Name FROM Companies WHERE Name LIKE '%ø%' OR Name LIKE '%æ%' OR Name LIKE '%å%' LIMIT 5;"
# Read the output in a UTF-8 terminal. Mangled output (æ, ø) = charset problem in the pipeline.
```
## Production access
**This environment has no route to the production database** — no `/opt/job-tracker`, no production
connection string, and the local stack runs SQLite. Per the task constraints, **no credential
discovery and no SSH guessing were attempted.**
**Manual step the owner must perform** (only the owner has production access):
1. On the production host, run one out-of-band backup: `deploy/deploy.sh` takes one automatically, or
dump by hand with the command in `deploy/README.md`.
2. Restore that dump into a **scratch** database (not production) and confirm:
- table count (~42) and row counts for `AspNetUsers`, `JobApplications`, `Companies`,
`CareerProfiles` match production;
- a real record containing `æ`/`ø`/`å` reads back correctly (the check above).
Until that is done, backup/restore is proven **on the mechanism and on synthetic Norwegian data**, not
on the production dataset.
+72
View File
@@ -0,0 +1,72 @@
# Manual smoke test — post-deployment
> Run this after every production deploy, in a real browser, signed in as a real user.
>
> **Authentication requires the owner.** No step here is automated and no password is handled by any
> tool or script — signing in is the owner's job. The automated suite verifies the authorization
> *boundary* exists (every user endpoint returns 401 unauthenticated); only a human with credentials
> can verify what is behind it.
Each item names what **wrong** looks like, because "it loaded" is not a check. Stop and consider
rollback on any ✗.
## Authentication
- [ ] **Login succeeds** with an existing account. ✗ = password rejected, or a 5xx on submit.
- [ ] **Existing session works** — reload the page after login and stay signed in. ✗ = session drops
on refresh, which means `AUTH_JWT_KEY` changed between deploys.
- [ ] **Logout works** and returns to the signed-out state. ✗ = still authenticated after logout.
## Applications
- [ ] **Existing applications load.** ✗ = empty list for a user who had applications.
- [ ] **Counts are correct** — the number matches what you saw before the deploy. ✗ = any drop. This
is the single most important check.
- [ ] **Workspace opens** for one application — Overview, Checklist, Timeline, Analysis, Match render.
✗ = a section erroring. Empty new sections on old applications are **correct**, not a fault.
## Career profile
- [ ] **Profile loads** with real experience, education and skills.
- [ ] **Languages are present.** ✗ = languages missing, or reduced to English only. That is the
`9681618` / `fba858e` ICU-dependence class of bug; if it reappears the container's ICU differs
from what was tested. Check a profile that lists Norwegian specifically.
- [ ] **Structured career data is intact** — experiences, education, projects, certifications all
show their fields, not blanks.
- [ ] **No data loss from the relational migration** — spot-check a profile edited before the deploy
against what you remember. ✗ = fields silently emptied.
## CV builder
- [ ] **Existing CV variants load** in the builder list; open one.
- [ ] **Editing works** — change a field, confirm autosave persists after reload.
- [ ] **Preview works** — the themed preview renders the variant.
- [ ] **PDF export works** — export produces a valid PDF (this exercises the in-container Chromium).
✗ = export hangs or errors, usually a Chromium/`CV_PDF_BROWSER_PATH` problem.
- [ ] **Public CV loads directly after refresh** — open `/cv/<slug>` for an already-public variant,
then **hard-refresh**. ✗ = 404 on refresh, which is SPA deep-link routing, not the CV itself.
## AI features
- [ ] **AI generation works** — run one generation (interview prep or cover letter). ✗ = 5xx or an
indefinite hang. If `ai-service` is down the *deploy* still succeeds (AI is not a deploy gate),
so this must be checked by hand.
- [ ] **Suggestions are generated** and shown for review.
- [ ] **No unwanted writes occur** — the generation does **not** modify the Career Profile, a CV
variant, or application fields until you explicitly save. Confirm the source records are
unchanged after generating but before saving. ✗ = anything written without your action.
## Files
- [ ] **Attachment upload** — upload a file to an application; it appears in the list.
- [ ] **Attachment download** — download an existing attachment from an old application. ✗ = 404,
which means the `jobtracker_data` volume did not survive the deploy.
- [ ] **Permissions** — confirm you cannot reach another user's attachment. Signed in as user A,
requesting user B's attachment id must return 404/403, never the file. (The automated suite
already asserts tenant scoping; this is the human confirmation.)
## After the checklist
- [ ] `/health` reports the version you deployed, not `1.0.0.0` (the assembly fallback means
`APP_VERSION` did not reach the container).
- [ ] Row counts for applications and companies still match the pre-deploy numbers.
+72 -3
View File
@@ -1,9 +1,17 @@
# Infrastructure Investigation — CI runner and deploy failures
> 2026-07-18. Supersedes `docs/ci-runner-investigation.md`.
> **Conclusion: both failures are outside the repository.** Application code has been eliminated as a
> cause by direct experiment. Confirmation and repair require host access — the exact asks are at the
> end.
> **Conclusion (2026-07-18): failures A and B are outside the repository.** Application code was
> eliminated as a cause for those two by direct experiment. Confirmation and repair require host
> access — the exact asks are at the end.
>
> **Update 2026-07-19 — a THIRD, unrelated failure was a real code bug, and the runner caught it
> correctly.** After the Phase 5 work added `HumanLanguageCatalogTests`, the `test` job failed on
> `Norwegian_aliases_resolve_to_the_canonical_name("nynorsk")`. This was **not** runner instability:
> it was a genuine host-ICU-dependence defect that reproduces deterministically on the runner's ICU
> version and is now fixed (`fba858e`, commit "force language alias precedence over host culture data").
> Full evidence in *Finding C* below. The lesson: do not assume every red on this runner is
> environmental — this one was the runner doing its job.
There are **two independent infrastructure failures**:
@@ -188,6 +196,67 @@ changed host key / rotated `PROD_SSH_KEY`. Confidence: **medium (~50 %)** — th
---
## C — Backend test fails on the runner's ICU version (real code bug, FIXED 2026-07-19)
Unlike A and B, this failure **was** in the application code. The runner reported it correctly.
### Symptom
```
JobTrackerApi.Tests.HumanLanguageCatalogTests.Norwegian_aliases_resolve_to_the_canonical_name(alias: "nynorsk") [FAIL]
Assert.Equal() Failure: Strings differ
Expected: "Norwegian"
Actual: "Norwegian Nynorsk"
Failed! - Failed: 1, Passed: 419
```
Passed on the author's Windows machine and in Debian/Ubuntu-22.04+ containers; failed only on the
runner. That pattern *looks* like instability — but it is deterministic, and the cause is the runner's
**ICU (libicu) version**, not its stability.
### Root cause — ruled in, not assumed
`HumanLanguageCatalog.BuildLanguageLookup` seeds explicit aliases (`nynorsk`, `bokmål`, `norsk`
`Norwegian`) using `Dictionary.TryAdd`, *after* enumerating `CultureInfo.GetCultures`. `TryAdd` keeps
the first value written, so if culture enumeration already claimed a key, the explicit alias silently
loses.
The `nn` (Norwegian Nynorsk) culture's **NativeName differs by libicu version**:
| ICU | `nn` NativeName (cleaned) | key `nynorsk` populated by enumeration? | old code result |
|---|---|---|---|
| libicu66 (Ubuntu 20.04) | `nynorsk` | **yes**`Norwegian Nynorsk` | `nynorsk`**`Norwegian Nynorsk`** ❌ |
| libicu70 (Ubuntu 22.04) | `norsk nynorsk` | no | `nynorsk``Norwegian` ✓ |
| libicu72 (Debian 12) | `norsk nynorsk` | no | `nynorsk``Norwegian` ✓ |
| libicu74 (Ubuntu 24.04) | `norsk nynorsk` | no | `nynorsk``Norwegian` ✓ |
So the runner is on **old ICU (libicu66 / Ubuntu 20.04-class)**. This is the same host-ICU-dependence
class as the earlier language-drop bug (`9681618`).
### Evidence (reproduced end to end)
- **Probe on `runtime:6.0-focal` (libicu66):** the `nn` culture's `NativeName` is the bare word
`nynorsk`; the old `TryAdd` logic resolves `nynorsk``Norwegian Nynorsk` — exactly the CI failure.
- **Same probe with the fix logic on the same libicu66:** `nynorsk``Norwegian`.
- **The real net9 test DLL on Ubuntu 20.04 / libicu66, .NET 9.0.316 installed via `dotnet-install.sh`
(identical to CI):** 420/420 pass with the fix.
- Also 420/420 on libicu72 (Debian) and libicu74 (Ubuntu 24.04), and locally on Windows.
### Fix
`fba858e` — an `Override` helper (`map[key] = value`) applied to the alias block so the explicit
mappings win regardless of what culture enumeration inserted. Correct for any ICU version by
construction; no test weakened.
### Why this matters for A and B
It does not exonerate the runner — A (test host dies at ~3 s) and B (deploy SSH fails at ~3 s) remain
separately evidenced as environmental. But it is a caution: **not every red on this runner is
infrastructure.** This one was a real defect the runner surfaced because its ICU is older than any
developer machine. Worth keeping the runner's OS/ICU in mind as a legitimate signal, not just noise.
---
## Required infrastructure changes
**Blocking — cannot proceed without one of these:**
+10 -3
View File
@@ -28,6 +28,13 @@ index that never created on MariaDB (and logged a false rollback signal on every
nondeterministic timeline test. Both are recorded below. Backup → restore → app-start was re-run
end to end. No open blocker remains in the code or deployment path.
**Correction 2026-07-19 (later).** The next CI run then failed on a *third*, real code bug — a
language-alias lookup that depended on the runner's older ICU version (`nynorsk``Norwegian Nynorsk`
instead of `Norwegian`). This was **not** runner instability; the runner caught a genuine defect. Fixed
in `fba858e` and verified on the runner's exact ICU (Ubuntu 20.04 / libicu66). So B2 below is no longer
"purely external": the most recent red was ours. See `docs/infrastructure/runner-investigation.md`
Finding C. The current status is carried by `docs/release-final-report.md`.
**Verdict: one blocker remains, and it is external — CI (B2).** Nothing in the application or the
deployment path is now known to be blocking. Everything verified here is *local* verification; CI has
proven none of it.
@@ -102,9 +109,9 @@ precede 375 and 382.
| Order | Step | Where |
|---|---|---|
| 1 | `ReconcileSchema()` — pass 1, repair and reconciler-owned tables | `StartupInitializationExtensions.cs:1965` |
| 2 | `Database.Migrate()` — migration-owned tables | `:1973` |
| 3 | `ReconcileSchema()` — pass 2, everything pass 1 had to skip | `:1984` |
| 1 | `ReconcileSchema()` — pass 1, repair and reconciler-owned tables | `StartupInitializationExtensions.cs:1972` |
| 2 | `Database.Migrate()` — migration-owned tables | `:1980` |
| 3 | `ReconcileSchema()` — pass 2, everything pass 1 had to skip | `:1991` |
**Migrations expected to run: none that create anything.** All seven Phase 4/5 migrations were
confirmed to have a **literally empty `Up` body** (0 statements each): `AddCareerProfileRelationalChildren`,
+106
View File
@@ -0,0 +1,106 @@
# Final release report
> 2026-07-19. Compiled at the release-candidate stage after the CI runner failure was investigated and
> a real code bug behind it was fixed. Consolidates the verification done across the release-hardening
> work. Every claim here was verified this session unless explicitly marked as owner-only or unproven.
## Release status
### READY WITH DOCUMENTED RISKS
The code and deployment path have no known blocker. The most recent CI failure — which had been assumed
environmental — was found to be a **real code bug** and is fixed and verified on the runner's exact ICU
version. What remains are (a) the runner's separately-evidenced instability, which the owner must
confirm is gone by re-running CI, and (b) owner-only verification that cannot be automated (sign-in,
production data, production backup). None of these is a code defect.
## Verification summary
### Backend
- **Tests:** 420 passing, Release configuration, real build (not `--no-build`).
- Windows (local).
- Linux, full ICU — Debian 12 / libicu72 and Ubuntu 24.04 / libicu74.
- Linux under `DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1`.
- **Ubuntu 20.04 / libicu66 with .NET 9.0.316 installed via `dotnet-install.sh` — the exact CI
runner environment — 420/420.** This is the environment that failed before the ICU fix.
- **Build:** backend Docker image builds.
- No test weakened, skipped, filtered, or disabled.
### Frontend
- **Tests:** 128 passing (36 suites).
- **TypeScript:** `tsc --noEmit` clean.
- **Build:** production build clean; frontend Docker image builds.
### Database
All four startup scenarios run against live MariaDB 11 and SQLite, with the real backend binary:
| Scenario | Result |
|---|---|
| Empty MariaDB | 42 tables created, healthy, 0 exceptions, 0 skipped indexes |
| Populated MariaDB restart | idempotent — 42 tables, rows preserved |
| Partially-migrated MariaDB | healed 35 → 42, 7 Phase 4/5 tables recreated, surviving rows preserved |
| Fresh SQLite + restart | schema built, idempotent, 0 errors |
Startup order confirmed in logs: **reconcile → `Database.Migrate()` → reconcile → seed**. All seven
Phase 4/5 migrations have empty `Up` bodies; the reconciler owns their DDL. No duplicate ownership.
### Deployment
- **Backup:** the real `deploy.sh backup_database` produces a valid gzip dump with 42 `CREATE TABLE`
statements and the `Dump completed` trailer. Validation rejects empty/truncated/wrong-type files.
Backup runs **before** build/replace (verified by line order).
- **Restore:** dump restored into a clean MariaDB 11 container; all 42 tables and every row count
matched; the application started healthy against the restored database.
- **UTF-8 / Norwegian characters:** `Ærøskøbing`, `Søknad Bjørn Håkonsen`, `bjørn@dåg.no`, and
career-profile Norwegian text survived a full backup → restore **byte-exact** (HEX compared).
- **Health checks:** `/health` 200 (anonymous, no DB touch); frontend `wget`; compose gates frontend
on backend `service_healthy`; 4 healthcheck blocks defined.
### Security
- **Authorization:** every user-owned controller has class-level `[Authorize]`; 5 user endpoints return
401 unauthenticated; independent of the `Auth:Require` flag.
- **Tenant isolation:** 25 global query filters, all deny-on-null.
- **Public endpoints:** `/health` 200, `api/auth/config` 200, public CV unknown slug 404 (not 401/500,
no leak). `ai-service` unreachable from outside its private network; authenticates its caller.
## Remaining risks
Separated by kind. Nothing here is hidden.
### Code issues
- **None open.** Fixed this release cycle: the ICU language-alias precedence bug (`fba858e`), the
MariaDB follow-up-reminder index (`95646e1`), the nondeterministic timeline test (`c1ff98f`), the CV
language ICU drop (`9681618`), `/health` version (`8f6f2ba`), and the deploy backup env-loading
(`66b02bc`).
### Infrastructure issues
- **Runner instability (Findings A & B)** — separate from the ICU bug and still unconfirmed-fixed. The
test host has died at ~3 s and the deploy SSH step at ~3 s on prior runs, with host-access-only
diagnostics. **The ICU fix removes the known code cause of red, but does not prove the runner is
stable.** The owner should re-run CI; if it still dies at a fixed ~3 s mark regardless of the change,
Findings A/B are confirmed and need host access (`journalctl -u act_runner`, runner resources).
- **Runner ICU is old (libicu66 / Ubuntu 20.04-class).** Legitimate as a signal — it caught a real bug
— but worth upgrading so the CI environment is closer to production and to developer machines.
### Manual owner verification (cannot be automated)
- **Sign-in and the authenticated journey** — requires a password; no tool here handles one. Use
`docs/deployment/manual-smoke-test.md`.
- **Production backup on real data** — no production access from this environment; no credentials were
discovered or guessed. Owner must run one real backup + scratch restore, including the æøå check.
See `docs/deployment/backup-restore.md`.
- **Production dataset scale** — dump duration, disk headroom, and lock behaviour are unproven on a
large real database.
## Deployment recommendation
**Deploy with documented risks**, in this order:
1. **Re-run CI** with the ICU fix included. If the `test` job now passes, the current known red is
cleared. If it still dies at a fixed early mark, Findings A/B are confirmed environmental — decide
whether to deploy manually from the verified commit or fix the runner first.
2. **Owner runs a production backup + scratch restore** (with the æøå check) before deploying.
3. **Deploy** via `deploy/deploy.sh` — backup is automatic and gates the deploy.
4. **Owner runs `docs/deployment/manual-smoke-test.md`** immediately after.
Do not treat a green pipeline as sufficient on its own: two real defects this cycle passed local test
runs and only surfaced against a real container or the runner's older ICU. The manual smoke test is
not optional.