Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 83206df9d7 | |||
| 57fabe9a97 | |||
| b705cbaf60 | |||
| fba858e8eb |
@@ -0,0 +1,4 @@
|
||||
# Shell scripts must be LF: they run on the Linux deploy host and in Docker.
|
||||
# A CRLF deploy.sh fails with "bad interpreter: /usr/bin/env bash\r" or silently
|
||||
# mis-parses. This guarantees LF in every checkout regardless of core.autocrlf.
|
||||
*.sh text eol=lf
|
||||
@@ -103,6 +103,20 @@ public static class HumanLanguageCatalog
|
||||
map.TryAdd(normalizedAlias, normalizedCanonical);
|
||||
}
|
||||
|
||||
// Force an alias to a canonical, overriding whatever culture enumeration inserted for that
|
||||
// key. TryAdd is not enough here: on a host whose ICU data carries a "Norwegian Nynorsk"
|
||||
// culture, the enumeration below claims the key "nynorsk" -> "Norwegian Nynorsk" first, and a
|
||||
// later TryAdd("nynorsk", "Norwegian") silently loses. That made "nynorsk" resolve to
|
||||
// "Norwegian Nynorsk" on the CI runner but "Norwegian" locally — the same host-ICU dependence
|
||||
// this seeding exists to remove. Overrides must win regardless of insertion order.
|
||||
void Override(string alias, string canonical)
|
||||
{
|
||||
var normalizedAlias = NormalizeKey(alias);
|
||||
var normalizedCanonical = NormalizeDisplayName(canonical);
|
||||
if (string.IsNullOrWhiteSpace(normalizedAlias) || string.IsNullOrWhiteSpace(normalizedCanonical)) return;
|
||||
map[normalizedAlias] = normalizedCanonical;
|
||||
}
|
||||
|
||||
// Seeded FIRST, and deliberately not derived from the host.
|
||||
//
|
||||
// This table used to come only from CultureInfo.GetCultures, which returns whatever
|
||||
@@ -145,14 +159,20 @@ public static class HumanLanguageCatalog
|
||||
Add(native, english);
|
||||
}
|
||||
|
||||
Add("norsk", "Norwegian");
|
||||
Add("bokmal", "Norwegian");
|
||||
Add("bokmål", "Norwegian");
|
||||
Add("nynorsk", "Norwegian");
|
||||
Add("mandarin", "Chinese");
|
||||
Add("cantonese", "Chinese");
|
||||
Add("farsi", "Persian");
|
||||
Add("persian", "Persian");
|
||||
// These collapse regional/script variants and common exonyms to the umbrella language a CV
|
||||
// means. They must beat culture enumeration (see Override), because ICU carries "Norwegian
|
||||
// Bokmål"/"Norwegian Nynorsk" and "Chinese (Simplified/Traditional)" as their own cultures.
|
||||
Override("norsk", "Norwegian");
|
||||
Override("bokmal", "Norwegian");
|
||||
Override("bokmål", "Norwegian");
|
||||
Override("nynorsk", "Norwegian");
|
||||
Override("norwegian bokmal", "Norwegian");
|
||||
Override("norwegian bokmål", "Norwegian");
|
||||
Override("norwegian nynorsk", "Norwegian");
|
||||
Override("mandarin", "Chinese");
|
||||
Override("cantonese", "Chinese");
|
||||
Override("farsi", "Persian");
|
||||
Override("persian", "Persian");
|
||||
|
||||
return map;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
# Production backup & restore
|
||||
|
||||
> 2026-07-19. The backup **mechanism** is verified end to end against MariaDB 11 containers, including a
|
||||
> byte-exact Norwegian-character round trip. **No production database has been backed up from this
|
||||
> environment** — see *Production access*. This document is the checklist to run against production,
|
||||
> plus the evidence for what is already proven.
|
||||
|
||||
Companion to `deploy/deploy.sh` (the implementation), `deploy/README.md` (backup location/retention),
|
||||
and `docs/operations/production-backup-verification.md` (the earlier container rehearsal).
|
||||
|
||||
## Production backup checklist
|
||||
|
||||
Run top to bottom. Every item is checked automatically by `deploy.sh` **before** it builds or replaces
|
||||
anything — this list is for a manual pre-flight and for an out-of-band backup.
|
||||
|
||||
- [ ] **`DATABASE_PROVIDER=mariadb`** in `/opt/job-tracker/shared/.env`. No default — `deploy.sh`
|
||||
aborts if it is missing. If it were wrong, the SQLite path would run and back up the wrong thing.
|
||||
- [ ] **`JOBTRACKER_CONNECTION_STRING` present** and pointing at the production database. The host
|
||||
resolves from the deploy shell, not from inside a container, for the backup step.
|
||||
- [ ] **`deploy.sh` loads the environment.** It parses `/opt/job-tracker/shared/.env` into its own
|
||||
shell before deciding anything (verified: the "Loaded deployment environment" line prints first).
|
||||
- [ ] **Backup runs before replacement.** `validate_deploy_config` → `backup_database` → build →
|
||||
`up -d --force-recreate`. Confirmed by line order in `deploy.sh` (validate/backup precede
|
||||
build/replace). A failed backup aborts the deploy with the running stack untouched.
|
||||
- [ ] **Backup validation works.** The dump must be valid gzip, contain `CREATE TABLE`, and end with
|
||||
the `-- Dump completed` trailer; otherwise the file is deleted and the deploy stops. A SQLite
|
||||
archive must contain `jobtracker.db`.
|
||||
- [ ] **Filename confirms the type.** `jobtracker-<db>-<UTC>.sql.gz` = MariaDB dump.
|
||||
A `jobtracker-sqlite-*.tar.gz` on a MariaDB host means the environment is wrong.
|
||||
- [ ] **Password never on the command line** — `deploy.sh` passes it via `MYSQL_PWD`.
|
||||
|
||||
## Restore procedure
|
||||
|
||||
Restore is a **separate** operation from code rollback. A bad deploy usually needs only the rollback;
|
||||
restore the database **only if the data itself is wrong**, because it discards everything written since
|
||||
the dump.
|
||||
|
||||
```bash
|
||||
# 1. Restore into a SEPARATE, empty database first — never straight over the live one.
|
||||
gzip -dc /opt/job-tracker/backups/jobtracker-<db>-<UTC>.sql.gz \
|
||||
| MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker_scratch
|
||||
|
||||
# 2. Sanity-check row counts and character fidelity (below), THEN, if replacing production:
|
||||
docker compose stop backend
|
||||
gzip -dc <backup> | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 jobtracker
|
||||
docker compose start backend
|
||||
```
|
||||
|
||||
Always pass `--default-character-set=utf8mb4` on both dump and restore so multibyte text is not
|
||||
mangled.
|
||||
|
||||
## UTF-8 / Norwegian character verification — **VERIFIED 2026-07-19**
|
||||
|
||||
The earlier rehearsal used ASCII-only seed data, so character fidelity was unproven. It has now been
|
||||
tested explicitly through the real `deploy.sh` backup path.
|
||||
|
||||
Seeded into a MariaDB 11 database on the real 42-table schema:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Company name | `Ærøskøbing Systemutvikling AS` |
|
||||
| CV variant name | `Søknad – Bjørn Håkonsen` (note the em-dash `–`) |
|
||||
| User email | `bjørn@dåg.no` |
|
||||
| Career profile JSON | `Erfaren utvikler frå Tromsø … Språk: norsk … flåten … Morsmål` |
|
||||
|
||||
Backed up with the real `backup_database` function, restored into a **clean** MariaDB 11 container,
|
||||
compared byte-for-byte:
|
||||
|
||||
```
|
||||
source HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
|
||||
restored HEX(Name): C38672C3B8736BC3B862696E672053797374656D757476696B6C696E67204153
|
||||
BYTE-EXACT MATCH
|
||||
```
|
||||
|
||||
`C386` = `Æ`, `C3B8` = `ø`, `C3A5` = `å` — correct UTF-8, not double-encoded or stripped. **æ ø å**,
|
||||
plus the em-dash, survived the full dump → gzip → restore cycle unchanged.
|
||||
|
||||
To repeat this against production after a restore:
|
||||
|
||||
```bash
|
||||
MYSQL_PWD='<pw>' mariadb --host=<host> --user=<user> --default-character-set=utf8mb4 -N -B jobtracker_scratch \
|
||||
-e "SELECT Name FROM Companies WHERE Name LIKE '%ø%' OR Name LIKE '%æ%' OR Name LIKE '%å%' LIMIT 5;"
|
||||
# Read the output in a UTF-8 terminal. Mangled output (æ, ø) = charset problem in the pipeline.
|
||||
```
|
||||
|
||||
## Production access
|
||||
|
||||
**This environment has no route to the production database** — no `/opt/job-tracker`, no production
|
||||
connection string, and the local stack runs SQLite. Per the task constraints, **no credential
|
||||
discovery and no SSH guessing were attempted.**
|
||||
|
||||
**Manual step the owner must perform** (only the owner has production access):
|
||||
|
||||
1. On the production host, run one out-of-band backup: `deploy/deploy.sh` takes one automatically, or
|
||||
dump by hand with the command in `deploy/README.md`.
|
||||
2. Restore that dump into a **scratch** database (not production) and confirm:
|
||||
- table count (~42) and row counts for `AspNetUsers`, `JobApplications`, `Companies`,
|
||||
`CareerProfiles` match production;
|
||||
- a real record containing `æ`/`ø`/`å` reads back correctly (the check above).
|
||||
|
||||
Until that is done, backup/restore is proven **on the mechanism and on synthetic Norwegian data**, not
|
||||
on the production dataset.
|
||||
@@ -0,0 +1,72 @@
|
||||
# Manual smoke test — post-deployment
|
||||
|
||||
> Run this after every production deploy, in a real browser, signed in as a real user.
|
||||
>
|
||||
> **Authentication requires the owner.** No step here is automated and no password is handled by any
|
||||
> tool or script — signing in is the owner's job. The automated suite verifies the authorization
|
||||
> *boundary* exists (every user endpoint returns 401 unauthenticated); only a human with credentials
|
||||
> can verify what is behind it.
|
||||
|
||||
Each item names what **wrong** looks like, because "it loaded" is not a check. Stop and consider
|
||||
rollback on any ✗.
|
||||
|
||||
## Authentication
|
||||
|
||||
- [ ] **Login succeeds** with an existing account. ✗ = password rejected, or a 5xx on submit.
|
||||
- [ ] **Existing session works** — reload the page after login and stay signed in. ✗ = session drops
|
||||
on refresh, which means `AUTH_JWT_KEY` changed between deploys.
|
||||
- [ ] **Logout works** and returns to the signed-out state. ✗ = still authenticated after logout.
|
||||
|
||||
## Applications
|
||||
|
||||
- [ ] **Existing applications load.** ✗ = empty list for a user who had applications.
|
||||
- [ ] **Counts are correct** — the number matches what you saw before the deploy. ✗ = any drop. This
|
||||
is the single most important check.
|
||||
- [ ] **Workspace opens** for one application — Overview, Checklist, Timeline, Analysis, Match render.
|
||||
✗ = a section erroring. Empty new sections on old applications are **correct**, not a fault.
|
||||
|
||||
## Career profile
|
||||
|
||||
- [ ] **Profile loads** with real experience, education and skills.
|
||||
- [ ] **Languages are present.** ✗ = languages missing, or reduced to English only. That is the
|
||||
`9681618` / `fba858e` ICU-dependence class of bug; if it reappears the container's ICU differs
|
||||
from what was tested. Check a profile that lists Norwegian specifically.
|
||||
- [ ] **Structured career data is intact** — experiences, education, projects, certifications all
|
||||
show their fields, not blanks.
|
||||
- [ ] **No data loss from the relational migration** — spot-check a profile edited before the deploy
|
||||
against what you remember. ✗ = fields silently emptied.
|
||||
|
||||
## CV builder
|
||||
|
||||
- [ ] **Existing CV variants load** in the builder list; open one.
|
||||
- [ ] **Editing works** — change a field, confirm autosave persists after reload.
|
||||
- [ ] **Preview works** — the themed preview renders the variant.
|
||||
- [ ] **PDF export works** — export produces a valid PDF (this exercises the in-container Chromium).
|
||||
✗ = export hangs or errors, usually a Chromium/`CV_PDF_BROWSER_PATH` problem.
|
||||
- [ ] **Public CV loads directly after refresh** — open `/cv/<slug>` for an already-public variant,
|
||||
then **hard-refresh**. ✗ = 404 on refresh, which is SPA deep-link routing, not the CV itself.
|
||||
|
||||
## AI features
|
||||
|
||||
- [ ] **AI generation works** — run one generation (interview prep or cover letter). ✗ = 5xx or an
|
||||
indefinite hang. If `ai-service` is down the *deploy* still succeeds (AI is not a deploy gate),
|
||||
so this must be checked by hand.
|
||||
- [ ] **Suggestions are generated** and shown for review.
|
||||
- [ ] **No unwanted writes occur** — the generation does **not** modify the Career Profile, a CV
|
||||
variant, or application fields until you explicitly save. Confirm the source records are
|
||||
unchanged after generating but before saving. ✗ = anything written without your action.
|
||||
|
||||
## Files
|
||||
|
||||
- [ ] **Attachment upload** — upload a file to an application; it appears in the list.
|
||||
- [ ] **Attachment download** — download an existing attachment from an old application. ✗ = 404,
|
||||
which means the `jobtracker_data` volume did not survive the deploy.
|
||||
- [ ] **Permissions** — confirm you cannot reach another user's attachment. Signed in as user A,
|
||||
requesting user B's attachment id must return 404/403, never the file. (The automated suite
|
||||
already asserts tenant scoping; this is the human confirmation.)
|
||||
|
||||
## After the checklist
|
||||
|
||||
- [ ] `/health` reports the version you deployed, not `1.0.0.0` (the assembly fallback means
|
||||
`APP_VERSION` did not reach the container).
|
||||
- [ ] Row counts for applications and companies still match the pre-deploy numbers.
|
||||
@@ -1,9 +1,17 @@
|
||||
# Infrastructure Investigation — CI runner and deploy failures
|
||||
|
||||
> 2026-07-18. Supersedes `docs/ci-runner-investigation.md`.
|
||||
> **Conclusion: both failures are outside the repository.** Application code has been eliminated as a
|
||||
> cause by direct experiment. Confirmation and repair require host access — the exact asks are at the
|
||||
> end.
|
||||
> **Conclusion (2026-07-18): failures A and B are outside the repository.** Application code was
|
||||
> eliminated as a cause for those two by direct experiment. Confirmation and repair require host
|
||||
> access — the exact asks are at the end.
|
||||
>
|
||||
> **Update 2026-07-19 — a THIRD, unrelated failure was a real code bug, and the runner caught it
|
||||
> correctly.** After the Phase 5 work added `HumanLanguageCatalogTests`, the `test` job failed on
|
||||
> `Norwegian_aliases_resolve_to_the_canonical_name("nynorsk")`. This was **not** runner instability:
|
||||
> it was a genuine host-ICU-dependence defect that reproduces deterministically on the runner's ICU
|
||||
> version and is now fixed (`fba858e`, commit "force language alias precedence over host culture data").
|
||||
> Full evidence in *Finding C* below. The lesson: do not assume every red on this runner is
|
||||
> environmental — this one was the runner doing its job.
|
||||
|
||||
There are **two independent infrastructure failures**:
|
||||
|
||||
@@ -188,6 +196,67 @@ changed host key / rotated `PROD_SSH_KEY`. Confidence: **medium (~50 %)** — th
|
||||
|
||||
---
|
||||
|
||||
## C — Backend test fails on the runner's ICU version (real code bug, FIXED 2026-07-19)
|
||||
|
||||
Unlike A and B, this failure **was** in the application code. The runner reported it correctly.
|
||||
|
||||
### Symptom
|
||||
|
||||
```
|
||||
JobTrackerApi.Tests.HumanLanguageCatalogTests.Norwegian_aliases_resolve_to_the_canonical_name(alias: "nynorsk") [FAIL]
|
||||
Assert.Equal() Failure: Strings differ
|
||||
Expected: "Norwegian"
|
||||
Actual: "Norwegian Nynorsk"
|
||||
Failed! - Failed: 1, Passed: 419
|
||||
```
|
||||
|
||||
Passed on the author's Windows machine and in Debian/Ubuntu-22.04+ containers; failed only on the
|
||||
runner. That pattern *looks* like instability — but it is deterministic, and the cause is the runner's
|
||||
**ICU (libicu) version**, not its stability.
|
||||
|
||||
### Root cause — ruled in, not assumed
|
||||
|
||||
`HumanLanguageCatalog.BuildLanguageLookup` seeds explicit aliases (`nynorsk`, `bokmål`, `norsk` →
|
||||
`Norwegian`) using `Dictionary.TryAdd`, *after* enumerating `CultureInfo.GetCultures`. `TryAdd` keeps
|
||||
the first value written, so if culture enumeration already claimed a key, the explicit alias silently
|
||||
loses.
|
||||
|
||||
The `nn` (Norwegian Nynorsk) culture's **NativeName differs by libicu version**:
|
||||
|
||||
| ICU | `nn` NativeName (cleaned) | key `nynorsk` populated by enumeration? | old code result |
|
||||
|---|---|---|---|
|
||||
| libicu66 (Ubuntu 20.04) | `nynorsk` | **yes** → `Norwegian Nynorsk` | `nynorsk` → **`Norwegian Nynorsk`** ❌ |
|
||||
| libicu70 (Ubuntu 22.04) | `norsk nynorsk` | no | `nynorsk` → `Norwegian` ✓ |
|
||||
| libicu72 (Debian 12) | `norsk nynorsk` | no | `nynorsk` → `Norwegian` ✓ |
|
||||
| libicu74 (Ubuntu 24.04) | `norsk nynorsk` | no | `nynorsk` → `Norwegian` ✓ |
|
||||
|
||||
So the runner is on **old ICU (libicu66 / Ubuntu 20.04-class)**. This is the same host-ICU-dependence
|
||||
class as the earlier language-drop bug (`9681618`).
|
||||
|
||||
### Evidence (reproduced end to end)
|
||||
|
||||
- **Probe on `runtime:6.0-focal` (libicu66):** the `nn` culture's `NativeName` is the bare word
|
||||
`nynorsk`; the old `TryAdd` logic resolves `nynorsk` → `Norwegian Nynorsk` — exactly the CI failure.
|
||||
- **Same probe with the fix logic on the same libicu66:** `nynorsk` → `Norwegian`.
|
||||
- **The real net9 test DLL on Ubuntu 20.04 / libicu66, .NET 9.0.316 installed via `dotnet-install.sh`
|
||||
(identical to CI):** 420/420 pass with the fix.
|
||||
- Also 420/420 on libicu72 (Debian) and libicu74 (Ubuntu 24.04), and locally on Windows.
|
||||
|
||||
### Fix
|
||||
|
||||
`fba858e` — an `Override` helper (`map[key] = value`) applied to the alias block so the explicit
|
||||
mappings win regardless of what culture enumeration inserted. Correct for any ICU version by
|
||||
construction; no test weakened.
|
||||
|
||||
### Why this matters for A and B
|
||||
|
||||
It does not exonerate the runner — A (test host dies at ~3 s) and B (deploy SSH fails at ~3 s) remain
|
||||
separately evidenced as environmental. But it is a caution: **not every red on this runner is
|
||||
infrastructure.** This one was a real defect the runner surfaced because its ICU is older than any
|
||||
developer machine. Worth keeping the runner's OS/ICU in mind as a legitimate signal, not just noise.
|
||||
|
||||
---
|
||||
|
||||
## Required infrastructure changes
|
||||
|
||||
**Blocking — cannot proceed without one of these:**
|
||||
|
||||
@@ -28,6 +28,13 @@ index that never created on MariaDB (and logged a false rollback signal on every
|
||||
nondeterministic timeline test. Both are recorded below. Backup → restore → app-start was re-run
|
||||
end to end. No open blocker remains in the code or deployment path.
|
||||
|
||||
**Correction 2026-07-19 (later).** The next CI run then failed on a *third*, real code bug — a
|
||||
language-alias lookup that depended on the runner's older ICU version (`nynorsk` → `Norwegian Nynorsk`
|
||||
instead of `Norwegian`). This was **not** runner instability; the runner caught a genuine defect. Fixed
|
||||
in `fba858e` and verified on the runner's exact ICU (Ubuntu 20.04 / libicu66). So B2 below is no longer
|
||||
"purely external": the most recent red was ours. See `docs/infrastructure/runner-investigation.md`
|
||||
Finding C. The current status is carried by `docs/release-final-report.md`.
|
||||
|
||||
**Verdict: one blocker remains, and it is external — CI (B2).** Nothing in the application or the
|
||||
deployment path is now known to be blocking. Everything verified here is *local* verification; CI has
|
||||
proven none of it.
|
||||
@@ -102,9 +109,9 @@ precede 375 and 382.
|
||||
|
||||
| Order | Step | Where |
|
||||
|---|---|---|
|
||||
| 1 | `ReconcileSchema()` — pass 1, repair and reconciler-owned tables | `StartupInitializationExtensions.cs:1965` |
|
||||
| 2 | `Database.Migrate()` — migration-owned tables | `:1973` |
|
||||
| 3 | `ReconcileSchema()` — pass 2, everything pass 1 had to skip | `:1984` |
|
||||
| 1 | `ReconcileSchema()` — pass 1, repair and reconciler-owned tables | `StartupInitializationExtensions.cs:1972` |
|
||||
| 2 | `Database.Migrate()` — migration-owned tables | `:1980` |
|
||||
| 3 | `ReconcileSchema()` — pass 2, everything pass 1 had to skip | `:1991` |
|
||||
|
||||
**Migrations expected to run: none that create anything.** All seven Phase 4/5 migrations were
|
||||
confirmed to have a **literally empty `Up` body** (0 statements each): `AddCareerProfileRelationalChildren`,
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
# Final release report
|
||||
|
||||
> 2026-07-19. Compiled at the release-candidate stage after the CI runner failure was investigated and
|
||||
> a real code bug behind it was fixed. Consolidates the verification done across the release-hardening
|
||||
> work. Every claim here was verified this session unless explicitly marked as owner-only or unproven.
|
||||
|
||||
## Release status
|
||||
|
||||
### READY WITH DOCUMENTED RISKS
|
||||
|
||||
The code and deployment path have no known blocker. The most recent CI failure — which had been assumed
|
||||
environmental — was found to be a **real code bug** and is fixed and verified on the runner's exact ICU
|
||||
version. What remains are (a) the runner's separately-evidenced instability, which the owner must
|
||||
confirm is gone by re-running CI, and (b) owner-only verification that cannot be automated (sign-in,
|
||||
production data, production backup). None of these is a code defect.
|
||||
|
||||
## Verification summary
|
||||
|
||||
### Backend
|
||||
- **Tests:** 420 passing, Release configuration, real build (not `--no-build`).
|
||||
- Windows (local).
|
||||
- Linux, full ICU — Debian 12 / libicu72 and Ubuntu 24.04 / libicu74.
|
||||
- Linux under `DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1`.
|
||||
- **Ubuntu 20.04 / libicu66 with .NET 9.0.316 installed via `dotnet-install.sh` — the exact CI
|
||||
runner environment — 420/420.** This is the environment that failed before the ICU fix.
|
||||
- **Build:** backend Docker image builds.
|
||||
- No test weakened, skipped, filtered, or disabled.
|
||||
|
||||
### Frontend
|
||||
- **Tests:** 128 passing (36 suites).
|
||||
- **TypeScript:** `tsc --noEmit` clean.
|
||||
- **Build:** production build clean; frontend Docker image builds.
|
||||
|
||||
### Database
|
||||
All four startup scenarios run against live MariaDB 11 and SQLite, with the real backend binary:
|
||||
|
||||
| Scenario | Result |
|
||||
|---|---|
|
||||
| Empty MariaDB | 42 tables created, healthy, 0 exceptions, 0 skipped indexes |
|
||||
| Populated MariaDB restart | idempotent — 42 tables, rows preserved |
|
||||
| Partially-migrated MariaDB | healed 35 → 42, 7 Phase 4/5 tables recreated, surviving rows preserved |
|
||||
| Fresh SQLite + restart | schema built, idempotent, 0 errors |
|
||||
|
||||
Startup order confirmed in logs: **reconcile → `Database.Migrate()` → reconcile → seed**. All seven
|
||||
Phase 4/5 migrations have empty `Up` bodies; the reconciler owns their DDL. No duplicate ownership.
|
||||
|
||||
### Deployment
|
||||
- **Backup:** the real `deploy.sh backup_database` produces a valid gzip dump with 42 `CREATE TABLE`
|
||||
statements and the `Dump completed` trailer. Validation rejects empty/truncated/wrong-type files.
|
||||
Backup runs **before** build/replace (verified by line order).
|
||||
- **Restore:** dump restored into a clean MariaDB 11 container; all 42 tables and every row count
|
||||
matched; the application started healthy against the restored database.
|
||||
- **UTF-8 / Norwegian characters:** `Ærøskøbing`, `Søknad – Bjørn Håkonsen`, `bjørn@dåg.no`, and
|
||||
career-profile Norwegian text survived a full backup → restore **byte-exact** (HEX compared).
|
||||
- **Health checks:** `/health` 200 (anonymous, no DB touch); frontend `wget`; compose gates frontend
|
||||
on backend `service_healthy`; 4 healthcheck blocks defined.
|
||||
|
||||
### Security
|
||||
- **Authorization:** every user-owned controller has class-level `[Authorize]`; 5 user endpoints return
|
||||
401 unauthenticated; independent of the `Auth:Require` flag.
|
||||
- **Tenant isolation:** 25 global query filters, all deny-on-null.
|
||||
- **Public endpoints:** `/health` 200, `api/auth/config` 200, public CV unknown slug 404 (not 401/500,
|
||||
no leak). `ai-service` unreachable from outside its private network; authenticates its caller.
|
||||
|
||||
## Remaining risks
|
||||
|
||||
Separated by kind. Nothing here is hidden.
|
||||
|
||||
### Code issues
|
||||
- **None open.** Fixed this release cycle: the ICU language-alias precedence bug (`fba858e`), the
|
||||
MariaDB follow-up-reminder index (`95646e1`), the nondeterministic timeline test (`c1ff98f`), the CV
|
||||
language ICU drop (`9681618`), `/health` version (`8f6f2ba`), and the deploy backup env-loading
|
||||
(`66b02bc`).
|
||||
|
||||
### Infrastructure issues
|
||||
- **Runner instability (Findings A & B)** — separate from the ICU bug and still unconfirmed-fixed. The
|
||||
test host has died at ~3 s and the deploy SSH step at ~3 s on prior runs, with host-access-only
|
||||
diagnostics. **The ICU fix removes the known code cause of red, but does not prove the runner is
|
||||
stable.** The owner should re-run CI; if it still dies at a fixed ~3 s mark regardless of the change,
|
||||
Findings A/B are confirmed and need host access (`journalctl -u act_runner`, runner resources).
|
||||
- **Runner ICU is old (libicu66 / Ubuntu 20.04-class).** Legitimate as a signal — it caught a real bug
|
||||
— but worth upgrading so the CI environment is closer to production and to developer machines.
|
||||
|
||||
### Manual owner verification (cannot be automated)
|
||||
- **Sign-in and the authenticated journey** — requires a password; no tool here handles one. Use
|
||||
`docs/deployment/manual-smoke-test.md`.
|
||||
- **Production backup on real data** — no production access from this environment; no credentials were
|
||||
discovered or guessed. Owner must run one real backup + scratch restore, including the æøå check.
|
||||
See `docs/deployment/backup-restore.md`.
|
||||
- **Production dataset scale** — dump duration, disk headroom, and lock behaviour are unproven on a
|
||||
large real database.
|
||||
|
||||
## Deployment recommendation
|
||||
|
||||
**Deploy with documented risks**, in this order:
|
||||
|
||||
1. **Re-run CI** with the ICU fix included. If the `test` job now passes, the current known red is
|
||||
cleared. If it still dies at a fixed early mark, Findings A/B are confirmed environmental — decide
|
||||
whether to deploy manually from the verified commit or fix the runner first.
|
||||
2. **Owner runs a production backup + scratch restore** (with the æøå check) before deploying.
|
||||
3. **Deploy** via `deploy/deploy.sh` — backup is automatic and gates the deploy.
|
||||
4. **Owner runs `docs/deployment/manual-smoke-test.md`** immediately after.
|
||||
|
||||
Do not treat a green pipeline as sufficient on its own: two real defects this cycle passed local test
|
||||
runs and only surfaced against a real container or the runner's older ICU. The manual smoke test is
|
||||
not optional.
|
||||
Reference in New Issue
Block a user