chore: pin shell scripts to LF via .gitattributes
deploy.sh and the other scripts run on the Linux deploy host and in Docker. A CRLF checkout breaks them with "bad interpreter: bash\r". The committed blobs are already LF, but nothing guaranteed it against a host with core.autocrlf=true. `*.sh text eol=lf` makes it explicit. Verified: git ls-files --eol shows attr/text eol=lf on all .sh files; renormalize produced no index churn (already LF). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
cesnimda
@@ -1,9 +1,17 @@
|
||||
# Infrastructure Investigation — CI runner and deploy failures
|
||||
|
||||
> 2026-07-18. Supersedes `docs/ci-runner-investigation.md`.
|
||||
> **Conclusion: both failures are outside the repository.** Application code has been eliminated as a
|
||||
> cause by direct experiment. Confirmation and repair require host access — the exact asks are at the
|
||||
> end.
|
||||
> **Conclusion (2026-07-18): failures A and B are outside the repository.** Application code was
|
||||
> eliminated as a cause for those two by direct experiment. Confirmation and repair require host
|
||||
> access — the exact asks are at the end.
|
||||
>
|
||||
> **Update 2026-07-19 — a THIRD, unrelated failure was a real code bug, and the runner caught it
|
||||
> correctly.** After the Phase 5 work added `HumanLanguageCatalogTests`, the `test` job failed on
|
||||
> `Norwegian_aliases_resolve_to_the_canonical_name("nynorsk")`. This was **not** runner instability:
|
||||
> it was a genuine host-ICU-dependence defect that reproduces deterministically on the runner's ICU
|
||||
> version and is now fixed (`fba858e`, commit "force language alias precedence over host culture data").
|
||||
> Full evidence in *Finding C* below. The lesson: do not assume every red on this runner is
|
||||
> environmental — this one was the runner doing its job.
|
||||
|
||||
There are **two independent infrastructure failures**:
|
||||
|
||||
@@ -188,6 +196,67 @@ changed host key / rotated `PROD_SSH_KEY`. Confidence: **medium (~50 %)** — th
|
||||
|
||||
---
|
||||
|
||||
## C — Backend test fails on the runner's ICU version (real code bug, FIXED 2026-07-19)
|
||||
|
||||
Unlike A and B, this failure **was** in the application code. The runner reported it correctly.
|
||||
|
||||
### Symptom
|
||||
|
||||
```
|
||||
JobTrackerApi.Tests.HumanLanguageCatalogTests.Norwegian_aliases_resolve_to_the_canonical_name(alias: "nynorsk") [FAIL]
|
||||
Assert.Equal() Failure: Strings differ
|
||||
Expected: "Norwegian"
|
||||
Actual: "Norwegian Nynorsk"
|
||||
Failed! - Failed: 1, Passed: 419
|
||||
```
|
||||
|
||||
Passed on the author's Windows machine and in Debian/Ubuntu-22.04+ containers; failed only on the
|
||||
runner. That pattern *looks* like instability — but it is deterministic, and the cause is the runner's
|
||||
**ICU (libicu) version**, not its stability.
|
||||
|
||||
### Root cause — ruled in, not assumed
|
||||
|
||||
`HumanLanguageCatalog.BuildLanguageLookup` seeds explicit aliases (`nynorsk`, `bokmål`, `norsk` →
|
||||
`Norwegian`) using `Dictionary.TryAdd`, *after* enumerating `CultureInfo.GetCultures`. `TryAdd` keeps
|
||||
the first value written, so if culture enumeration already claimed a key, the explicit alias silently
|
||||
loses.
|
||||
|
||||
The `nn` (Norwegian Nynorsk) culture's **NativeName differs by libicu version**:
|
||||
|
||||
| ICU | `nn` NativeName (cleaned) | key `nynorsk` populated by enumeration? | old code result |
|
||||
|---|---|---|---|
|
||||
| libicu66 (Ubuntu 20.04) | `nynorsk` | **yes** → `Norwegian Nynorsk` | `nynorsk` → **`Norwegian Nynorsk`** ❌ |
|
||||
| libicu70 (Ubuntu 22.04) | `norsk nynorsk` | no | `nynorsk` → `Norwegian` ✓ |
|
||||
| libicu72 (Debian 12) | `norsk nynorsk` | no | `nynorsk` → `Norwegian` ✓ |
|
||||
| libicu74 (Ubuntu 24.04) | `norsk nynorsk` | no | `nynorsk` → `Norwegian` ✓ |
|
||||
|
||||
So the runner is on **old ICU (libicu66 / Ubuntu 20.04-class)**. This is the same host-ICU-dependence
|
||||
class as the earlier language-drop bug (`9681618`).
|
||||
|
||||
### Evidence (reproduced end to end)
|
||||
|
||||
- **Probe on `runtime:6.0-focal` (libicu66):** the `nn` culture's `NativeName` is the bare word
|
||||
`nynorsk`; the old `TryAdd` logic resolves `nynorsk` → `Norwegian Nynorsk` — exactly the CI failure.
|
||||
- **Same probe with the fix logic on the same libicu66:** `nynorsk` → `Norwegian`.
|
||||
- **The real net9 test DLL on Ubuntu 20.04 / libicu66, .NET 9.0.316 installed via `dotnet-install.sh`
|
||||
(identical to CI):** 420/420 pass with the fix.
|
||||
- Also 420/420 on libicu72 (Debian) and libicu74 (Ubuntu 24.04), and locally on Windows.
|
||||
|
||||
### Fix
|
||||
|
||||
`fba858e` — an `Override` helper (`map[key] = value`) applied to the alias block so the explicit
|
||||
mappings win regardless of what culture enumeration inserted. Correct for any ICU version by
|
||||
construction; no test weakened.
|
||||
|
||||
### Why this matters for A and B
|
||||
|
||||
It does not exonerate the runner — A (test host dies at ~3 s) and B (deploy SSH fails at ~3 s) remain
|
||||
separately evidenced as environmental. But it is a caution: **not every red on this runner is
|
||||
infrastructure.** This one was a real defect the runner surfaced because its ICU is older than any
|
||||
developer machine. Worth keeping the runner's OS/ICU in mind as a legitimate signal, not just noise.
|
||||
|
||||
---
|
||||
|
||||
## Required infrastructure changes
|
||||
|
||||
**Blocking — cannot proceed without one of these:**
|
||||
|
||||
@@ -28,6 +28,13 @@ index that never created on MariaDB (and logged a false rollback signal on every
|
||||
nondeterministic timeline test. Both are recorded below. Backup → restore → app-start was re-run
|
||||
end to end. No open blocker remains in the code or deployment path.
|
||||
|
||||
**Correction 2026-07-19 (later).** The next CI run then failed on a *third*, real code bug — a
|
||||
language-alias lookup that depended on the runner's older ICU version (`nynorsk` → `Norwegian Nynorsk`
|
||||
instead of `Norwegian`). This was **not** runner instability; the runner caught a genuine defect. Fixed
|
||||
in `fba858e` and verified on the runner's exact ICU (Ubuntu 20.04 / libicu66). So B2 below is no longer
|
||||
"purely external": the most recent red was ours. See `docs/infrastructure/runner-investigation.md`
|
||||
Finding C. The current status is carried by `docs/release-final-report.md`.
|
||||
|
||||
**Verdict: one blocker remains, and it is external — CI (B2).** Nothing in the application or the
|
||||
deployment path is now known to be blocking. Everything verified here is *local* verification; CI has
|
||||
proven none of it.
|
||||
@@ -102,9 +109,9 @@ precede 375 and 382.
|
||||
|
||||
| Order | Step | Where |
|
||||
|---|---|---|
|
||||
| 1 | `ReconcileSchema()` — pass 1, repair and reconciler-owned tables | `StartupInitializationExtensions.cs:1965` |
|
||||
| 2 | `Database.Migrate()` — migration-owned tables | `:1973` |
|
||||
| 3 | `ReconcileSchema()` — pass 2, everything pass 1 had to skip | `:1984` |
|
||||
| 1 | `ReconcileSchema()` — pass 1, repair and reconciler-owned tables | `StartupInitializationExtensions.cs:1972` |
|
||||
| 2 | `Database.Migrate()` — migration-owned tables | `:1980` |
|
||||
| 3 | `ReconcileSchema()` — pass 2, everything pass 1 had to skip | `:1991` |
|
||||
|
||||
**Migrations expected to run: none that create anything.** All seven Phase 4/5 migrations were
|
||||
confirmed to have a **literally empty `Up` body** (0 statements each): `AddCareerProfileRelationalChildren`,
|
||||
|
||||
Reference in New Issue
Block a user