docs(ops): record release-candidate audit findings

Adds the two issues found and fixed during the release-candidate audit to release-candidate-review.md: the follow-up reminder index that never created on MariaDB (fix in the preceding commit), and the nondeterministic timeline day-grouping test. Corrects database-ownership.md drift: the MariaDB startup scenarios now report 42 tables (measured in every scenario this audit), not the stale 40 from before the last Phase 5 tables were added, and adds the partially-migrated heal scenario (35 -> 42) that was verified. All claims reflect behaviour verified in this audit: 420 tests on Windows and Linux in both ICU modes, all three Docker images built, four database startup scenarios against live MariaDB 11 and SQLite, and a full backup -> restore -> app-start cycle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix(db): create follow-up reminder index on MariaDB
2026-07-19 22:53:43 +02:00 · 2026-07-19 22:52:39 +02:00 · 2026-07-19 22:52:24 +02:00 · 2026-07-19 21:36:26 +02:00 · 2026-07-19 21:33:27 +02:00 · 2026-07-19 19:19:17 +02:00
15 changed files with 1143 additions and 66 deletions
@@ -1,6 +1,18 @@
 # Copy this file to `.env` (same folder as docker-compose.yml) and fill in values.
 #
 # Used by docker-compose.yml
 #
 # Database. deploy/deploy.sh REQUIRES DATABASE_PROVIDER to be set explicitly and
 # refuses to deploy without it — it selects which backup to take, and guessing it
 # wrong means backing up the wrong database. Use `mariadb` (or `mysql`) for a
 # server deployment, `sqlite` for a single-file local one.
 DATABASE_PROVIDER=sqlite
 # Required when DATABASE_PROVIDER is mariadb/mysql. Ignored for sqlite, which
 # stores its file in the jobtracker_data volume.
 # The host resolves from INSIDE the backend container: 127.0.0.1 means the
 # container, not the Docker host.
 JOBTRACKER_CONNECTION_STRING=
 AUTH_JWT_KEY=CHANGE_ME_LONG_RANDOM_SECRET
 AUTH_ADMIN_EMAIL=admin@example.com
 AUTH_ADMIN_PASSWORD=CHANGE_ME_STRONG_PASSWORD
@@ -115,8 +115,11 @@ public sealed class ApplicationIntelligenceTests
        var (db, _, timeline) = New("user-1");
        await using var _d = db;
        var job = await SeedJobAsync(db, "user-1");
-        db.JobEvents.Add(new JobEvent { JobApplicationId = job.Id, Type = "Created", At = DateTime.Now.AddDays(-3) });
+        // Anchor the two older events to a fixed time-of-day so they always land on the same
-        db.JobEvents.Add(new JobEvent { JobApplicationId = job.Id, Type = "AiRefreshed", At = DateTime.Now.AddDays(-3).AddHours(2) });
+        // calendar day. Using DateTime.Now.AddDays(-3).AddHours(2) straddled midnight whenever the
        // wall clock was within two hours of it, splitting one day into two and failing the test.
        db.JobEvents.Add(new JobEvent { JobApplicationId = job.Id, Type = "Created", At = DateTime.Today.AddDays(-3).AddHours(9) });
        db.JobEvents.Add(new JobEvent { JobApplicationId = job.Id, Type = "AiRefreshed", At = DateTime.Today.AddDays(-3).AddHours(11) });
        db.JobEvents.Add(new JobEvent { JobApplicationId = job.Id, Type = "ReplyReceived", At = DateTime.Now });
        await db.SaveChangesAsync();
@@ -0,0 +1,91 @@
 using JobTrackerApi.Services;
 using Microsoft.Extensions.Configuration;
 using Xunit;
 namespace JobTrackerApi.Tests;
 // Pins the configuration KEY, not just the behaviour. /health previously read the
 // APP_VERSION environment variable while docker-compose passed App__Version, so the
 // endpoint silently reported "unknown" in every container. Nothing failed; the value
 // was just wrong. These tests fail if the key drifts again.
 public sealed class BuildMetadataTests
 {
    private static IConfiguration Config(params (string Key, string Value)[] values) =>
        new ConfigurationBuilder()
            .AddInMemoryCollection(values.Select(v => new KeyValuePair<string, string?>(v.Key, v.Value)))
            .Build();
    [Fact]
    public void Reads_the_App_Version_configuration_key()
    {
        var version = BuildMetadata.ResolveVersion(Config(("App:Version", "1.2.3")));
        Assert.Equal("1.2.3", version);
    }
    [Fact]
    public void App__Version_environment_variable_binds_to_the_same_key()
    {
        // This is the exact path docker-compose uses: App__Version in the container
        // environment must reach App:Version in configuration.
        var cfg = new ConfigurationBuilder()
            .AddInMemoryCollection(new Dictionary<string, string?>())
            .Add(new FakeEnvironmentSource(("App__Version", "42")))
            .Build();
        Assert.Equal("42", BuildMetadata.ResolveVersion(cfg));
    }
    [Fact]
    public void Falls_back_to_the_assembly_version_when_unconfigured()
    {
        // Local development sets nothing. "unknown" would be a regression, not a default.
        var version = BuildMetadata.ResolveVersion(Config());
        Assert.False(string.IsNullOrWhiteSpace(version));
        Assert.NotEqual("unknown", version);
    }
    [Theory]
    [InlineData("$(git rev-parse --short HEAD)")]
    [InlineData("${APP_COMMIT_SHA}")]
    [InlineData("   ")]
    [InlineData(null)]
    public void Unresolved_placeholders_and_blanks_are_rejected(string? raw)
    {
        Assert.Null(BuildMetadata.Normalize(raw));
    }
    [Fact]
    public void A_real_value_survives_normalization_trimmed()
    {
        Assert.Equal("abc1234", BuildMetadata.Normalize("  abc1234  "));
    }
    // Minimal stand-in for the environment-variable provider, so the test does not
    // mutate the real process environment and race other tests.
    private sealed class FakeEnvironmentSource : IConfigurationSource
    {
        private readonly (string Key, string Value)[] _values;
        public FakeEnvironmentSource(params (string Key, string Value)[] values) => _values = values;
        public IConfigurationProvider Build(IConfigurationBuilder builder) => new Provider(_values);
        private sealed class Provider : ConfigurationProvider
        {
            private readonly (string Key, string Value)[] _values;
            public Provider((string Key, string Value)[] values) => _values = values;
            public override void Load()
            {
                foreach (var (key, value) in _values)
                {
                    // "__" is the environment-variable spelling of the ":" section separator.
                    Data[key.Replace("__", ":")] = value;
                }
            }
        }
    }
 }
@@ -0,0 +1,57 @@
 using JobTrackerApi.Models;
 using Xunit;
 namespace JobTrackerApi.Tests;
 // The catalog used to be built solely from CultureInfo.GetCultures, so which languages it
 // recognised depended on the host's ICU data: 806 cultures on a normal machine, exactly 1
 // under globalization-invariant mode. CV imports silently lost their Languages section on a
 // container with trimmed or absent ICU data, and every existing test still passed locally.
 //
 // These pin the explicitly seeded catalog, so deleting the seed list fails here rather than
 // in production on a machine nobody tested.
 public sealed class HumanLanguageCatalogTests
 {
    [Theory]
    [InlineData("English")]
    [InlineData("Norwegian")]
    [InlineData("French")]
    [InlineData("Spanish")]
    [InlineData("German")]
    [InlineData("Arabic")]
    [InlineData("Chinese")]
    [InlineData("Polish")]
    public void Common_languages_resolve_without_relying_on_host_culture_data(string language)
    {
        Assert.Equal(language, HumanLanguageCatalog.NormalizeLanguageName(language));
    }
    [Theory]
    [InlineData("norsk")]
    [InlineData("bokmål")]
    [InlineData("nynorsk")]
    public void Norwegian_aliases_resolve_to_the_canonical_name(string alias)
    {
        Assert.Equal("Norwegian", HumanLanguageCatalog.NormalizeLanguageName(alias));
    }
    [Theory]
    [InlineData("C#")]
    [InlineData("Leadership")]
    [InlineData("Public speaking")]
    [InlineData("Go")]      // a programming language, not a human one
    [InlineData("Java")]    // Javanese is a language; Java is not
    [InlineData("Swift")]
    [InlineData("Rust")]
    public void Technical_skills_are_not_treated_as_human_languages(string skill)
    {
        Assert.Null(HumanLanguageCatalog.NormalizeLanguageName(skill));
    }
    [Fact]
    public void A_language_embedded_in_a_phrase_is_extracted_with_its_level()
    {
        Assert.Equal("Norwegian", HumanLanguageCatalog.NormalizeLanguageName("Native Norwegian speaker"));
        Assert.Equal("Native", HumanLanguageCatalog.ExtractLevel("Native Norwegian speaker"));
    }
 }
@@ -1,4 +1,3 @@
 using System.Reflection;
 using System.Runtime.InteropServices;
 using JobTrackerApi.Data;
 using JobTrackerApi.Services;
@@ -50,20 +49,8 @@ public sealed class AdminSystemController : ControllerBase
        AiServiceMetrics Ai
    );
-    private static string? NormalizeBuildMetadata(string? value)
+    // Shared with the /health endpoint so both report the same version from the same key.
-    {
+    private static string? NormalizeBuildMetadata(string? value) => BuildMetadata.Normalize(value);
        var trimmed = (value ?? string.Empty).Trim();
        if (string.IsNullOrWhiteSpace(trimmed)) return null;
        // Ignore unresolved shell/compose placeholders that would otherwise leak
        // directly into the admin UI, e.g. $(git rev-parse --short HEAD) or ${APP_COMMIT_SHA}.
        if ((trimmed.StartsWith("$(") && trimmed.EndsWith(")")) || (trimmed.StartsWith("${") && trimmed.EndsWith("}")))
        {
            return null;
        }
        return trimmed;
    }
    private EmailSettingsSnapshot BuildFallbackEmailSettingsSnapshot()
    {
@@ -221,11 +208,7 @@ public sealed class AdminSystemController : ControllerBase
                LastError: ex.Message);
        }
-        var version = NormalizeBuildMetadata(_cfg["App:Version"]);
+        var version = BuildMetadata.ResolveVersion(_cfg);
        if (string.IsNullOrWhiteSpace(version))
        {
            version = Assembly.GetExecutingAssembly().GetName().Version?.ToString() ?? "unknown";
        }
        var commitSha = NormalizeBuildMetadata(_cfg["App:CommitSha"]);
        var buildStamp = NormalizeBuildMetadata(_cfg["App:BuildStamp"]);
@@ -525,10 +525,13 @@ app.MapControllers();
 // that queried MariaDB would restart a perfectly healthy backend whenever the database blipped, and
 // would also be a free unauthenticated way to probe database availability.
 // docs/production-readiness-review.md.
-app.MapGet("/health", () => Results.Ok(new
+// Version comes from configuration (App:Version), which is how docker-compose supplies it as
 // App__Version. Reading the APP_VERSION environment variable directly meant /health always
 // reported "unknown" in a container. Falls back to the assembly version for local development.
 app.MapGet("/health", (IConfiguration cfg) => Results.Ok(new
 {
    status = "ok",
-    version = Environment.GetEnvironmentVariable("APP_VERSION") ?? "unknown",
+    version = BuildMetadata.ResolveVersion(cfg),
 })).AllowAnonymous();
 // API schema for tooling/docs. Dev-only: not exposed in production deployments.
@@ -0,0 +1,39 @@
 using System.Reflection;
 namespace JobTrackerApi.Services;
 // Build metadata (version, commit, stamp) comes from configuration, populated by
 // docker-compose as App__Version / App__CommitSha / App__BuildStamp.
 //
 // This lives here because two callers need the same answer: the admin system page and
 // the anonymous /health endpoint. /health used to read the APP_VERSION *environment
 // variable* directly, which compose never sets under that name — it passes App__Version,
 // binding to the App:Version configuration key. So /health always reported "unknown".
 // See docs/release-candidate-review.md (N2).
 public static class BuildMetadata
 {
    // Ignore unresolved shell/compose placeholders that would otherwise leak into the
    // admin UI or a health response, e.g. $(git rev-parse --short HEAD) or ${APP_COMMIT_SHA}.
    public static string? Normalize(string? value)
    {
        var trimmed = (value ?? string.Empty).Trim();
        if (string.IsNullOrWhiteSpace(trimmed)) return null;
        if ((trimmed.StartsWith("$(") && trimmed.EndsWith(")")) || (trimmed.StartsWith("${") && trimmed.EndsWith("}")))
        {
            return null;
        }
        return trimmed;
    }
    // Configured version, else the assembly version, else "unknown". The assembly fallback
    // is what makes local development useful: nothing sets App:Version there.
    public static string ResolveVersion(IConfiguration cfg)
    {
        var version = Normalize(cfg["App:Version"]);
        if (!string.IsNullOrWhiteSpace(version)) return version;
        return Assembly.GetExecutingAssembly().GetName().Version?.ToString() ?? "unknown";
    }
 }
@@ -1937,7 +1937,14 @@ public static class StartupInitializationExtensions
                    // Hot-path composite indexes for tenant-scoped list/board/stats/analytics
                    // (OwnerUserId + IsDeleted) and reminders (OwnerUserId + FollowUpAt).
                    TryCreateIndex("JobApplications", "IX_JobApplications_OwnerUserId_IsDeleted", "`OwnerUserId`(191), `IsDeleted`");
-                    TryCreateIndex("JobApplications", "IX_JobApplications_OwnerUserId_FollowUpAt", "`OwnerUserId`(191), `FollowUpAt`");
+                    // FollowUpAt is `text` on MariaDB (the migration was scaffolded against SQLite, which
                    // stores DateTimeOffset as TEXT). A text column cannot be indexed without a prefix
                    // length, so without one this index ALWAYS failed the 3072-byte key check, was caught
                    // and skipped on every boot, and left the follow-up reminder query unindexed — while
                    // logging a "Specified key was too long" line the deploy runbook flags as a rollback
                    // signal. Prefix it like Status(50) below. ISO-8601 date strings sort lexicographically,
                    // so a 20-char prefix ("YYYY-MM-DD HH:MM:SS") keeps the index useful for the reminder scan.
                    TryCreateIndex("JobApplications", "IX_JobApplications_OwnerUserId_FollowUpAt", "`OwnerUserId`(191), `FollowUpAt`(20)");
                    // Status is longtext in MySQL (see JobTrackerContext.OnModelCreating), so it
                    // needs an explicit prefix length to be indexable under MariaDB's key-length rules.
                    TryCreateIndex("JobApplications", "IX_JobApplications_OwnerUserId_IsDeleted_Status", "`OwnerUserId`(191), `IsDeleted`, `Status`(50)");
@@ -103,6 +103,40 @@ public static class HumanLanguageCatalog
            map.TryAdd(normalizedAlias, normalizedCanonical);
        }
        // Seeded FIRST, and deliberately not derived from the host.
        //
        // This table used to come only from CultureInfo.GetCultures, which returns whatever
        // culture data the machine happens to carry: 806 entries on a normal Linux or Windows
        // box, exactly 1 under globalization-invariant mode, and an English-only subset on a
        // container with trimmed ICU data. So whether "Norwegian" was recognised as a human
        // language depended on the deployment environment, not on the CV. Under invariant mode
        // every language was silently dropped and a CV import lost its Languages section with
        // no error at all.
        //
        // These are the languages a CV realistically lists. Culture enumeration still runs
        // below and still adds breadth for free, but nothing here depends on it.
        //
        // Nothing in this list may collide with a technical skill — "Go", "Java", "Swift",
        // "Rust" and "Basic" are deliberately absent. "Basic" is also a proficiency level.
        string[] seed =
        [
            "English", "Norwegian", "Swedish", "Danish", "Finnish", "Icelandic",
            "German", "Dutch", "French", "Spanish", "Portuguese", "Italian",
            "Polish", "Czech", "Slovak", "Slovenian", "Croatian", "Serbian", "Bosnian",
            "Bulgarian", "Romanian", "Hungarian", "Greek", "Albanian", "Macedonian",
            "Russian", "Ukrainian", "Belarusian", "Lithuanian", "Latvian", "Estonian",
            "Turkish", "Arabic", "Hebrew", "Persian", "Kurdish", "Pashto", "Urdu",
            "Hindi", "Bengali", "Punjabi", "Gujarati", "Marathi", "Tamil", "Telugu",
            "Malayalam", "Kannada", "Sinhala", "Nepali",
            "Chinese", "Japanese", "Korean", "Vietnamese", "Thai", "Lao", "Khmer",
            "Burmese", "Malay", "Indonesian", "Filipino", "Tagalog", "Javanese",
            "Swahili", "Amharic", "Somali", "Hausa", "Yoruba", "Igbo", "Zulu", "Afrikaans",
            "Catalan", "Basque", "Galician", "Welsh", "Irish", "Scottish Gaelic", "Maltese",
            "Latin", "Esperanto", "Armenian", "Georgian", "Azerbaijani", "Kazakh", "Uzbek",
        ];
        foreach (var language in seed) Add(language, language);
        foreach (var culture in CultureInfo.GetCultures(CultureTypes.NeutralCultures | CultureTypes.SpecificCultures))
        {
            var english = CleanCultureLanguageName(culture.EnglishName);
@@ -109,22 +109,72 @@ hat links open the correct job/tab
 only the rollback. Restore the database only if the data itself is wrong or lost — it discards
 everything written since the dump.
 ## Environment loading
 `deploy/deploy.sh` **loads `/opt/job-tracker/shared/.env` into its own shell** before it decides
 anything. The symlink it creates in the checkout is for docker compose, which reads `.env` itself;
 the script needs the values too, to pick the right backup and to check its own configuration.
 - Parsed line by line, not `source`d — a compose `.env` is not a shell script, so an unquoted value
  containing spaces would execute as a command.
 - **Variables already set in the environment win**, so CI-provided `APP_VERSION`, `APP_COMMIT_SHA`
  and `APP_BUILD_STAMP` still override the file.
 - No value is ever echoed. Error messages name variables, never their contents.
 This was added on 2026-07-19. Before it, the script read an empty environment: `DATABASE_PROVIDER`
 fell back to `sqlite` on a MariaDB host, so the deploy tarred the data volume, printed
 `Backup verified`, and continued with no database dump at all. See
 `docs/release-candidate-review.md` (B1).
 ## Required production variables
 `validate_deploy_config` runs **before** the backup, and therefore before anything is built, stopped
 or replaced. A missing variable aborts the deploy while the running stack is still untouched.
 | Variable | Required | Why it is checked here |
 |---|---|---|
 | `DATABASE_PROVIDER` | **Always** — no default | Selects the backup. Guessing it wrong backs up the wrong database and reports success |
 | `JOBTRACKER_CONNECTION_STRING` | When provider is `mariadb`/`mysql` | Without it there is no way to dump the database |
 | `AI_SERVICE_TOKEN` | Always | `docker-compose.yml` declares it with `:?`; missing it kills the stack *after* the images are built |
 | `AUTH_JWT_KEY` | Always | Compose sets `Auth__Require=true`, and the backend throws at startup on a blank key — after the containers have been replaced |
 | `APP_PUBLIC_BASE_URL` | Optional | If unset the post-deploy public smoke check is skipped, and the script says so rather than skipping silently |
 `DATABASE_PROVIDER` deliberately has **no default**. An unset value used to mean "sqlite"; it now
 means "stop and tell me".
 ## Backup creation
 `deploy/deploy.sh` takes a backup **before** it builds, stops or replaces anything, and **aborts the
 deploy if the backup fails**. Nothing else in the deploy runs without a restore point.
 - **Location:** `/opt/job-tracker/backups` — override with `BACKUP_DIR`.
- **Naming:** `jobtracker-<database>-<UTC timestamp>.sql.gz`, e.g.
+- **Selection:** driven solely by `DATABASE_PROVIDER`, which must be set.
-  `jobtracker-jobtracker-20260719T153759Z.sql.gz`. The timestamp makes every file unique, so a deploy
+  - `mariadb` / `mysql` → SQL dump, `jobtracker-<database>-<UTC timestamp>.sql.gz`, e.g.
-  never overwrites an earlier backup.
+    `jobtracker-jobtracker-20260719T153759Z.sql.gz`
- **SQLite deployments** (`DATABASE_PROVIDER` unset or `sqlite`) get the data volume instead:
+  - `sqlite` → data volume archive, `jobtracker-sqlite-<UTC timestamp>.tar.gz`
-  `jobtracker-sqlite-<UTC timestamp>.tar.gz`.
+  - anything else → the deploy stops
 - **The filename tells you which path ran.** If you expect a MariaDB deploy and find a
  `jobtracker-sqlite-*.tar.gz`, the environment is wrong — that is the exact failure this check exists
  to make visible.
 - **Naming:** the UTC timestamp makes every file unique, so a deploy never overwrites an earlier backup.
 - **Credentials** come from `JOBTRACKER_CONNECTION_STRING` and are passed via `MYSQL_PWD`, never on the
  command line, so they cannot appear in the process list or the deploy log.
 - **Compression:** gzip. A small database compresses to a few KB.
- **Verification:** the script rejects a dump that is empty or missing `CREATE TABLE`, because a
+- **SQLite volume resolution:** compose prefixes volume names with the project name, so the script
-  truncated file that *looks* like a restore point is worse than none.
+  resolves `<project>_jobtracker_data` and **fails if that volume does not exist**. Naming the bare
  volume would silently create an empty one and back *that* up.
 ### Verification — each provider gets the check that proves its own format
 A backup that exists but is empty, truncated, or the wrong *kind* is worse than none, because it looks
 like a restore point.
 | Provider | Checks |
 |---|---|
 | MariaDB | non-empty; valid gzip; contains `CREATE TABLE`; contains the `Dump completed` trailer that `mariadb-dump` writes last, so a dump that died partway through is rejected |
 | SQLite | non-empty; valid gzip; the archive actually contains `jobtracker.db` |
 A failed check deletes the file rather than leaving something that looks like a backup.
 ### Taking one by hand
@@ -19,6 +19,48 @@ if [ ! -L "$ENV_TARGET" ] && [ ! -f "$ENV_TARGET" ]; then
  exit 1
 fi
 # ---------------------------------------------------------------------------
 # Load the shared environment into THIS shell.
 #
 # The symlink above is for docker compose, which reads .env itself. The script's
 # own decisions — which backup to take, which variables are missing — ran against
 # an empty environment until this existed, so DATABASE_PROVIDER defaulted to
 # sqlite and a MariaDB host silently got a volume tar instead of a dump, with a
 # "Backup verified" line to match. See docs/release-candidate-review.md (B1).
 #
 # Parsed line by line rather than sourced: a compose .env is not a shell script,
 # so an unquoted value containing spaces would execute as a command under `.`.
 # Values already present in the environment win, so CI-provided APP_VERSION and
 # friends still override the file. No value is ever echoed.
 # ---------------------------------------------------------------------------
 load_env_file() {
  local file="$1" line key value
  while IFS= read -r line || [ -n "$line" ]; do
    line="${line%$'\r'}"
    case "$line" in ''|'#'*) continue ;; esac
    case "$line" in *=*) ;; *) continue ;; esac
    key="${line%%=*}"
    value="${line#*=}"
    key="${key#export }"
    key="${key//[[:space:]]/}"
    case "$key" in ''|*[!A-Za-z0-9_]*) continue ;; esac
    case "$value" in
      \"*\") value="${value#\"}"; value="${value%\"}" ;;
      \'*\') value="${value#\'}"; value="${value%\'}" ;;
    esac
    # Already set (CI, or an explicit override on the command line) wins.
    [ -n "${!key+x}" ] && continue
    export "$key=$value"
  done < "$file"
 }
 load_env_file "$ENV_TARGET"
 echo "Loaded deployment environment from ${ENV_SOURCE}"
 export APP_VERSION="${APP_VERSION:-0.0.0}"
 export APP_COMMIT_SHA="${APP_COMMIT_SHA:-unknown}"
 export APP_BUILD_STAMP="${APP_BUILD_STAMP:-unknown}"
@@ -28,6 +70,74 @@ compose() {
  docker compose "$@"
 }
 # ---------------------------------------------------------------------------
 # Configuration validation, before anything is built, stopped or replaced.
 #
 # Everything checked here is fatal later anyway — compose declares AI_SERVICE_TOKEN
 # with `:?`, and the backend throws on a blank Auth__JwtKey. Failing here costs an
 # aborted deploy; failing there costs a half-replaced stack.
 #
 # Names only in the output. Never values.
 # ---------------------------------------------------------------------------
 require_var() {
  local name="$1" hint="$2"
  if [ -z "${!name:-}" ]; then
    echo "Missing required deployment variable: ${name}"
    echo "    ${hint}"
    echo "    Set it in ${ENV_SOURCE}."
    return 1
  fi
  return 0
 }
 validate_deploy_config() {
  local failed=0
  # Deliberately no default. Guessing this wrong means backing up the wrong
  # database and reporting success — the exact failure this block exists to stop.
  if [ -z "${DATABASE_PROVIDER:-}" ]; then
    echo "Missing required deployment variable: DATABASE_PROVIDER"
    echo "    Set to 'mariadb' (or 'mysql') for a MariaDB deployment, or 'sqlite'."
    echo "    There is no default: the wrong value backs up the wrong database."
    echo "    Set it in ${ENV_SOURCE}."
    failed=1
  else
    case "$(printf '%s' "$DATABASE_PROVIDER" | tr '[:upper:]' '[:lower:]')" in
      mysql|mariadb)
        require_var JOBTRACKER_CONNECTION_STRING \
          "MariaDB connection string. Required to take a database dump." || failed=1
        ;;
      sqlite)
        ;;
      *)
        echo "DATABASE_PROVIDER is not a recognised value. Use mariadb, mysql or sqlite."
        failed=1
        ;;
    esac
  fi
  require_var AI_SERVICE_TOKEN \
    "Shared secret between backend and ai-service. docker compose refuses to start without it." || failed=1
  # docker-compose.yml sets Auth__Require=true unconditionally, and the backend
  # throws at st�rtup rather than issuing tokens signed with a blank key.
  require_var AUTH_JWT_KEY \
    "JWT signing key. With Auth__Require=true the backend refuses to start without it." || failed=1
  if [ -z "${APP_PUBLIC_BASE_URL:-}" ]; then
    echo "Note: APP_PUBLIC_BASE_URL is not set — the post-deploy public smoke check will be skipped."
  fi
  if [ "$failed" -ne 0 ]; then
    echo "Deployment configuration is incomplete. Nothing was built, stopped or replaced."
    return 1
  fi
  echo "Deployment configuration validated (database provider: ${DATABASE_PROVIDER})."
  return 0
 }
 # ---------------------------------------------------------------------------
 # Database backup, taken BEFORE anything is stopped, built or replaced.
 #
@@ -44,27 +154,43 @@ backup_database() {
    return 0
  fi
-  local provider="${DATABASE_PROVIDER:-sqlite}"
+  # validate_deploy_config has already established this is set and recognised.
  local provider
  provider="$(printf '%s' "$DATABASE_PROVIDER" | tr '[:upper:]' '[:lower:]')"
  local stamp
  stamp="$(date -u +%Y%m%dT%H%M%SZ)"
  mkdir -p "$BACKUP_DIR"
-  if [ "$provider" != "mysql" ] && [ "$provider" != "mariadb" ]; then
+  if [ "$provider" = "sqlite" ]; then
    # SQLite lives in the jobtracker_data volume. Tar it from a throwaway container
    # so the host needs no sqlite tooling and no knowledge of the volume layout.
    #
    # docker compose prefixes volume names with the project name, which defaults to
    # the directory name. Naming the bare volume here would silently CREATE an empty
    # one and back that up instead, so resolve it and fail if it is not there.
    local volume="${COMPOSE_PROJECT_NAME:-$(basename "$PWD")}_jobtracker_data"
    if ! docker volume inspect "$volume" >/dev/null 2>&1; then
      echo "SQLite data volume '${volume}' does not exist. Aborting deploy."
      echo "    Volumes are prefixed with the compose project name; set COMPOSE_PROJECT_NAME if it differs."
      echo "    Available: $(docker volume ls -q | grep jobtracker_data | tr '\n' ' ')"
      return 1
    fi
    local target="$BACKUP_DIR/jobtracker-sqlite-${stamp}.tar.gz"
-    echo "Backing up SQLite data volume to ${target}"
+    echo "Backing up SQLite data volume '${volume}' to ${target}"
    # The archive path is built inside the container: passing /backup/... as an argument
    # gets rewritten by MSYS path translation when the script is run from Git Bash.
    if ! docker run --rm \
-        -v jobtracker_data:/data:ro \
+        -v "$volume":/data:ro \
        -v "$BACKUP_DIR":/backup \
        -e ARCHIVE_NAME="$(basename "$target")" \
        alpine:3 sh -c 'tar czf "/backup/$ARCHIVE_NAME" -C /data .'; then
      echo "SQLite volume backup FAILED. Aborting deploy."
      rm -f "$target"
      return 1
    fi
-    verify_backup "$target"
+    verify_volume_backup "$target"
    return $?
  fi
@@ -120,14 +246,20 @@ backup_database() {
    return 1
  fi
-  verify_backup "$target" "CREATE TABLE"
+  verify_sql_backup "$target"
 }
-# A dump that exists but is empty or truncated is worse than none, because it
+# A backup that exists but is empty, truncated, or is the wrong KIND of backup is
-# looks like a restore point. Check size, and content when we know what to expect.
+# worse than none, because it looks like a restore point. Each provider gets the
-verify_backup() {
+# check that proves its own format — a tar cannot pass the dump check and a dump
-  local target="$1"
+# cannot pass the archive check.
-  local expect="${2:-}"
+#
 # Note on `set +o pipefail` below: `grep -q` exits at the first match, which
 # SIGPIPEs the decompressor upstream. Under pipefail that fails the pipeline and
 # would reject a perfectly good backup.
 verify_backup_file_shape() {
  local target="$1" kind="$2"
  if [ ! -s "$target" ]; then
    echo "Backup file ${target} is missing or empty. Aborting deploy."
@@ -135,20 +267,83 @@ verify_backup() {
    return 1
  fi
-  local size
+  if ! gzip -t "$target" 2>/dev/null; then
-  size="$(du -h "$target" | cut -f1)"
+    echo "Backup ${target} is not a valid gzip archive (truncated ${kind}?). Aborting deploy."
  if [ -n "$expect" ] && ! gzip -dc "$target" | grep -q "$expect"; then
    echo "Backup ${target} does not contain '${expect}' — it is not a usable dump. Aborting deploy."
    rm -f "$target"
    return 1
  fi
  return 0
 }
 report_backup() {
  local target="$1" size
  size="$(du -h "$target" | cut -f1)"
  echo "Backup verified: ${target} (${size})"
  echo "Retention is manual — old backups in ${BACKUP_DIR} are never deleted automatically."
  return 0
 }
 verify_sql_backup() {
  local target="$1"
  verify_backup_file_shape "$target" "dump" || return 1
  local has_schema=1 has_trailer=1
  set +o pipefail
  if gzip -dc "$target" | grep -q "CREATE TABLE"; then has_schema=0; fi
  # mariadb-dump / mysqldump write "-- Dump completed on ..." as the last line.
  # Its absence means the dump died partway through.
  if gzip -dc "$target" | grep -q "Dump completed"; then has_trailer=0; fi
  set -o pipefail
  if [ "$has_schema" -ne 0 ]; then
    echo "Backup ${target} contains no CREATE TABLE statement — it is not a usable dump. Aborting deploy."
    rm -f "$target"
    return 1
  fi
  if [ "$has_trailer" -ne 0 ]; then
    echo "Backup ${target} has no dump trailer — it is truncated. Aborting deploy."
    rm -f "$target"
    return 1
  fi
  report_backup "$target"
 }
 verify_volume_backup() {
  local target="$1"
  verify_backup_file_shape "$target" "archive" || return 1
  # Prove this is the DATABASE volume, not merely a non-empty tar. This is what
  # catches an archive taken from the wrong volume, or from a freshly created
  # empty one.
  # Listed in the same container that wrote it, not with the host's tar: GNU tar
  # reads a leading "C:/" as an rsh host and fails outright when this script runs
  # from Git Bash. Docker is already required to have produced the archive.
  local has_db=1
  if docker run --rm \
      -v "$(dirname "$target")":/backup:ro \
      -e ARCHIVE_NAME="$(basename "$target")" \
      alpine:3 sh -c 'tar tzf "/backup/$ARCHIVE_NAME" | grep -q "jobtracker\.db"' >/dev/null 2>&1; then
    has_db=0
  fi
  if [ "$has_db" -ne 0 ]; then
    echo "Archive ${target} contains no jobtracker.db — it is not a SQLite data backup. Aborting deploy."
    rm -f "$target"
    return 1
  fi
  report_backup "$target"
 }
 if ! validate_deploy_config; then
  exit 1
 fi
 if ! backup_database; then
  exit 1
 fi
@@ -11,16 +11,25 @@
 Verified by reading `deploy/deploy.sh` and `JobTrackerApi/Program.cs`:
-1. `deploy.sh` links `/opt/job-tracker/shared/.env` into the checkout as `.env`.
+1. `deploy.sh` links `/opt/job-tracker/shared/.env` into the checkout as `.env`, **and loads it into
-2. **It takes a database backup and aborts if that fails.** Nothing else runs without a restore point.
+   its own shell.** The link is for docker compose; the script needs the values itself to pick the
-3. `docker compose pull`, then builds `backend` and `frontend` (with one prune-and-retry on failure).
+   right backup. Values already in the environment (CI's `APP_VERSION` and friends) win.
-4. `docker compose up -d --force-recreate --remove-orphans backend frontend`.
+2. **It validates the deployment configuration**, before anything is built, stopped or replaced:
   `DATABASE_PROVIDER` (required, no default), the connection string when that provider needs one,
   `AI_SERVICE_TOKEN` and `AUTH_JWT_KEY`. A missing variable aborts the deploy with the running stack
   untouched. Names are printed, never values.
 3. **It takes a database backup and aborts if that fails.** Nothing else runs without a restore point.
   The provider chooses the backup: `mariadb`/`mysql` gives a `.sql.gz` dump, `sqlite` gives a
   `.tar.gz` of the data volume. Each is verified against its own format — the dump must contain
   `CREATE TABLE` and the `Dump completed` trailer; the archive must contain `jobtracker.db`.
 4. `docker compose pull`, then builds `backend` and `frontend` (with one prune-and-retry on failure).
 5. `docker compose up -d --force-recreate --remove-orphans backend frontend`.
   **There is no `compose down`** — containers are replaced in place, so the window is short.
-5. On backend start, `InitializeJobTrackerAsync` runs: **reconcile → `Database.Migrate()` → reconcile**.
+6. On backend start, `InitializeJobTrackerAsync` runs: **reconcile → `Database.Migrate()` → reconcile**.
   Every Phase 4/5 migration is a no-op; the reconciler creates those tables with correct per-provider
   DDL. `Migrate()` throws on failure, so a schema problem exits the container rather than limping on.
-6. `deploy.sh` waits, then fails the deploy if `backend` is not running, and runs a public smoke check
+7. `deploy.sh` waits, then fails the deploy if `backend` is not running, and runs a public smoke check
-   against `APP_PUBLIC_BASE_URL` if it is set.
+   against `APP_PUBLIC_BASE_URL`.
 ---
@@ -31,12 +40,17 @@ Verified by reading `deploy/deploy.sh` and `JobTrackerApi/Program.cs`:
      restored is a hypothesis.
 - [ ] **Disk space checked.** `df -h` on the host. You need room for the backup, two image sets during
      the build, and the build cache. `docker system df` shows what Docker is holding.
- [ ] **Environment variables present.** Check `/opt/job-tracker/shared/.env` contains:
+- [ ] **Environment variables present.** `deploy.sh` now checks these itself and aborts before it
-      - `AI_SERVICE_TOKEN` — **compose refuses to start without it**
+      builds or replaces anything, so a miss costs an aborted deploy rather than a broken one. Check
      first anyway and skip the round trip — `/opt/job-tracker/shared/.env` must contain:
      - `DATABASE_PROVIDER=mariadb` (or `mysql`) — **required, no default.** An absent value used to
        mean "sqlite", which silently produced the wrong backup; it now stops the deploy
      - `JOBTRACKER_CONNECTION_STRING` — see the host-resolution note below
      - `AI_SERVICE_TOKEN` — compose refuses to start without it
      - `AUTH_JWT_KEY` — with `Auth__Require=true`, a blank key **throws at startup** (this is good;
        it fails loud rather than silently invalidating every session on restart)
-      - `DATABASE_PROVIDER=mysql` — **defaults to `sqlite` if absent**
+      - `APP_PUBLIC_BASE_URL` — optional; without it the post-deploy public smoke check is skipped,
-      - `JOBTRACKER_CONNECTION_STRING` — see the host-resolution note below
+        and the script prints that it is skipping
      - `AUTH_ADMIN_EMAIL` / `AUTH_ADMIN_PASSWORD` only if you want admin seeding on this boot
 - [ ] **Connection string host resolves from inside the container.** `Server=127.0.0.1` means *the
      backend container*, not the host — this bit me during validation. Use the host's LAN address, a
@@ -65,8 +79,12 @@ Automatic — `deploy.sh` runs it first and aborts on failure. Confirm afterward
 ls -lt /opt/job-tracker/backups | head -3
 ```
-Expect a new `jobtracker-<db>-<UTC timestamp>.sql.gz`. The script already rejected it if it were empty
+Expect a new `jobtracker-<db>-<UTC timestamp>.sql.gz`. The script already rejected it if it were empty,
-or missing `CREATE TABLE`.
+not valid gzip, missing `CREATE TABLE`, or missing the `Dump completed` trailer.
 **Check the filename, not just that a file appeared.** A `jobtracker-sqlite-<stamp>.tar.gz` on a
 MariaDB deploy means the environment is wrong — that is the exact failure the provider check exists to
 prevent, and it should now abort rather than reach this point.
 ### 2. Pull code
@@ -105,7 +123,7 @@ docker compose logs -f backend
 | `Unhandled exception ... Table '...' doesn't exist` | Reconciler ordering problem |
 | `no such table: INFORMATION_SCHEMA.TABLES` | `DATABASE_PROVIDER=mysql` but the connection string is **empty** — verified failure mode |
 | `Unable to connect to any of the specified MySQL hosts` | Connection string host unreachable from inside the container |
-| `Auth is required but Auth:JwtKey is not configured` | `AUTH_JWT_KEY` missing from `.env` |
+| `Auth is required but Auth:JwtKey is not configured` | `AUTH_JWT_KEY` missing from `.env` — `deploy.sh` should now catch this before the build |
 ### 8. Health verification
@@ -126,7 +144,12 @@ upstream cannot be resolved.
 ```bash
 # Health endpoint — anonymous, does not touch the database
 curl -fsS https://<host>/health
-# expect: {"status":"ok","version":"..."}
+# expect: {"status":"ok","version":"<the APP_VERSION you deployed>"}
 #
 # The version comes from App:Version, which compose passes as App__Version from APP_VERSION.
 # CI sets it to the workflow run number. A version of "1.0.0.0" (the assembly fallback) means
 # APP_VERSION did not reach the container — harmless in itself, but it tells you the build
 # metadata is not flowing, so the admin system page will be vague about what is deployed.
 # Auth still enforced (this is the check that proves the API is not open)
 curl -s -o /dev/null -w '%{http_code}\n' https://<host>/api/jobapplications
@@ -236,6 +259,11 @@ Against MariaDB 11 containers, no production data:
 | Backup against a seeded MariaDB | ✅ verified dump written |
 | Restore into a clean MariaDB | ✅ rows identical |
 | Backup failure paths | ✅ bad credentials and missing connection string both abort, no partial file |
 | Shared `.env` loaded into the deploy shell | ✅ MariaDB path selected from the file alone; dump contains schema, rows and trailer |
 | `DATABASE_PROVIDER` missing or unrecognised | ✅ deploy stops before build/replace, names the variable, leaves no backup file |
 | `AI_SERVICE_TOKEN` / `AUTH_JWT_KEY` missing | ✅ both reported in one pass, deploy stops |
 | SQLite volume backup, and its failure paths | ✅ valid archive verified; a volume with no `jobtracker.db`, and a volume name that does not exist, both abort |
 | Secret leakage in deploy output | ✅ zero occurrences of any password, token or key across every test |
 | Empty connection string with `provider=mysql` | ✅ fails loudly (`no such table: INFORMATION_SCHEMA.TABLES`) rather than silently serving an empty database |
 | Backend with unreachable database | ✅ exits, reported `unhealthy` |
@@ -150,9 +150,10 @@ All four scenarios, 2026-07-19, against MariaDB 11 and SQLite:
 | Scenario | Result |
 |---|---|
-| Empty MariaDB | 40 tables created, app starts |
+| Empty MariaDB | 42 tables created, app starts |
-| Populated MariaDB, restart | idempotent — still 40 tables, rows preserved |
+| Populated MariaDB, restart | idempotent — still 42 tables, rows preserved |
-| Empty MariaDB via the Docker image | 40 tables created, app starts |
+| Partially-migrated MariaDB (Phase 4/5 tables dropped) | healed 35 → 42, surviving rows preserved |
 | Empty MariaDB via the Docker image | 42 tables created, app starts |
 | Fresh SQLite | 42 tables created, app starts |
 | Existing partially-migrated SQLite dev DB (34 tables) | upgraded to 44 tables, 13 applications and 8 companies preserved |
@@ -0,0 +1,141 @@
 # Backup and restore verification
 > **2026-07-19 19:32 UTC.** Full backup → verify → restore → start-the-app rehearsal of the
 > `deploy/deploy.sh` backup path, exercised end to end against MariaDB 11.
 >
 > **This is a rehearsal, not a verification of production data.** See *Limitations* — that section is
 > the most important part of this document, and the checklist at the end is what actually closes the
 > gap.
 ## Scope — read this first
 | | |
 |---|---|
 | **What was verified** | The backup *mechanism*: the real `backup_database` function from `deploy.sh`, against a real MariaDB 11 database carrying the real 42-table schema, restored into a separate clean MariaDB 11 container, with the real application started against the result |
 | **What was NOT verified** | **Your production database.** No production host was contacted, no production credentials were used, and no production data was read, copied or restored |
 | **Why** | The machine this ran on has no route to production: no `/opt/job-tracker`, no `DATABASE_PROVIDER` or `JOBTRACKER_CONNECTION_STRING` in its `.env`, and the local stack runs SQLite. The production host, user and key are CI secrets (`PROD_HOST`, `PROD_USER`, `PROD_SSH_KEY`) that are not available here |
 The commands below are the ones to run against production. They are recorded so the owner can execute
 the same sequence with production values substituted.
 ## 1. Backup configuration
 Read from `deploy/deploy.sh` and `docker-compose.yml`:
 - `deploy.sh` loads `/opt/job-tracker/shared/.env` into its own shell before any decision.
 - `validate_deploy_config` runs **before** the backup, and therefore before any build, stop or replace.
  It requires `DATABASE_PROVIDER` (no default), `JOBTRACKER_CONNECTION_STRING` when the provider is
  MariaDB, plus `AI_SERVICE_TOKEN` and `AUTH_JWT_KEY`.
 - Backups land in `/opt/job-tracker/backups` (override with `BACKUP_DIR`), UTC-timestamped, gzipped.
  Nothing is ever overwritten and nothing is ever auto-deleted.
 - The password travels via `MYSQL_PWD`, never on the command line.
 ## 2. Provider resolution
 `DATABASE_PROVIDER=mariadb` selected the MariaDB dump path. Confirmed by the script's own output:
 ```
 Deployment configuration validated (database provider: mariadb).
 Backing up MariaDB database 'jobtracker' on <host>:<port> to <dir>/jobtracker-jobtracker-20260719T193022Z.sql.gz
 Backup verified: <dir>/jobtracker-jobtracker-20260719T193022Z.sql.gz (8.0K)
 ```
 **The filename is the check that matters.** `jobtracker-<database>-<stamp>.sql.gz` means the MariaDB
 path ran. A `jobtracker-sqlite-<stamp>.tar.gz` on a MariaDB host would mean the environment is wrong.
 ## 3. Commands used
 Schema built by the real application, not by hand — `Database.Migrate()` plus the startup reconciler
 against an empty MariaDB, producing 42 tables. Representative rows were then seeded across users,
 companies, applications, career profile and CV variants.
 ```bash
 # Backup — the real function from deploy.sh, not a hand-written dump
 DATABASE_PROVIDER=mariadb \
 JOBTRACKER_CONNECTION_STRING='Server=<host>;Port=<port>;Database=jobtracker;User Id=<user>;Password=<password>;' \
 BACKUP_DIR=/opt/job-tracker/backups \
  deploy/deploy.sh          # takes the backup first and aborts the deploy if it fails
 # Integrity and content
 gzip -t  "$BACKUP"                                  # valid archive
 gzip -dc "$BACKUP" | grep -c 'CREATE TABLE'         # schema present
 gzip -dc "$BACKUP" | tail -1                        # "-- Dump completed on ..." trailer
 # Restore into a SEPARATE, empty database — never over a live one
 gzip -dc "$BACKUP" | MYSQL_PWD='<password>' mariadb --host=<host> --user=<user> jobtracker
 ```
 Passwords are supplied via `MYSQL_PWD` so they never reach the process list or the shell history.
 ## 4. Backup verification result
 | Check | Result |
 |---|---|
 | Correct backup type | ✅ `.sql.gz` MariaDB dump, not a volume archive |
 | File integrity | ✅ `gzip -t` passed |
 | Schema markers | ✅ 42 `CREATE TABLE` statements |
 | Dump trailer | ✅ `-- Dump completed on 2026-07-19 19:30:23` — proves the dump was not truncated |
 | Expected tables | ✅ `AspNetUsers`, `Companies`, `JobApplications`, `CareerProfiles`, `CvVariants`, `JobEvents`, `ApplicationChecklistItems`, `AiInteractions` all present |
 | Data present | ✅ 7 `INSERT INTO` statements |
 ## 5. Restore result
 Restored into a **separate, empty** MariaDB 11 container (0 tables before, 42 after).
 | Entity | Source | Restored | |
 |---|---|---|---|
 | Users | 1 | 1 | ✅ |
 | Companies | 2 | 2 | ✅ |
 | Applications | 2 | 2 | ✅ |
 | Career profiles | 1 | 1 | ✅ |
 | CV variants | 1 | 1 | ✅ |
 Beyond counts:
 - **All 42 tables compared** — table lists identical, and **every table's row count matched**.
 - **Content survived, not just cardinality** — the restored applications resolve their company
  foreign keys (`Senior Backend Engineer @ Northwind Consulting`), the CV variant kept its name and
  `IsPublic` flag, and the career profile JSON still contained its languages.
 - **The application starts against the restored database.** `/health` returned
  `{"status":"ok","version":"restore-rehearsal"}`, the schema stayed at 42 tables (the reconciler
  correctly found nothing to do) and rows were preserved. A restore that produces a database the app
  cannot boot against is not a restore.
 ## Limitations
 Read these before treating the deployment as backed up.
 1. **No production data was touched.** Everything above ran against a locally built MariaDB with
   seeded rows. It proves the mechanism; it proves nothing about your database.
 2. **The seeded dataset is tiny** (7 rows). It does not exercise dump duration, disk headroom, lock
   behaviour under load, or timeout limits on a real dataset. A production database large enough to
   make `mariadb-dump` slow could behave differently.
 3. **Character-set and collation fidelity was not stress-tested.** The seeded data was mostly ASCII.
   Real CV content contains non-ASCII text — Norwegian `æøå`, accents, CJK. The dump defaults should
   handle this, but it was not proven here.
 4. **No restore was performed over a populated database.** The destination was empty. Restoring over
   an existing database is a different operation with different failure modes.
 5. **`deploy.sh` was exercised up to and including the backup**, not through the build and container
   replacement, which would have required a full deployment.
 6. **The `mariadb-dump` client came from a container** (`mariadb:11`). If the production host has its
   own client installed, `deploy.sh` uses that instead, and version differences are possible.
 ## What the owner still needs to do
 This is the checklist that turns a rehearsal into a verified backup.
 - [ ] **Run one backup by hand against production** and confirm the filename is
      `jobtracker-<database>-<stamp>.sql.gz`.
 - [ ] **Check the dump size is plausible** for the amount of data you have. A suspiciously small file
      is the signal worth catching.
 - [ ] **Restore it into a scratch database** — never over the live one — and confirm row counts for
      `AspNetUsers`, `JobApplications`, `Companies` and `CareerProfiles` match production.
 - [ ] **Check non-ASCII text survived.** Open one CV or career profile containing `æ`, `ø` or `å` in
      the restored copy and confirm it is not mangled. This is the most likely silent failure.
 - [ ] **Confirm `/opt/job-tracker/backups` has disk headroom** for several dumps.
 - [ ] **Note how long the dump takes.** It runs before every deploy and blocks it.
 ## Cleanup
 Both MariaDB containers and all rehearsal files were removed. No test artefacts remain, and the
 existing local `jobtracker-backend-1` / `jobtracker-frontend-1` stack was left untouched.
@@ -0,0 +1,433 @@
 # Release candidate review
 > 2026-07-19. Final validation pass before the first production deployment. **No application code was
 > changed by this review** — findings only. Every claim below was checked against the implementation,
 > not against the other documents. Where a document and the code disagreed, the code is reported.
 >
 > Companions: `deploy/first-production-deployment.md` (how to deploy),
 > `docs/production-readiness-review.md` (what was audited),
 > `docs/infrastructure/database-ownership.md` (who creates which table),
 > `docs/release-checklist.md` (state of the build).
 **Updated 2026-07-19 (final pass).** Three findings are now fixed and verified:
 - **B1** — the pre-deploy backup silently backed up the wrong thing (`66b02bc`)
 - **N2** — `/health` always reported `version: unknown` (`8f6f2ba`)
 - **CV languages** — human languages were dropped depending on the host's ICU data (`9681618`)
 The first two are recorded under *Closed since the first pass* with their original text, because the
 failure modes are worth understanding. Also since the previous pass: the deployment sequence was
 re-verified by line number rather than by prose, and a full backup → restore → start-the-app rehearsal
 was completed and recorded in
 [`docs/operations/production-backup-verification.md`](operations/production-backup-verification.md).
 **Release-candidate audit, 2026-07-19.** A full verification pass — 420 backend tests (Windows + Linux,
 both ICU modes), frontend tests/typecheck/build, all three Docker images, and all four database startup
 scenarios run against live MariaDB 11 and SQLite — found and fixed two more issues: a follow-up reminder
 index that never created on MariaDB (and logged a false rollback signal on every boot), and a
 nondeterministic timeline test. Both are recorded below. Backup → restore → app-start was re-run
 end to end. No open blocker remains in the code or deployment path.
 **Verdict: one blocker remains, and it is external — CI (B2).** Nothing in the application or the
 deployment path is now known to be blocking. Everything verified here is *local* verification; CI has
 proven none of it.
 | Section | Meaning |
 |---|---|
 | [READY](#ready--completed-technical-checks) | Verified. Nothing further needed before deploying |
 | [BLOCKED](#blocked--requires-external-action) | Requires action outside this repository. Deployment is gated on it |
 | [MANUAL VERIFICATION](#manual-verification--only-the-owner-can-do-these) | Only the owner can confirm — sign-in, real data, production smoke tests |
 | [Accepted](#accepted-for-the-first-release-non-blocking) | Known limitations, deliberately shipped as-is |
 ---
 ## READY — completed technical checks
 ### Database
 | Item | Verified against |
 |---|---|
 | Startup order is connect → reconcile → `Migrate()` → reconcile → seed | `StartupInitializationExtensions.InitializeJobTrackerAsync`; matches `database-ownership.md` exactly |
 | Migration ownership | `Companies`, `JobApplications`, `Jobs`, `Correspondences`, `Attachments`, `JobEvents`, `RuleSettings`, Identity — created only by migrations; reconciler repairs but never `CREATE TABLE`s them |
 | Reconciler ownership | Every Phase 4/5 table is reconciler-owned with a paired no-op migration holding the snapshot. The seven no-op migrations named in `database-ownership.md` all exist on disk |
 | Dependency guards | Reconciler tables with an FK are guarded on the parent existing, so pass 1 skips and pass 2 creates. `EnsureMySqlIndex` guards on table existence, not just index existence |
 | Reconciler is non-destructive | `DropMalformedMySqlTable` checks row count first and skips any table holding rows |
 | Provider selection accepts both spellings | `Program.cs:68` — `provider is "mysql" or "mariadb"`. `deploy/README.md` documents `DATABASE_PROVIDER=mariadb`; that value works |
 | Rollback reasoning is sound | Phase 4/5 migrations are no-ops, so reverting code never leaves migration history ahead of the schema. Older code ignores the extra tables |
 ### Security
 | Item | Verified against |
 |---|---|
 | Explicit authorization | Every user-owned controller carries a class-level `[Authorize(AuthenticationSchemes = "local")]`. Confirmed directly on `JobApplicationsController`, `CompaniesController`, `CorrespondenceController`, `RulesController`, `AttachmentsController`, `BackupController`, `ExportController` |
 | Independent of `Auth:Require` | The `FallbackPolicy` at `Program.cs:380` is *additional*, not the only defence. `/api/jobapplications` returns 401 with `Auth:Require` unset |
 | Anonymous surface is a deliberate allow-list | `PublicCvController` (class-level `[AllowAnonymous]`), `ClientErrorsController`, `/health`, and per-method anonymity on `AuthController` / `TwoFactorController` / the two OAuth callbacks |
 | OAuth callbacks are not an open door | `GmailController.Callback` and `MicrosoftGraphController.Callback` are anonymous by necessity but gated on `ConsumeState(state)`; an unknown or replayed state is rejected before any token exchange |
 | Tenant isolation | 25 global query filters, all deny-on-null (`CurrentUserId != null && OwnerUserId == CurrentUserId`), covering every owner-scoped root entity |
 | Attachments are tenant-scoped and path-safe | Resolved through the parent `JobApplication` query (so the job-level filter applies); stored names go through `Path.GetFileName` + `BuildStoredFileName`, so a crafted upload name cannot escape the attachments root |
 | AI service is not reachable from outside | `ai-service` is on `ai_internal` only — not on `default`, not on the external `shared_services`, no published port. Backend is the only member that can route to it |
 | AI service authenticates its caller | `X-Ai-Service-Token` middleware with `hmac.compare_digest`; only `/health` is open. Compose declares the token with `:?` so a deploy that forgets it fails loudly |
 | Public CV is opt-in | Off by default, per-variant, unguessable slug, `noindex` |
 | Secrets not committed | `.env` is gitignored (`.gitignore:8`); nothing sensitive tracked |
 | Backup handles the password safely | `MYSQL_PWD`, never on the command line, so it cannot reach the process list or the deploy log. Dump stderr is scrubbed before it is echoed |
 ### Application integrity — architecture rules
 | Rule | Verified |
 |---|---|
 | **CareerProfile is the only editable career source** | The two Phase 5 services that touch `CareerProfiles` — `ApplicationChecklistService:343` and `ApplicationIntelligenceService:199` — both read `AsNoTracking()`. Nothing downstream writes to it |
 | **CvVariant is a derived lens** | `ApplicationAssetsService.AttachVariantAsync` writes only `JobApplicationId` and `UpdatedAtUtc`. Attaching a CV to an application never touches variant content |
 | **Application Workspace aggregates only** | `ApplicationIntelligenceService` and `ApplicationTimelineService` contain **zero** `SaveChanges` calls. They are pure projections |
 | **JobEvent is the timeline source of truth** | `ApplicationTimelineService` interprets `JobEvent` rows and stores nothing. Emission is centralised in `JobLifecycleEvents` |
 | **AI is suggestion-only** | Generation appends to `AiInteraction`; nothing is written to a profile, variant, cover letter or prep item without an explicit user save |
 ### Deployment sequence — verified against the code, not the prose
 Checked by line number in `deploy/deploy.sh` and `StartupInitializationExtensions.cs` on 2026-07-19.
 **Before anything is replaced** — the order is what matters, and it holds:
 | Order | Step | Where |
 |---|---|---|
 | 1 | Load `/opt/job-tracker/shared/.env` into the deploy shell | `deploy.sh` (before any decision) |
 | 2 | **Validate configuration** — provider, connection string, `AI_SERVICE_TOKEN`, `AUTH_JWT_KEY` | `deploy.sh:343` |
 | 3 | **Take and verify the database backup**; abort the deploy if it fails | `deploy.sh:347` |
 | 4 | Build images | `deploy.sh:375` |
 | 5 | Replace containers (`up -d --force-recreate`) | `deploy.sh:382` |
 Nothing is built, stopped or replaced before validation and backup. Confirmed: 343 and 347 both
 precede 375 and 382.
 **During deployment** — backend startup, `InitializeJobTrackerAsync`:
 | Order | Step | Where |
 |---|---|---|
 | 1 | `ReconcileSchema()` — pass 1, repair and reconciler-owned tables | `StartupInitializationExtensions.cs:1965` |
 | 2 | `Database.Migrate()` — migration-owned tables | `:1973` |
 | 3 | `ReconcileSchema()` — pass 2, everything pass 1 had to skip | `:1984` |
 **Migrations expected to run: none that create anything.** All seven Phase 4/5 migrations were
 confirmed to have a **literally empty `Up` body** (0 statements each): `AddCareerProfileRelationalChildren`,
 `AddCvVariants`, `AddAiInteractions`, `AddApplicationChecklistItems`, `SyncCareerChildKeyLengths`,
 `AddCoverLetterVersions`, `AddInterviewPrepItems`. The reconciler owns their DDL with correct
 per-provider types. This is what makes a code rollback safe — migration history never runs ahead of
 the schema.
 **Health checks and failure behaviour:**
 | Item | Verified |
 |---|---|
 | Backend health check | `curl` against `/health`, 90s start period for first-boot reconciliation |
 | Frontend health check | `wget` against nginx |
 | Dependency gate | `frontend` declares `depends_on: backend: condition: service_healthy` — a backend that never reports healthy makes `compose up` fail and `set -e` aborts the deploy |
 | Rollback procedure documented | `deploy/first-production-deployment.md` and `deploy/README.md`, both distinguishing code rollback from database restore |
 ### Deployment
 | Item | Verified |
 |---|---|
 | CI gates deployment | `deploy` job declares `needs: test` and `if: push && ref == refs/heads/main` |
 | Deploy is pinned to the tested commit | The remote script `git reset --hard ${{ github.sha }}` and aborts if that commit is not fetchable |
 | Container dependency ordering | `frontend` declares `depends_on: backend: condition: service_healthy`. A backend that never reports healthy makes `compose up` fail, and `set -e` aborts the deploy — a broken backend cannot present as a running stack |
 | Health checks exist on both services | `curl` against `/health` for backend (90s start period, covering first-boot reconciliation), `wget` against nginx for frontend |
 | `/health` does not touch the database | Deliberate: a DB-querying probe would restart a healthy backend on any database blip, and would hand out an unauthenticated way to probe database availability |
 | Missing `AI_SERVICE_TOKEN` fails the stack | `${AI_SERVICE_TOKEN:?…}` in compose, on both `backend` and `ai-service` |
 | Missing `AUTH_JWT_KEY` fails the backend | With `Auth__Require=true`, `Program.cs:241` throws `InvalidOperationException`. It does not silently generate an ephemeral key |
 | Backup rejects a useless dump | `verify_backup` fails the deploy on an empty file, and on a MariaDB dump lacking `CREATE TABLE` |
 | Backups never overwrite | UTC-timestamped filenames; retention is explicitly manual and the script says so |
 | **Deploy script loads its own environment** | `deploy.sh` parses the shared `.env` into its own shell before any decision. Values already in the environment (CI's `APP_VERSION`) still win. No value is echoed |
 | **`DATABASE_PROVIDER` is required, not defaulted** | Missing or unrecognised aborts the deploy. Verified: exits 1, names the variable, leaves no backup file |
 | **Configuration validated before the stack is touched** | Connection string, `AI_SERVICE_TOKEN` and `AUTH_JWT_KEY` are checked before the backup, and therefore before any build, stop or replace |
 | **Backups verified per provider** | A dump needs valid gzip, `CREATE TABLE` and the `Dump completed` trailer; an archive needs `jobtracker.db`. Truncated and trailer-stripped dumps both rejected |
 | **SQLite volume resolved by real name** | Project-prefixed, and fails if absent — the bare name would have created an empty volume and backed that up |
 | **No secret leakage in deploy output** | Zero occurrences of any password, token or key across every failure-path test |
 | **`/health` reports the deployed version** | Reads `App:Version`; verified end to end that `App__Version=9.9.9-test` surfaces as `9.9.9-test`, and that an unset value falls back to the assembly version |
 | Test suite | 420 backend tests pass in Release, and also pass under `DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1` and on Linux — see the ICU finding below |
 ---
 ## Closed since the first pass
 Both were found by this review, fixed in their own commits, and re-verified. Original findings kept.
 ### B1. ~~The pre-deploy database backup does not back up the database~~ — **CLOSED 2026-07-19**
 *Original finding, kept because the failure mode is worth understanding:*
 **Severity: critical. This invalidates the restore point that the entire deployment plan depends on.**
 `deploy/deploy.sh` symlinks `/opt/job-tracker/shared/.env` into the checkout so that **docker compose**
 can read it. It never *sources* it. There is no `set -a`, no `. .env`, no `export` of the database
 variables anywhere in the script.
 So the script's own shell evaluates:
 ```sh
 local provider="${DATABASE_PROVIDER:-sqlite}"
 ```
 `DATABASE_PROVIDER` is not set in that shell. The CI deploy step exports only `APP_VERSION`,
 `APP_COMMIT_SHA` and `APP_BUILD_STAMP`, and a non-interactive `ssh` session does not read a profile.
 **`provider` resolves to `sqlite` on a MariaDB production host.**
 The consequence is not a loud failure, which is what makes this serious:
 1. The SQLite branch runs and tars the `jobtracker_data` volume.
 2. That volume exists in production (the backend mounts it for `/data`, exports and attachments), so
   the tar succeeds and produces a non-empty file.
 3. `verify_backup` is called **without** the `CREATE TABLE` expectation on this path — it only checks
   the file is non-empty.
 4. The script prints `Backup verified: …` and the deploy proceeds.
 The operator sees a green backup line and a new file in `/opt/job-tracker/backups`. There is no
 MariaDB dump. If the deploy then damages the schema, there is nothing to restore.
 **Same root cause, two further silent effects:**
 - `APP_PUBLIC_BASE_URL` is likewise unset, so the public smoke check at the end of `deploy.sh` never
  runs. `deploy/first-production-deployment.md` states it does.
 - `OLLAMA_MODEL` is unset, so the post-deploy Ollama warmup never runs.
 **How to confirm before trusting any fix:** run the deploy and check that the newest file in
 `/opt/job-tracker/backups` is named `jobtracker-<dbname>-<stamp>.sql.gz`, not
 `jobtracker-sqlite-<stamp>.tar.gz`. The filename alone distinguishes the two paths.
 **Closed.** `deploy/deploy.sh` now:
 1. **Loads `/opt/job-tracker/shared/.env` into its own shell** before any decision. Parsed line by
   line rather than sourced, because a compose `.env` is not a shell script. Variables already set in
   the environment win, so CI-provided `APP_VERSION` and friends still override the file. No value is
   echoed.
 2. **Requires `DATABASE_PROVIDER` explicitly.** The `:-sqlite` default is gone. Missing means stop and
   say which variable is missing; an unrecognised value means stop.
 3. **Validates the rest of the deployment configuration before the backup**, and therefore before
   anything is built, stopped or replaced: the connection string when the provider needs one,
   `AI_SERVICE_TOKEN` (compose declares it with `:?`, so missing it would otherwise kill the stack
   after the images are built) and `AUTH_JWT_KEY` (the backend throws at startup on a blank key, after
   the containers have been replaced). Names only in the output, never values.
 4. **Verifies each backup against its own format.** A MariaDB dump must be valid gzip, contain
   `CREATE TABLE`, and carry the `Dump completed` trailer that `mariadb-dump` writes last — so a dump
   that died partway through is rejected. A SQLite archive must be valid gzip and actually contain
   `jobtracker.db`. A tar can no longer pass the dump check, which is precisely what went wrong.
 5. **Resolves the SQLite volume by its real, project-prefixed name** and fails if it does not exist.
   The old code named the bare `jobtracker_data`, which on a real deploy would have silently *created*
   an empty volume and backed that up — a second instance of the same class of bug, found while
   testing the fix.
 Two side effects of the same root cause are also resolved: `APP_PUBLIC_BASE_URL` now reaches the
 script, so the post-deploy public smoke check actually runs (and the script says so explicitly when it
 is unset rather than skipping in silence), as does `OLLAMA_MODEL` for the warmup.
 *Verified against a seeded MariaDB 11 container and real Docker volumes:*
 | Scenario | Result |
 |---|---|
 | MariaDB production-style `.env` | ✅ env loaded, MariaDB path selected, `.sql.gz` written containing `CREATE TABLE`, the seeded row, and the `Dump completed` trailer |
 | `DATABASE_PROVIDER` missing | ✅ exits 1 naming the variable; nothing built, stopped or replaced; no backup file |
 | `DATABASE_PROVIDER` unrecognised | ✅ exits 1 |
 | `AI_SERVICE_TOKEN` / `AUTH_JWT_KEY` missing | ✅ both reported in one pass, exits 1 |
 | SQLite with a populated volume | ✅ `.tar.gz` written and verified |
 | SQLite volume containing no `jobtracker.db` | ✅ rejected, file deleted |
 | SQLite volume name not present | ✅ rejected before any container ran; no stray empty volume created |
 | Broken database credentials | ✅ exits 1, no partial file left behind |
 | Truncated dump (no `CREATE TABLE`) | ✅ rejected |
 | Dump with the trailer stripped | ✅ rejected as truncated |
 | Secret leakage across every test's output | ✅ zero occurrences of any password, token or key |
 ### N2. ~~`/health` always reports `version: unknown` under Docker~~ — **CLOSED 2026-07-19**
 *Original finding:* the endpoint read the `APP_VERSION` **environment variable**, but compose passes
 `App__Version`, which binds to the `App:Version` **configuration key**. No variable by that name
 existed in the container, so the version was always `unknown`.
 **Closed.** `/health` now reads `App:Version` through `IConfiguration` — the approach
 `AdminSystemController` already used for the same value. The resolution rule (configured version, else
 assembly version) moved to a shared `BuildMetadata` helper rather than being written twice, so the
 admin page and `/health` cannot drift apart.
 *Verified against a running backend:* `App__Version=9.9.9-test` reports `9.9.9-test`; unset reports the
 assembly version rather than `unknown`. Tests pin the configuration **key**, including that an
 `App__Version` environment variable binds to `App:Version` — the original bug failed silently, so a
 behavioural test alone would not have caught it.
 ---
 ### CV languages depended on the host's ICU data — **CLOSED 2026-07-19**
 Found while investigating a reported CI failure. `HumanLanguageCatalog` built its lookup table solely
 from `CultureInfo.GetCultures`, so which languages counted as human languages depended on the
 machine's culture data rather than on the CV. Measured: **806 cultures** on a normal Windows or Linux
 box, **exactly 1** under globalization-invariant mode.
 Consequences, all silent — no error, no log line:
 | Environment | Behaviour |
 |---|---|
 | Full ICU | Correct |
 | Trimmed ICU data | Names present in the reduced set survive, the rest are dropped — a CV keeps English and loses Norwegian |
 | Invariant mode | **Every** language dropped; a CV import loses its Languages section entirely |
 **The tests were right and were not modified.** The catalog is now seeded explicitly with the languages
 a CV realistically lists, before the culture enumeration, which still runs and still adds breadth.
 Nothing in the seed collides with a technical skill — `Go`, `Java`, `Swift`, `Rust` and `Basic` are
 deliberately absent, and `Basic` is also a proficiency level.
 *Verified:* 420 tests pass in four environments — Windows and Linux, each with full ICU and with
 `DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1`. Before the fix the invariant runs failed 5 tests. A new
 `HumanLanguageCatalogTests` pins the seeded catalog, confirmed non-vacuous by removing the seed and
 watching 15 tests fail.
 **Why this matters beyond the bug:** the defect was invisible to a normal local test run. It is the
 clearest evidence in this review that *passing locally* and *correct in the deployed container* are
 different claims — which is exactly what B2 below leaves unresolved.
 ---
 ### Follow-up reminder index never created on MariaDB — **CLOSED 2026-07-19**
 Found during the release-candidate audit, watching a real backend boot against an empty MariaDB.
 `IX_JobApplications_OwnerUserId_FollowUpAt` was declared `(OwnerUserId(191), FollowUpAt)` with **no
 prefix length on `FollowUpAt`**. But `FollowUpAt` is `text` on MariaDB — `JobApplications` is
 migration-owned and the migration was scaffolded against SQLite, which stores `DateTimeOffset` as
 `TEXT`. A text column cannot be indexed without a prefix length, so the index failed the 3072-byte key
 check on **every** MariaDB boot, was caught by `TryCreateIndex`, and was silently skipped.
 Two consequences, both real:
 - The follow-up reminder query (`OwnerUserId + FollowUpAt`) ran unindexed — the index the code says it
  creates never existed.
 - Every healthy boot logged `Specified key was too long` — the exact string
  `deploy/first-production-deployment.md` lists as a **stop-and-roll-back** signal. An operator
  following the runbook could abort a perfectly good deploy on a false alarm.
 **Fixed** by prefixing `FollowUpAt(20)`, matching the `Status(50)` fix the author already applied one
 line below for the identical longtext problem. ISO-8601 date strings sort lexicographically, so a
 20-char prefix stays useful for the reminder scan.
 *Verified* on a fresh empty MariaDB 11 container: index now created with both key parts, **zero**
 "too long" lines, **zero** skipped indexes, **zero** unhandled exceptions, 42 tables, app healthy.
 Audited the whole class — `FollowUpAt` was the only unprefixed text column in any reconciler composite
 index; the datetime columns on reconciler-owned tables are `datetime(6)`.
 ### Timeline day-grouping test was nondeterministic — **CLOSED 2026-07-19**
 `Timeline_groups_by_day_newest_first` seeded two "same day" events with
 `DateTime.Now.AddDays(-3).AddHours(2)`, which crossed midnight whenever the wall clock was within two
 hours of it — failing the test ~2 hours out of every 24, including in CI. The service is correct
 (groups by `.Date`); the test was fixed to anchor the older events to `DateTime.Today` plus fixed
 hours. Verified passing at 22:35 local (inside the failing window) and on Linux, both ICU modes.
 ---
 ## BLOCKED — requires external action
 One item. Nothing in this repository can clear it.
 ### B2. CI is red — deployment is gated on it
 Unchanged and still external to the repository. Commit `8f73548` changed one markdown file and its
 test job failed in the same duration band as every other run; a change that cannot affect compilation
 cannot fail a test job. The `deploy` job declares `needs: test`, so nothing promotes until this clears.
 Everything in this review is therefore **local verification**. Blocked on, and needing you:
 - **Job logs** — the Gitea API returns 401 unauthenticated. A read-scoped token would unblock this
 - **`journalctl -u act_runner --since '2 hours ago'`** on the runner host
 - **The act_runner container/service configuration**
 Evidence in `docs/infrastructure/runner-investigation.md`.
 **The decision this forces.** Either fix the runner, or deploy deliberately from a locally verified
 commit with CI knowingly red. The second is a defensible choice for a self-hosted single-node
 deployment — but it should be a decision rather than a default, and it means the 420 backend tests and
 128 frontend tests have only ever run on one machine.
 ---
 ## Accepted for the first release (non-blocking)
 Known limitations, deliberately shipped as-is. None should stop a deploy; all are worth knowing.
 | # | Finding | Why it is not blocking |
 |---|---|---|
 | N1 | ~~**`.env.example` omits `DATABASE_PROVIDER` and `JOBTRACKER_CONNECTION_STRING`**~~ — **CLOSED 2026-07-19** | Both added to the template with the connection-string host caveat. `deploy.sh` now hard-fails without `DATABASE_PROVIDER`, so the template had to name it |
 | N2 | ~~**`/health` always reports `"version":"unknown"` under Docker**~~ — **CLOSED 2026-07-19** | Fixed; see *Closed since the first pass* above |
 | N3 | **The post-deploy gate in `deploy.sh` checks `.State == running`, not health** — a container can be `running` while `starting` or `unhealthy` | Harmless in practice: `compose up` already blocks on `service_healthy` for the frontend's dependency, so an unhealthy backend aborts the deploy before this check is reached. The check is weaker than it looks, not wrong |
 | N4 | **Table-count discrepancy across documents** — `database-ownership.md` records 40 tables on MariaDB; `release-checklist.md` and the runbook say ~42 | Both were measured, at different points in Phase 5. Use "the tables listed in `database-ownership.md` all exist" as the check, not a number |
 | N5 | **`ai-service` opens completely if `AI_SERVICE_TOKEN` is the empty string** — the middleware is `if AI_SERVICE_TOKEN and …`, so a blank token disables the check rather than failing closed | Compose declares the variable with `:?` on both services, so the stack refuses to start without it. No defence in depth behind that, though |
 | N6 | **Logging is console-only** | Captured by `docker logs`; adequate for a single-node self-hosted deployment. No retention, structure or aggregation |
 | N7 | **No global exception handler** | Unhandled errors return a bare 500 with no correlation id. No stack traces leak outside Development, so this is a supportability cost, not a security one |
 | N8 | **`ClientErrorsController` is anonymous** | By design — browser error reports must work on pages reached before sign-in. Size-limited to 32 KB and field-truncated; stores nothing |
 | N9 | **A code rollback does not recover rows written to the new tables** | Correct and documented. A *code* rollback loses nothing; only a *database restore* discards checklist items, cover letter versions, interview prep and AI history created since the dump |
 | N10 | **Prompt quality is unmeasured** | Interview generation now receives analysis and match context. Whether the output is *better* is a judgement no test makes |
 ---
 ## MANUAL VERIFICATION — only the owner can do these
 Nothing in this review, and nothing in the test suite, covers any of these — every automated check
 stops at the authentication boundary, and none of it has touched production data.
 ### Post-deployment checklist — walk this in order
 Each item names what "wrong" looks like, because "it loaded" is not a check.
 | # | Area | Check | Wrong looks like |
 |---|---|---|---|
 | 1 | **Login** | Sign in with a real existing account | Password rejected, or the session drops on refresh — the latter means `AUTH_JWT_KEY` changed |
 | 2 | **Existing applications** | The list loads and the **count matches the pre-deploy number** | Any drop. This is the single most important check on the page |
 | 3 | **Application workspace** | Open one application: Overview, Checklist, Timeline, Analysis, Match | A section erroring. Empty is **correct** for applications that predate the feature |
 | 4 | **Career profile** | Opens with your real experience, education, skills — and **languages** | Languages missing or reduced to English only. That is the bug fixed in `9681618`; if it reappears the container's ICU data differs from what was tested |
 | 5 | **CV builder** | Lists existing variants; open one; it renders with its theme | A variant that opens blank, or loses its theme |
 | 6 | **Public CV** | Open `/cv/<slug>` for an already-public variant, then **refresh it** | A 404 on refresh — that is SPA deep-link routing, not the CV |
 | 7 | **AI features** | Run one generation (interview prep or cover letter) | A 5xx, or a hang. If `ai-service` is down the deploy still succeeds — it is not a deploy gate |
 | 8 | **Attachments** | Download an existing attachment from an old application | A 404. Attachments live in the `jobtracker_data` volume; a missing file means the volume did not survive |
 Then confirm `/health` reports the version you deployed rather than `1.0.0.0`.
 ### Backup and restore readiness
 A full backup → verify → restore → start-the-app rehearsal was completed on 2026-07-19 and is recorded
 in [`docs/operations/production-backup-verification.md`](operations/production-backup-verification.md):
 42 tables dumped and restored into a clean MariaDB 11 container, every table's row count identical,
 content and foreign keys intact, and the application started healthy against the restored database.
 **That rehearsal used seeded data, not production data.** No production host was contacted. What
 remains unproven is the one thing a rehearsal cannot prove: that it works **on your database**.
 - [ ] **Take a MariaDB dump by hand from production and restore it into a scratch database.** Not a
      formality. Container verification says the mechanism is sound; it says nothing about your data
      volume, your disk space, or your MariaDB version's dump quirks.
 - [ ] **Confirm `/opt/job-tracker/backups` exists and has room.** `deploy.sh` creates it, but a
      full disk fails the backup and therefore the deploy.
 - [ ] **Know which backup is the restore point** before you start. After the deploy, confirm the newest
      file is `jobtracker-<database>-<stamp>.sql.gz` — a `jobtracker-sqlite-*.tar.gz` would mean the
      environment is wrong, though the provider check should now stop that first.
 - [ ] **Check non-ASCII text survived the round trip.** Open a restored CV or career profile containing
      `æ`, `ø` or `å` and confirm it is not mangled. The rehearsal data was ASCII-heavy, so this is the
      most likely silent failure and the least likely to be noticed.
 ### Before deploying
 - [ ] **Confirm `/opt/job-tracker/shared/.env` contains `DATABASE_PROVIDER=mariadb` and
      `JOBTRACKER_CONNECTION_STRING`.** `deploy.sh` now aborts without them, so a missing value costs
      an aborted deploy rather than a bad backup — but check first and skip the round trip.
 - [ ] **Confirm the connection-string host resolves *from inside the backend container*** —
      `127.0.0.1` there means the container, not the Docker host.
 - [ ] **Record the current commit** (`git rev-parse HEAD`) and the current row counts for
      `JobApplications` and `Companies`. The row counts are the check that matters most afterwards.
 ### After deploying — beyond the eight-point checklist above
 - [ ] **Email and calendar integrations still connect**, if you use them — the OAuth callbacks were
      reviewed statically but never exercised against live Google or Microsoft.
 - [ ] **The full journey once, deliberately** — empty profile through to a recorded outcome. No one has
      walked it. Friction in that path is currently unmapped.
 ### Why this section cannot shrink
 Authenticated smoke testing is not an automation gap that more work would close. Signing in requires a
 password, and no automated step in this repository should ever handle one. The 420 backend tests
 verify that the authorization boundary *exists* and holds; only you can verify what is behind it, with
 your data.
Author	SHA1	Message	Date
cesnimda	5c5a572cfc	docs(ops): record release-candidate audit findings CI and Deploy / test (push) Failing after 1m8s Details CI and Deploy / deploy (push) Has been skipped Details Adds the two issues found and fixed during the release-candidate audit to release-candidate-review.md: the follow-up reminder index that never created on MariaDB (fix in the preceding commit), and the nondeterministic timeline day-grouping test. Corrects database-ownership.md drift: the MariaDB startup scenarios now report 42 tables (measured in every scenario this audit), not the stale 40 from before the last Phase 5 tables were added, and adds the partially-migrated heal scenario (35 -> 42) that was verified. All claims reflect behaviour verified in this audit: 420 tests on Windows and Linux in both ICU modes, all three Docker images built, four database startup scenarios against live MariaDB 11 and SQLite, and a full backup -> restore -> app-start cycle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-19 22:53:43 +02:00
cesnimda	95646e1d53	fix(db): create follow-up reminder index on MariaDB The reconciler's IX_JobApplications_OwnerUserId_FollowUpAt was declared as (OwnerUserId(191), FollowUpAt) with no prefix length on FollowUpAt. But FollowUpAt is `text` on MariaDB -- JobApplications is migration-owned and the migration was scaffolded against SQLite, which stores DateTimeOffset as TEXT. A text column cannot be indexed without a prefix length, so this index failed the 3072-byte key check on EVERY MariaDB boot, was caught by TryCreateIndex, and was silently skipped -- leaving the follow-up reminder query (OwnerUserId + FollowUpAt) unindexed. Two consequences, both real: - the index the code intends to create never existed on MariaDB - every healthy boot logged "Specified key was too long", which deploy/first-production-deployment.md lists as a STOP-AND-ROLL-BACK signal -- so an operator following the runbook could abort a good deploy The author already handled the identical problem for the longtext Status column one line below with Status(50). Apply the same fix: FollowUpAt(20). ISO-8601 date strings sort lexicographically, so a 20-char prefix ("YYYY-MM-DD HH:MM:SS") keeps the index useful for the reminder scan. Verified on a fresh empty MariaDB 11 container: the index is now created (both key parts present), zero "Specified key was too long" lines, zero skipped indexes, zero unhandled exceptions, 42 tables, app healthy. This was the only unprefixed text column in any reconciler composite index -- the datetime columns on reconciler-owned tables are datetime(6). SQLite is unaffected (its CREATE INDEX has no key-length limit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-19 22:52:39 +02:00
cesnimda	c1ff98ff4c	fix(test): make timeline day-grouping test deterministic Timeline_groups_by_day_newest_first seeded two "same day" events as DateTime.Now.AddDays(-3) and DateTime.Now.AddDays(-3).AddHours(2). When the wall clock was within two hours of midnight the second timestamp crossed into the next calendar day, so the service grouped them into two days instead of one and the test failed (expected 2 day-groups, got 3). The service is correct -- it groups by e.At.Date, which is the intended behaviour and what the test name asserts. The test was nondeterministic, failing roughly two hours out of every twenty-four, including in CI whenever CI ran late in the day. Anchor the two older events to DateTime.Today plus fixed hours (9 and 11) so they always land on the same calendar day regardless of wall-clock time. The "today" event stays DateTime.Now so the "Today" label assertion still exercises the real path. Verified: 420 tests pass at 22:35 local (the failing window) and on Linux with full ICU and under DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-19 22:52:24 +02:00
cesnimda	834a775c9d	docs(release): final deployment checklist Validation only. No application behaviour changed. Verified against the code rather than the prose, by line number: Before deployment -- validate_deploy_config (deploy.sh:343) and backup_database (:347) both precede the build (:375) and the container replacement (:382), so nothing is built, stopped or replaced without a verified restore point. During deployment -- startup runs ReconcileSchema (:1965), Database.Migrate (:1973), ReconcileSchema (:1984). All seven Phase 4/5 migrations confirmed to have a literally empty Up body, which is what makes a code rollback safe. Health checks and rollback -- backend and frontend healthchecks present, frontend gated on backend health, rollback documented in two places with the code-vs-database distinction. After deployment -- added an eight-point owner checklist covering login, existing applications, workspace, career profile, CV builder, public CV, AI features and attachments. Each item names what wrong looks like, because "it loaded" is not a check. Merged the previous overlapping "After deploying" list into it rather than leaving two competing checklists. Recorded the CV language ICU defect as closed, with the note that it was invisible to a normal local test run -- the clearest evidence in this review that passing locally and correct in the deployed container are different claims. Sections are now READY / BLOCKED / MANUAL VERIFICATION, with accepted limitations kept separate. Test count updated to 420. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-19 21:36:26 +02:00
cesnimda	df9322f5c0	docs(ops): verify production backup restore Full backup -> verify -> restore -> start-the-app rehearsal of the deploy.sh backup path against MariaDB 11. Verified: the real backup_database function selected the MariaDB path from DATABASE_PROVIDER=mariadb, produced a valid .sql.gz with 42 CREATE TABLE statements and an intact "Dump completed" trailer, restored into a separate empty MariaDB container, and the application then started healthy against the restored database with the reconciler finding nothing to do. All 42 tables matched on row count, and content survived including foreign key relationships and career profile JSON. This is a rehearsal, NOT a verification of production data. No production host was contacted and no production data was read. This machine has no route to production: no /opt/job-tracker, no DATABASE_PROVIDER or connection string in its .env, and the local stack runs SQLite. Production host, user and key are CI secrets not available here. The document leads with that scope limit, records the commands to run against production with values substituted, and ends with the checklist that actually closes the gap -- including checking that non-ASCII CV text survives the round trip, which the ASCII-heavy seed data did not prove. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-19 21:33:27 +02:00
cesnimda	96816186cb	fix(cv): preserve human languages during structured CV normalization HumanLanguageCatalog built its lookup table solely from CultureInfo.GetCultures, so which languages counted as human languages depended on the host's ICU data rather than on the CV. Measured: 806 cultures on a normal Windows or Linux machine, exactly 1 under globalization-invariant mode, and an English-only subset on a container with trimmed ICU data. Consequences by environment, all silent: - full ICU: correct - trimmed ICU: canonical names present in the reduced data survive and the rest are dropped, so a CV keeps English and loses Norwegian - invariant: every language is dropped and a CV import loses its Languages section entirely, with no error The tests were right and are unchanged. Seed the catalog explicitly with the languages a CV realistically lists, before the culture enumeration, which still runs and still adds breadth. Nothing in the seed collides with a technical skill -- Go, Java, Swift, Rust and Basic are deliberately absent, and Basic is also a proficiency level. Verified 420 tests pass in four environments: Windows and Linux, each with full ICU and with DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1. Before this change the invariant runs failed 5 tests. No test was modified, skipped or relaxed. Added HumanLanguageCatalogTests to pin the seeded catalog, confirmed non-vacuous by removing the seed and watching 15 tests fail. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-19 19:19:17 +02:00
cesnimda	de35947244	docs(ops): finalize release candidate status Restructured into READY / BLOCKED / MANUAL, with accepted limitations kept separate. B1 (backup selected the wrong provider and reported success) and N2 (/health always reported version: unknown) are both closed and verified; their original findings are kept because the failure modes are worth understanding. N1 closed with the .env.example additions. One blocker remains and it is external: the CI runner. Stated with what it needs from the owner, and with the decision it forces -- fix the runner, or deploy deliberately from a locally verified commit knowing CI is red. MANUAL now leads with backup and restore readiness: the mechanism is verified against containers, but only the owner can prove it works on production data. Added a note that authenticated smoke testing is not an automation gap that more work would close -- sign-in needs a password, and no automated step here should handle one. Validation only. No application behaviour changed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-19 19:01:49 +02:00
cesnimda	8f6f2ba8d6	fix(health): report configured application version /health read the APP_VERSION environment variable directly, but docker-compose passes App__Version, which binds to the App:Version configuration key. The variable under that name never existed in the container, so the endpoint always reported "unknown". Read App:Version through IConfiguration, the approach AdminSystemController already used for the same value. The resolution rule (configured version, else assembly version) moves to a shared BuildMetadata helper rather than being written twice; AdminSystemController now calls it, so the admin page and /health cannot drift apart. Local development is unaffected: nothing sets App:Version there, and the assembly-version fallback still applies. Tests pin the configuration KEY, not just the behaviour, including that an App__Version environment variable binds to App:Version. The original bug failed silently, so a behavioural test alone would not have caught it. Verified against a running backend: App__Version=9.9.9-test reports 9.9.9-test; unset reports the assembly version rather than "unknown". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-19 18:59:26 +02:00
cesnimda	66b02bcab8	fix(deploy): load production environment before backup deploy.sh symlinked /opt/job-tracker/shared/.env for docker compose but never loaded it into its own shell. Its own decisions therefore ran against an empty environment: DATABASE_PROVIDER fell back to sqlite on a MariaDB host, so the deploy tarred the data volume, printed "Backup verified" and continued with no database dump. The operator saw a green backup line and a new file in the backups directory, and had no restore point. Load the shared env before any decision. Parsed line by line rather than sourced, because a compose .env is not a shell script and an unquoted value containing spaces would execute as a command. Values already in the environment win, so CI-provided APP_VERSION and friends still override the file. No value is echoed. Remove the sqlite default. DATABASE_PROVIDER must be stated; missing or unrecognised aborts the deploy. Validate deployment configuration before the backup, and so before anything is built, stopped or replaced: the connection string when the provider needs one, AI_SERVICE_TOKEN (compose declares it with :?) and AUTH_JWT_KEY (the backend throws on a blank key). Names in the output, never values. Verify each backup against its own format. A dump must be valid gzip, contain CREATE TABLE, and carry the "Dump completed" trailer, so a dump that died partway through is rejected. An archive must contain jobtracker.db. A tar can no longer pass the dump check. Also resolve the SQLite volume by its project-prefixed name and fail if absent. The bare jobtracker_data name would have silently created an empty volume and backed that up -- the same class of bug, found while testing this fix. Verified against a seeded MariaDB 11 container and real Docker volumes: provider selection, all four validation failures, both backup formats and their failure paths, truncated and trailer-stripped dumps, and zero secret occurrences across every test's output. Docs updated for the drift: deploy/README.md, deploy/first-production- deployment.md, docs/release-candidate-review.md (B1 closed) and .env.example, which now names the two database variables. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-19 18:44:23 +02:00
cesnimda	ab53582c71	docs(ops): add release candidate review Final validation pass before the first production deployment. Findings only; no application code changed. Verified against the implementation rather than the other documents: migration and reconciler ownership, startup order, authorization coverage, tenant isolation, AI service protection, file access, the five architecture rules, container dependency ordering and failure behaviour. Two blocking items, one new: deploy.sh symlinks the shared .env for docker compose but never sources it, so DATABASE_PROVIDER is unset in the script's own shell and backup_database takes the SQLite branch on a MariaDB host. It tars the data volume, verify_backup only checks the file is non-empty on that path, and the deploy reports a verified backup that contains no database dump. Same root cause silently disables the APP_PUBLIC_BASE_URL smoke check and the Ollama warmup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-19 18:19:37 +02:00