Production AI rollout and rollback

Updated: 2026-08-15

Status: plan only. The 2026-08-15 activity was read-only. No backup, model pull, service restart, firewall/network change, deployment, database mutation, or restore was performed.

Current rollback anchors

Deployed application commit: de937d25dc5e; configured version 157.
The production checkout is dirty because deploy/deploy.sh has a mode-only change. Preserve and review that state before any checkout/reset.
JobTracker Ollama image digest: sha256:f1a705f2bd113fb8d15f85f7c217f0dc5f6bebda6b0cc42b82c3ad165ffcb9dc.
Existing model: qwen2.5:7b, ID prefix 845dbda0ea48.
Existing model must not be removed during benchmark or rollout.
All four current application containers are healthy, restart unless-stopped, and had zero restart count at capture.

Environment secret values were deliberately not copied. Recovery still depends on the protected production environment file and data-protection material; their custody has not been verified.

Hard stop conditions before mutation

PR 28 must pass current CI and be approved for the normal deployment path.
Review/preserve the production-only mode change to deploy/deploy.sh; never erase it with a blind reset.
Remove JobTracker's host-published 11434 listener and broader network membership, and decide whether the separate shared Ollama listener is an approved private dependency. Verify no Internet/Traefik route and no unintended LAN clients.
Produce a current MariaDB dump, a recoverable owner-file volume snapshot, data-protection-key recovery, protected tombstone storage, and non-secret configuration inventory. Keep tombstones outside any restored application backup.
Restore the complete set into an isolated scratch environment and verify user/application/Career row counts, owned file access, non-ASCII content, key decryption, and tombstone replay. Never restore over live production for rehearsal.
Define RPO, RTO, backup expiry, tombstone retention, legal hold, and account-deletion provider/cache semantics.
Keep account deletion disabled and all previously inert workers off.
Resolve resource limits for the AI sidecar/Ollama or document measured stop thresholds. Current containers have no CPU, memory, or PID caps.
Confirm at least 25% free on the filesystem receiving image/model layers and enough root headroom for build/temp/log growth. Root is currently 83% used; Docker/model data is on the spacious secondary filesystem.
Use only synthetic benchmark inputs. Do not use production CVs, correspondence, job descriptions, or prompts.

If any item fails, stop. Do not work around it by deleting models, pruning unknown Docker data, weakening health checks, exposing ports, or enabling external processing.

Intended release-branch topology

Frontend: reachable only through the approved reverse proxy network; no direct host 3000 publication.
Backend: internal application/proxy networks; no host port.
AI sidecar: only backend plus private AI network; no host port; service token required for non-health endpoints.
Ollama: private AI network only when bundled, or a separately approved private endpoint. No all-interface/public host publication.
Routing: local-first with external processing globally disabled by default and additionally requiring per-user opt-in when later approved.
Workers: durable AI worker remains disabled until queue/privacy/entitlement/notification production checks pass.

The deployed runtime does not yet have this topology or routing contract.

Backup gate

The observed 21 MariaDB dumps are gzip-valid but old and database-only. Before an approved deployment:

create a new out-of-band MariaDB dump through the supported script;
record its UTC time, size, duration, client/server versions, and SHA-256 without recording credentials;
snapshot/copy the complete JobTracker data volume, including attachments, CV artifacts, generated exports, and account exports;
separately secure data-protection keys and the account-deletion tombstone ledger;
record the exact non-secret Compose/image/model digests;
restore all components to an isolated scratch stack and complete the documented verification matrix;
retain the pre-change set until the observation window and rollback decision expire.

Database-only success is not a complete backup.

Benchmark sequence after approval

Capture idle host/container/GPU metrics and ollama ps.
Keep qwen2.5:7b as the baseline; do not pull multiple candidates together.
Run one synthetic request, then verify loaded model, GPU/CPU split, VRAM, RAM/swap, temperature, power, latency, tokens/second, JSON validity, and logs free of content.
Stop if swap grows materially, memory pressure remains elevated, root free space drops below the agreed threshold, the GPU overheats/throttles, containers become unhealthy, or latency breaches the package threshold.
Run the checked-in synthetic evaluation set at 4K and 8K context. Try 16K only after measured headroom.
Repeat enough times to distinguish warm/cold load and variance. One successful response is not a model decision.
Test queue congestion, timeout, restart, local outage, privacy opt-out, Free/Pro denial, and two-tenant isolation before any rollout.
Record exact model tag, digest, license, quantization, context, options, resource use, quality and failure categories.

Do not change production's selected model merely because it fits in VRAM.

Staged rollout

Deploy configuration/topology/security changes with AI workers and account deletion still disabled.
Verify admin version badge equals the deployed commit and confirm direct ports are closed.
Re-run anonymous/authenticated health and core non-AI smoke checks.
Run one admin-only synthetic local inference while watching CPU/RAM/swap/GPU/container health.
Enable one controlled synthetic queue canary; verify durable operation, notification, cancellation, restart recovery, entitlement and tenant boundaries.
Observe for the agreed period before increasing concurrency above one.
External fallback remains off until explicit privacy/provider approval and separate synthetic verification.
Account deletion remains off until its retention/restore/provider/cache rehearsal is complete.

Rollback plan

Rollback must use the supported deploy/runbook path, not ad-hoc container deletion.

Stop admission of new AI work; leave durable queued records intact.
Request cancellation/drain and wait for the bounded deadline. Do not kill the database.
Set AI worker and external processing gates back to false.
Redeploy the recorded pre-change commit/images through the approved script, preserving the reviewed production script state.
Keep the pre-existing qwen2.5:7b; changing application selection does not require deleting candidate model files.
Restore database/files/keys only if application rollback is insufficient and only from the verified complete pre-change set. Apply deletion tombstones before readiness.
Verify container health, direct-port closure, login, application counts, Career/CV, attachments, non-AI behavior, operation reconciliation, and admin version.
Record incident times and sanitized failure categories. Do not copy prompts or user content into the report.

Expected interruption is one controlled application-container recreation plus model cold-load time. Exact duration remains unmeasured and must be captured during the approved rehearsal.

Outstanding approvals

network/firewall/container mutation to close Ollama and frontend host ports;
complete backup and scratch restore authority;
retention/tombstone/legal decisions;
model pull and synthetic production benchmark;
deployment and worker activation;
external-provider fallback and any real-provider checks.

Until those approvals and gates are complete, the correct state is the current read-only evidence plus disabled release-branch features—not a partial production rollout.

7.9 KiB Raw Blame History