docs(ops): record successful production rollout

This commit is contained in:
cesnimda
2026-09-01 09:16:29 +02:00
parent 6b8c560d63
commit 2a0672e1cd
2 changed files with 8 additions and 9 deletions
+6 -7
View File
@@ -1,6 +1,6 @@
# Blockers
Updated: 2026-08-31
Updated: 2026-09-01
## Stripe billing
@@ -19,14 +19,13 @@ Updated: 2026-08-31
- **Recommended:** Monitor Turnstile and rate-limit failures during the first public rollout; keep email verification required.
- **Current status:** Production returns `allowRegistration=true`, `turnstileEnabled=true`, `googleEnabled=true`, and `microsoftEnabled=false`. A registration request without a Turnstile token is rejected with HTTP 400. SMTP is configured and enabled. The operator reports `AUTH_REQUIRE_EMAIL_VERIFICATION` is now enabled; the disposable interactive signup is still required to prove the deployed behavior end to end.
## Production verification and deployment
## Production verification
- **Blocked:** Authenticated production smoke tests, backup restore verification against real data, OAuth-provider checks, and deployment.
- **Why:** Read-only host access is available, but authenticated smoke, port/network closure, complete backup restore, provider checks and deployment require credentials and/or operator-authorized production mutations.
- **Required:** After the current pull request passes CI and is approved, follow `docs/release-candidate-review.md` and `docs/release-checklist.md` on the production host. Confirm the admin-only version badge matches the deployed commit, then run the authenticated application workspace, Career, CV, attachment, email-verification and rollback checks.
- **Blocked:** Authenticated production smoke tests, backup restore verification against real data, OAuth-provider checks, and rollback rehearsal.
- **Why:** The authorised deployment is complete, but authenticated smoke, port/network closure, complete backup restore, provider checks and failure-injection rollback require credentials and/or separately authorised production mutations.
- **Required:** Confirm the admin-only version badge matches version 286 / commit `ea7d9f3`, then run the authenticated application workspace, Career, CV, attachment, email-verification and rollback checks from `docs/release-candidate-review.md` and `docs/release-checklist.md`.
- **Recommended:** Verify backup/restore before deployment, then exercise login, existing application counts, Career Workspace, public CV refresh/download, AI, and attachments in order.
- **Current access check:** Read-only SSH access is confirmed. All four JobTracker containers are healthy with zero observed restarts, but root free space is now 36 GiB (83% used). The production checkout is at `de937d25dc5e` / app version `157` and has an unreviewed mode-only change to `deploy/deploy.sh`. No production change or deployment was attempted.
- **Current status:** On 2026-08-31 the anonymous public liveness endpoint `https://jobs.cesnimda.uk/health` returned HTTP 200 with version `276`, and the Gitea host returned HTTP 200; the earlier public 502 is no longer present. Anonymous `https://jobs.cesnimda.uk/api/health` returned HTTP 401 as expected for an authenticated API route. This proves the proxy/frontend liveness path only, not authenticated application behavior or database readiness. Gitea run 696 failed after repeated truncated Playwright downloads and then a runner-level exit 139 before any test assertion; the workflow now installs through the lockfile CLI and retries only that transient process failure without hiding real test failures. Current remote CI, backup/restore, authenticated smoke and rollback still require operator-controlled environments. Read-only PROD-001 inventory found the JobTracker Ollama and frontend published on all host interfaces, the newest gzip-valid MariaDB backup dated 2026-08-02, no observed scheduled JobTracker backup, and no owner-file/key/tombstone recovery bundle. Close these rollout gates before deployment; see `docs/production/production-ai-hardware-assessment.md`.
- **Current status:** Gitea run 705 (application version 286, commit `ea7d9f3`) completed both the complete test job and remote deployment successfully on 2026-09-01. Public `/health`, `/ready`, and `/api/auth/config` returned HTTP 200 after replacement; liveness reported version 286 and readiness reported `ready`. This proves the proxy/frontend, API and database readiness path, but not authenticated application behavior, provider delivery or restore/rollback behavior. Read-only PROD-001 inventory previously found the JobTracker Ollama and frontend published on all host interfaces, the newest observed gzip-valid MariaDB backup dated 2026-08-02, no observed scheduled JobTracker backup, and no owner-file/key/tombstone recovery bundle. Those remaining gates still require operator-controlled verification; see `docs/production/production-ai-hardware-assessment.md`.
- **Repository-side rollback status:** Core images are now tagged by commit, the prior running backend/frontend images are retained, and any failure after replacement triggers an automatic core rollback while preserving the failed CI status. Compose configuration validates locally. A real rollback rehearsal remains blocked because the local Linux Docker daemon is unavailable and production mutation is not authorised; run the failure-injection rehearsal on a disposable Linux Docker host before relying on it in production.
## Account deletion retention and restore policy
+2 -2
View File
@@ -170,8 +170,8 @@ Updated: 2026-09-01
- **Immediate order:** all sixteen immediate repository items are complete locally, including the original UI/release queue plus SEC-009 cache/tombstone safety, worker restart clocks, universal AI accounting, email-token/Stripe lifecycle tests, exhaustive Job email selectors, the repaired migration chain, CV/public-edge hardening and measured admin/mail scaling. PROD-001 read-only evidence and the PROD-003 plan-only harness are also complete. The final audit is checking tooling/documentation before declaring only external blockers remain.
- **Status counts:** 8 `VERIFIED LOCALLY`; 27 `IMPLEMENTED — NOT VERIFIED`; 0 `IN PROGRESS`; 0 `NOT STARTED`; 4 `BLOCKED`; 0 `DONE`; 0 `DEFERRED`.
- **Test status:** backend 736/736; frontend 64/64 suites and 272/272 tests; ESLint and TypeScript zero findings; AI sidecar 37/37 without warnings; Ollama benchmark harness 5/5 plus safe dry-run; optimized production build; EF model parity; SQLite/MariaDB migration scripts; blank/idempotent/populated SQLite migration-chain tests; fresh complete-chain SQLite application startup; disposable earlier-chain MariaDB 11.8 application startup; Docker Compose config; safe-failure deployment preflight; and Playwright 10/10. Supply-chain controls pass 3/3, the tracked tree is clean, and the generated CycloneDX SBOM contains 964 components. npm audit remains at zero. The final ownership migrations have generated MariaDB SQL but still need a current MariaDB runtime rehearsal; the parser image and container scan are pending because Docker Desktop's daemon is offline. Jest's slow/open-handle behavior remains recorded.
- **Deployment status:** Gitea pull-request run 609 passed the complete CI job in 4m20s. Deploy was intentionally skipped because the workflow deploys only a `push` to `main`; live remains unchanged. No merge/deployment was performed directly, no production migrations were run and the AI operation worker remains disabled by default.
- **Production status:** State unchanged. Sanitized read-only SSH inventory was performed; no logs, prompts, private rows/content or secret values were read, and no provider/model call, model pull, service restart, file/config change, backup, restore, migration or deployment occurred. It confirmed all-interface Ollama/frontend listeners and stale database-only backups as rollout blockers.
- **Deployment status:** Gitea run 705 passed the complete test and deploy jobs. Production version 286 / commit `ea7d9f3` passed public frontend, API liveness, database readiness and auth-configuration smoke checks; the prior release stayed available until the health-checked replacement window.
- **Production status:** Public liveness and database readiness are verified for version 286. Authenticated journeys, provider delivery, complete backup/restore, failure-injection rollback, port closure and retention/tombstone behavior remain unverified and explicitly blocked. No private rows/content, secrets, model inference or provider call were inspected during this verification.
- **Known regressions:** None found by automated/local browser checks. Jest still needs `--forceExit` and reports its existing open-handle notice. Email-provider/send tests are fake/local only; real delivery is not claimed. Current MAIL browser evidence is 1280×720 only because the browser surface could not resize or perform native Tab traversal. Interrupted attempts are aged after 15 minutes and notified without retry; the five-minute scan is unmeasured on a large ledger. The direct clean EF-only SQLite defect and synchronous AI accounting gap are closed; migration/reconciler dual ownership remains architectural debt.
- **Outstanding security findings:** JT-001 repository ownership remains High deployment risk until migration/inventory/provider checks; production portion of JT-002; JT-006 and SEC-009 production retention/restore plus JT-011/JT-012/JT-022 prerequisites. JT-005 foundations are implemented; AI worker activation awaits controlled rollout. Production still exposes ports contrary to the release-branch contract, and JT-007/JT-008/JT-010 lack provider/production verification.