jobtrackingapp/docs/audits/2026-09-07/resilience.md

# Resilience, partial failure and the 50 failure answers

Source baseline, labels and limitations: [README](README.md). Findings: [F01–F21](findings.md). “Needs runtime verification” means the scenario was not executed, even when source provides a strong prediction.

## Dependency and operation failure matrix

Timeouts marked current come from source; operational alert thresholds are proposals. Normal CRUD should remain useful during optional-provider failures.

| Dependency / operation | Failure and user impact | Current timeout / retry | Recovery and target UX | Alert / evidence |
|---|---|---|---|---|
| SQLite / MariaDB | Writes, auth/session checks and reads fail; active AI worker can stop host F20 | Provider defaults; observed SQLite command timeout 30s; no central transient-DB policy | Preserve forms, show service unavailable/reference; bounded worker recovery; never retry ambiguous writes without key | Readiness failures + sustained DB errors; real outage needs runtime verification |
| AI sidecar | AI/extraction fails while manual work should continue | API named client 30s; classified queue failures retry within deadline | Persistent operation status and safe retry; synchronous edits remain unsent until reviewed | Task/provider failure rate and pending age; fake-provider tests passed |
| Ollama / external AI | Missing model, DNS/HTTP failure, malformed result | Per-task/provider routing; durable default 3 attempts / 15min deadline, worker timeout 300s | Explicit failure or policy-approved fallback; never silently send private content externally | Model/task/version/category counters; live model evaluation not performed |
| Parser | Malformed/hostile/oversized input, child timeout | 5MB file; 25s child, 512MiB configured child memory, one parser at a time; container 2GiB/2CPU | Stable error, review/import can retry, previous profile retained | Parser crash/timeout/rejection counts; 37 parser tests passed; container enforcement not exercised |
| Chromium PDF | Missing executable/crash/timeout, partial output | 60s process timeout; no shared admission | Retain CV; explain download failed; retry only after cleanup/lookup; F03 bounds required | Active process/RSS/timeouts/files; no heavy live stress test |
| SMTP | Verification/reminder send fails or is suppressed | Configured timeout (dev 15s), synchronous send in Task.Run; direct auth/reminder attempts | Outbox/attempt state; do not mark suppressed as delivered; resend verification within token validity | Queue age/send failures/bounces; actual delivery not tested |
| Gmail / Graph / IMAP | Mail sync/calendar/send unavailable, revoked credentials | Provider-specific paths; send attempts preserve uncertain status | Manual tracking usable; reconnect action; do not auto-resend uncertain employer mail | Last sync success, auth/provider error, unresolved send attempt |
| Stripe | Checkout/portal unavailable; delayed webhooks, role mismatch | SDK calls; failed webhook returns non-2xx | Preserve current local state pending authoritative reconciliation; status banner; deletion waits for cancellation | Inbox age/divergence/cancel failures; sandbox required |
| Job site / NAV / translation | URL unavailable, oversized response, DNS fail | Job import client; redirects rejected, 4MB stream cap | Allow manual paste/edit; no empty replacement of a valid job | Timeouts/rejections/cache hit; real sites not called |
| File storage | Permission, missing bytes, full disk | I/O exceptions; staged attachment recovery; no universal quota | Roll back or preserve recoverable stage; do not create valid-looking broken artifact | Disk/inode pressure, orphan/missing counters, failed purge |
| Proxy / DNS / TLS | Site unreachable despite live backend | External infrastructure unknown | Correct route/cert/DNS, verify from outside; static maintenance response if necessary | External probe independent of host; not inspected live |
| Browser network | Offline/delayed/partial save/download | Axios has no shared timeout; no blanket retry; many local async handlers | Keep dirty state, bounded deadlines, retry by operation identity; distinguish ambiguous outcome | UI safe errors and request references; offline CV probe observed failure state |
| Logging / telemetry | Sink unavailable, disk growth, blocked script | Console logging; client errors best-effort | Drop/sample bounded optional telemetry; never block saves on a collector | Local dropped-event counter, disk alerts; failure injection pending |
| Backups / off-site copy | Missing/partial/inconsistent set | Script checks artifact shape; defaults 30 daily days/12 monthly sets | Preserve previous verified set; restore drill; do not prune based on unverified success | Backup failure and age from independent monitor; restore pending |
| Redis / broker / standalone document service | Not present | Not applicable | Do not add failure domains without a measured requirement | Reassess if introduced |

## Transaction and background-work review

| Workflow | Existing transaction / state design | Partial failure that matters | Minimum change |
|---|---|---|---|
| Create CV + version 1 | Two saves inside a relational transaction | Concurrent later edits still lack preconditions | Preserve creation transaction; add F01 |
| Career Profile save | Relational children/profile/history transaction; controller later updates user projection | Canonical data committed but legacy projection/user update fails | One commit if same DB context can own all writes, or explicit repairable projection; do not tell user total failure after success |
| Job create/status | Job/opportunity/history saved together in normal EF unit | Client loses response and resubmits; no creation intent key | Return/retrieve result by idempotency key |
| Attachment upload/delete | Stage/quarantine files; DB commit; promote/purge; startup reconciliation | Commit uncertainty or crash leaves metadata/file transitions pending | Preserve existing protocol, add aggregate quota reservation and missing-file alert |
| Submitted package capture | Renders/reads before DB transaction; copies files during transaction; unique package version | Concurrent max-version selection collides; source can change during capture; cleanup race | Allocate package revision safely, use unique attempt storage then publish; freeze source revisions; test file ownership of cleanup |
| AI operation | Durable owner/idempotency/lease/deadline, result reference and attempts | Provider succeeds but result commit fails; worker crashes | Result persistence idempotency and lease recovery; F20 loop recovery; classify poison failures as terminal |
| Employer send | Durable attempt and payload hash; uncertain state preserved | Provider accepted send but response/DB completion lost | Reconcile by provider message ID when possible; explicit user decision before retry |
| Reminder / verification email | Direct send; reminder timestamps committed after loop | Duplicate/suppressed/unrecorded send F07 | Durable intent/attempt ledger; SMTP cannot guarantee exactly-once |
| Stripe webhook | Re-fetch then user update then role mutation | Partial role update; overlapping subscription events | Per-customer transactional reconciliation + durable inbox |
| Account deletion | Persist stages; revoke sessions/share, quarantine, delete rows, purge, tombstone | In-flight files/results after inventory; Stripe still active; crash replay | F02 cancellation stage; coordinate active work and final inventory; exercise restore tombstones |
| Export | Account ZIP on disk; table JSON buffers; delete-on-close stream | Crash leaves ZIP; changes during export can span different logical moments | Consistency semantics/manifest; stale file cleanup; bound memory or background export |

A transactional outbox is justified for required transactional mail and local billing work because an external side effect cannot join the database commit. It is not a reason to wrap every CRUD handler in infrastructure. Keep APIs/providers outside long DB transactions. Retries need operation identity, a payload hash and a known outcome; GET that triggers AI or writes should be examined before treating all GETs as harmless retry candidates.

## Resilience test matrix / Break Jobjakt

| Scenario | Test method | Expected behavior | Current result | Fix / next evidence |
|---|---|---|---|---|
| Two tabs save same CV | Read once, sequential stale PUTs; concurrent PUT pair | Conflict; newer state preserved | **Observed fail:** 200/200, stale state replaced newer edit | F01; extend to job/profile/restore |
| Duplicate create / Save double-click | Two identical job POSTs | Same intended action returns one resource | **Observed:** two 201s, distinct IDs | Introduce explicit intent-key contract; do not block legitimate repeat applications |
| Other user's CV IDs | Direct GET/PUT/DELETE as account B | 403/404, no mutation | **Observed pass:** all 404 | Keep permanent owner tests |
| Ordinary user admin call | GET admin audit | 403 | **Observed pass:** 403 | Extend all admin verbs |
| Same-IP account export | A exports twice, B exports once | Independent account limits | **Observed fail:** B gets 429 | F06 middleware integration test |
| Formula/Unicode/HTML input | Create synthetic job `=1+1`, Åse/company emoji, HTML note; export | Store safely, safe spreadsheet cells | **Observed:** storage works; formula preserved unsafe in CSV | F05; full stored-XSS surface still runtime needed |
| Offset/DST calendar | Four input instants through DB to ICS | Preserve UTC instant; repeated autumn hour distinct | **Observed fail:** all four return wrong UTC values, HTTP 200 | F21; both database/server-zone matrices |
| Offline CV save | Browser context offline during name edit | Failure shown; no false saved state | **Observed pass:** failure message present | Preserve visible state |
| Refresh after failed save | Accept unload warning, reload | Recoverable draft if recovery supported | **Observed gap:** draft not recovered | F09; warning exists, recovery does not |
| Logout in second tab | Shared-cookie logout + auth-marker event | Clear private UI; deny future requests | **Observed gap:** other editor remains mounted | F08; distinguish visual residue from backend denial |
| EN/NB, light/dark, 375/1440 | Static build + Chromium + axe + screenshots | No serious violations/overflow | **Observed:** outer overflow 0; contrast errors all 8 states; desktop scroll-focus errors | F14; manual assistive tech still needed |
| Malformed PDF/DOCX, ZIP bomb, parser hang | Existing pytest mocked/isolated child suite | Bounded rejection, no secret inheritance | **Observed:** 37 tests pass; not a live malware corpus | Run container boundary/fuzz corpus in disposable environment |
| DB down 30s / 1h | Disconnect disposable MariaDB or fault-inject repository | Bounded errors, live host, retry recovery | **Needs runtime verification**; F20 predicts host risk with AI worker | Add loop failure test then staging drill |
| Container restart mid-write/generation | Kill only a disposable deployment after known checkpoints | Durable work resumes; ambiguous writes recover | **Needs runtime verification**; unit lease/recovery tests pass | Exercise actual process/files/DB recovery, not mocks alone |
| API latency/timeout/partial response | Local reverse proxy delays/truncates selected responses | Dirty state kept; no blind POST retry | **Needs runtime verification** | Test stale search replies, file upload/download, proxy deadline |
| Browser storage corrupt/denied | Invalid JSON/unknown schema + throw on storage APIs | Safe defaults and retained form | Unit owner/theme tests pass; full route sweep **Needs runtime verification** | I18n reads/writes need denied-storage coverage |
| Screen reader/keyboard/200–400% zoom | Narrator/NVDA + actual browser zoom + text-spacing | Reach all controls, focus/announcements, usable reflow | **Needs runtime verification** beyond existing keyboard tests/axe | Do not equate viewport width or axe with screen-reader proof |
| PDF 1000 simultaneous / 100k jobs | Synthetic capped load/fake renderer first | Global admission, bounded queries/memory | **Needs runtime verification**, source gaps F03/F10 | Never execute uncapped exhaustion on shared host |
| Disk/inodes/RSS exhaustion | Quota-mounted temp volumes/cgroup-limited staging | Fail before corruption; alert/recover | **Needs runtime verification** | Preserve backups; no filling the user's actual disk |
| Billing/mailer/OAuth outage | Fake gateways then sandbox provider callbacks | Correct pending/failed/retry state | Fake-gateway suites pass; live integrations **Need runtime verification** | F02/F07 and provider-specific contract tests |
| Backup corrupt/missing / migration halfway | Restore isolated backup, interrupt test migration | Fail deploy or forward-recover to a known checkpoint | **Needs runtime verification** beyond current migration/unit tests | F12 and signed drill record |

The harness is observational: `meetsExpectation=false` records an unmet desired control, not a flaky test that should be retried until green. No PDF/CPU flood, actual malicious executable, payment, employer email or production outage was attempted. Logs distinguish deliberate offline/401/blocked-font errors from normal workflow errors. Full packet-loss simulation, clean-machine setup, production network/console/log cleanliness and exhaustive UI state testing are outstanding, not implied by this matrix.

## What happens if…?

| # | Scenario | Current answer and evidence boundary |
|---|---|---|
| 1 | Database offline for 30 seconds | Reads/writes and DB-backed session checks fail. `/health` is process-only; `/ready` checks DB. Enabled AI-worker claim/persist errors can stop the host F20. Forms need recoverable failure UX. **Source; Needs runtime verification.** |
| 2 | Database offline for an hour | No offline DB mode. Requests continue failing, leases age, worker/host restart behavior may recur; recovery requires DB availability and operation reconciliation. Alert early, preserve accepted work. **Needs runtime verification.** |
| 3 | Docker host restarts | Named DB/file/key/tombstone volumes should survive; `unless-stopped` services restart. In-memory caches/limits reset; durable leases recover. External MariaDB/proxy start order still matters. **Needs runtime verification.** |
| 4 | API dies halfway through request | Uncommitted DB transactions roll back; completed commits can be invisible to client. Staged attachments have recovery logic; synchronous PDF/email may be ambiguous or leave residue. Retry by operation ID, not blindly. **Source; Needs runtime verification.** |
| 5 | Frontend older than backend | Current unversioned DTOs have no negotiated compatibility/reload protocol. Additive changes usually survive; required/removed fields and enum changes can break open tabs. No automatic safe update guarantee. **Needs runtime verification.** |
| 6 | Migration fails halfway through deployment | Startup migration/reconciliation may prevent readiness; provider DDL rollback differs. Image rollback cannot unapply schema. Abort, preserve logs/checkpoint, determine forward fix versus isolated restore. **Needs runtime verification.** |
| 7 | Disk reaches 100% | DB/file/log/backup writes can fail; current API has no universal headroom admission policy. Staging helps attachment consistency, not host capacity. Stop costly admission and recover space safely. **Needs runtime verification.** |
| 8 | Memory exhausted | OS/container can kill API/renderer; normal process lifetime assumptions fail. Sidecar is constrained, API/PDF not globally bounded. F03/F10 apply. **Needs runtime verification.** |
| 9 | AI requests time out | Named client has 30s timeout; durable jobs classify/retry within deadlines, synchronous callers return failure/fallback. Manual data should remain usable. Fake timeout tests pass; live provider recovery **Needs runtime verification**. |
| 10 | Selected AI model removed | Configured route fails; privacy-approved fallback may apply. No evidence justifies silently choosing arbitrary replacement. Pin evaluated model and show retriable/unavailable state. **Source; Needs runtime verification.** |
| 11 | Email provider unavailable | Direct auth/reminder sends fail; some account changes already commit. Employer-send attempts can retain failed/uncertain state. Verification resend is necessary; outbox gap F07. **Needs runtime verification.** |
| 12 | Verification never delivered | User remains unverified where verification required; can request resend, subject to rate limits. No delivery guarantee, bounce proof or durable resend queue shown. Provide clear support/recovery path. **Source; Needs runtime verification.** |
| 13 | Stripe sends same webhook five times | Signature verified; each request re-fetches current subscription and reapplies roles. Sequential final state is generally idempotent; redundant work and concurrent writes remain. Handler does not charge five times. **Source; sandbox Needs runtime verification.** |
| 14 | Stripe events out of order | Current re-fetch mitigates sequential stale payloads. Concurrent handlers and multiple subscriptions can still conflict; no serialized customer reconciliation. F02. **Source; Needs runtime verification.** |
| 15 | Double-click Save | CV single-editor save queue serializes local changes; API does not dedup generic creation. **Observed:** identical job POSTs returned two IDs. A UI disabled button alone is insufficient. |
| 16 | Two tabs save same CV | **Observed:** stale and simultaneous PUTs both succeed; newer data can be overwritten. History is not conflict prevention. F01. |
| 17 | Refresh during CV generation | Queued extraction/AI can survive through operation records; synchronous preview/PDF request may abort/continue without recoverable browser result. Unsaved client state may be lost. **Source; Needs runtime verification for generation refresh.** |
| 18 | Session expires while editing | First 401 clears client auth and re-fetches identity; long-form state can unmount. No complete reauthentication-and-draft recovery contract found. Offline/unload warning behavior was tested separately. **Needs runtime verification for actual expiry.** |
| 19 | Logout in another tab | **Observed:** idle private CV editor stays mounted in the other tab. Backend session is revoked and subsequent requests fail. F08. |
| 20 | Delete CV referenced by application | Variant/history removed; job itself is not owned by CV. Already captured submitted package retains rendered content/copied bytes. Live selected-variant UI must handle missing resource. **Source and package tests; exact multi-tab deletion Needs runtime verification.** |
| 21 | Malformed PDF | Parser/upload bounds reject invalid data or return stable extraction failure; profile should not be replaced with invalid output. **Observed existing parser tests pass**; broader parser fuzzing remains. |
| 22 | Malicious uploaded document | Type/expansion/process limits reduce parser risk; no complete malware-scanning/quarantine-verdict service was found. Original download still requires safe headers. **Source; real malware execution not performed. Needs runtime verification of container isolation.** |
| 23 | Job URL redirects to localhost | Import HttpClient disallows auto-redirect; fetch rejects 3xx. Redirect is not followed. **Source and importer tests.** DNS rebinding is a separate remaining risk F11. |
| 24 | Job description contains prompt injection | It is untrusted prompt data; policy and review limit actions, but wording alone does not guarantee model obedience. No provider/network tools should be granted from model output. Synthetic eval fixtures exist; live adversarial model behavior **Needs runtime verification**. |
| 25 | AI returns invalid JSON | Parsing/normalization boundaries can return failure/fallback; durable status records category. Exact all-workload behavior is not proven by one parser. **Source/fake-provider tests; full contract corpus Needs runtime verification.** |
| 26 | AI invents experience | Reviewable suggestions and fact-preservation prompts exist; they cannot guarantee truth. Reject unsupported structured claims and require user review before export/send. Live English/Bokmål semantic evaluation **Needs runtime verification**. |
| 27 | 1000 users request PDFs | No shared admission: many Chromium processes could exhaust API host. Per-public-IP+slug throttling does not bound aggregate authenticated load. **Source F03; intentionally not stress executed. Needs runtime verification after bounding.** |
| 28 | One user creates 100k jobs | Creation lacks total row quota; list pagination does not bound board/analytics/export scans. Expect latency/memory growth, not demonstrated capacity. **Source F10; Needs runtime verification with synthetic data.** |
| 29 | Export extremely large | ZIP file output is streamed on download, but generation loads collections/serialized buffers; jobs export materializes all rows. Can exhaust memory/disk or time out. **Source; Needs runtime verification.** |
| 30 | Quota reached during operation | AI reserves estimates before work; actual final usage may differ. Attachment check-before-write is racy and excludes other file categories. Need reservation/reconciliation and retained inputs. **Source/unit tests; cross-pipeline load Needs runtime verification.** |
| 31 | Subscription expires while logged in | Pro policy reads current DB roles and queued workers recheck; UI may remain stale until re-fetch. Correctness depends on billing sync updating roles. **Source/entitlement tests; delayed-provider event Needs runtime verification.** |
| 32 | Account deletion with pending work | Staged deletion revokes sessions/shares and cancels/removes owner work, purges files and records tombstone. In-flight external effects/billing require coordination F02. **Source/deletion tests; concurrent runtime drill needed.** |
| 33 | Backups silently fail seven days | File-age/catch-up logic exists locally and scheduled production workflow exists, but an independent delivered alert was not proven. Operator could remain unaware. Gate on independently measured backup age. **Needs runtime verification.** |
| 34 | Latest backup corrupt | Shape checks can miss logical corruption. Restore previous independently verified set and establish lost interval; do not overwrite live state before validation. **Needs runtime verification F12.** |
| 35 | TLS certificate expires | Browsers block/warn before application loads; backend health can remain green. Renewal belongs to external proxy operator. Independent certificate/HTTPS alert required. **Needs runtime verification.** |
| 36 | DNS breaks | Users/provider lookups fail before API workflows; cached clients may behave differently. Check authoritative and recursive resolution separately from health. **Needs runtime verification.** |
| 37 | Production secret leaked | Scope depends on JWT/provider/DB/CI/key material. Revoke/rotate and invalidate affected sessions/tokens; preserve evidence and identify exposed data. Key-ring rotation must preserve/re-encrypt required secrets. **Runbook proposal; Needs runtime verification.** |
| 38 | Administrator compromised | Can mutate users/roles and read global activity metadata/content; host admin reaches DB/files/keys. Minimal support access and audited privileged reauth are incomplete F13. **Source; no live compromise simulated.** |
| 39 | Another container compromised | Shared-network access can reach attached services; isolated AI network is helpful, but backend shares several networks and has high privileges. Actual firewall/volume/socket exposure **Needs runtime verification**. |
| 40 | Attacker requests another user's resource ID | **Observed CV GET/PUT/DELETE 404 and admin 403; 753-test suite includes broader owner tests.** This does not prove every future/new route; maintain route inventory/owner regression gate. |
| 41 | Browser cache contains previous user's data | Owner-scoped caches exist, but cross-tab UI residue and absent no-store remain. No cross-user HTTP cache exploit was demonstrated. **Source/partial observed; complete Back/cache account-switch test Needs runtime verification.** |
| 42 | localStorage corrupt | Theme/auth helpers catch many failures, jobColumns JSON is guarded; language storage and some writes are less defensive. Manual modifications never grant backend roles. Full corrupt/blocked-storage route sweep **Needs runtime verification**. |
| 43 | JavaScript error mid-form | Error boundaries can display fallback/reference, but local form state may be lost on remount. Test recoverable draft behavior and safe telemetry; no guaranteed content preservation. **Source; Needs runtime verification.** |
| 44 | 200% zoom | Responsive layouts and tested narrow viewports help; actual browser zoom was not exercised. **Needs runtime verification.** Do not equate 375px viewport with zoom certification. |
| 45 | Keyboard only | Existing browser flows exercise selected buttons/navigation. New builder axe finds a scrolling-region focus issue; full tab/escape/reorder/return-focus journey remains. **Partial observed; manual verification needed.** |
| 46 | Screen reader | No live Narrator/NVDA run. Axe flags do not establish reading order, announcements or accessible generated PDFs. **Needs runtime verification.** |
| 47 | Norwegian text twice English length | Actual EN/NB builder states showed outer overflow 0 at 375/1440; not a doubled-string pseudolocale sweep. English template descriptions remain in NB. **Partial observed; pseudolocale/all-route verification needed.** |
| 48 | Hundreds-character company/job title | Normal browser suite has long-content workspace/CV cases and passed. Arbitrarily large inputs/100k rows are not covered; cap fields and test wrap/reflow. **Partial observed; extreme boundaries need runtime verification.** |
| 49 | Emoji/unusual Unicode | **Observed:** Åse/company emoji and profile content survive synthetic storage/render. Full combining/RTL/confusable corpus and generated glyphs need verification. |
| 50 | Paste formatted HTML | Plain inputs store text; React escaping/renderer encoding are intended barriers. Harness stored an HTML note but did not certify every rendering/email/Markdown sink. No rich-text sanitizer should be replaced by regex. **Source/partial observed; persistent-XSS/paste corpus Needs runtime verification.** |

## Time and compatibility policy

F21's four live HTTP/SQLite/ICS failures are concrete; the fix must distinguish **calendar date**, **absolute instant**, and **local recurring time with timezone**. Current use of DateTime, UTC timestamps and server-local reminders mixes those meanings. UTC is suitable for security events and leases. Date-only job deadlines must not shift with timezone. Recurring wall-clock behavior, if offered, needs an explicit zone and gap/fold policy. Do not add recurring reminders merely to justify a scheduling platform.

The production backup workflow checks for 02:00 Europe/Oslo at two UTC schedules. A spring-forward day has no 02:00 local hour; a fall-back day repeats it. Add a date-based idempotency/catch-up test rather than assuming the guard yields exactly one daily backup. Scheduler DST behavior was source-reviewed, not executed in Gitea.

Adopt expand/contract for externally visible DTO/schema changes: optional fields and readers first, dual-compatible writer period, measured old-client window, then removal. Keep stable IDs/routes and old enum meanings. Introduce explicit API versioning only when real independent clients require it. Build/version metadata already exists; a safe reload prompt may help, but must not discard dirty forms. Old-image compatibility is a precondition to automated rollback, not an assumption.