jobtrackingapp/docs/audits/2026-09-07/resilience.md

Resilience, partial failure and the 50 failure answers

Source baseline, labels and limitations: README. Findings: F01–F21. “Needs runtime verification” means the scenario was not executed, even when source provides a strong prediction.

Dependency and operation failure matrix

Timeouts marked current come from source; operational alert thresholds are proposals. Normal CRUD should remain useful during optional-provider failures.

Dependency / operation	Failure and user impact	Current timeout / retry	Recovery and target UX	Alert / evidence
SQLite / MariaDB	Writes, auth/session checks and reads fail; active AI worker can stop host F20	Provider defaults; observed SQLite command timeout 30s; no central transient-DB policy	Preserve forms, show service unavailable/reference; bounded worker recovery; never retry ambiguous writes without key	Readiness failures + sustained DB errors; real outage needs runtime verification
AI sidecar	AI/extraction fails while manual work should continue	API named client 30s; classified queue failures retry within deadline	Persistent operation status and safe retry; synchronous edits remain unsent until reviewed	Task/provider failure rate and pending age; fake-provider tests passed
Ollama / external AI	Missing model, DNS/HTTP failure, malformed result	Per-task/provider routing; durable default 3 attempts / 15min deadline, worker timeout 300s	Explicit failure or policy-approved fallback; never silently send private content externally	Model/task/version/category counters; live model evaluation not performed
Parser	Malformed/hostile/oversized input, child timeout	5MB file; 25s child, 512MiB configured child memory, one parser at a time; container 2GiB/2CPU	Stable error, review/import can retry, previous profile retained	Parser crash/timeout/rejection counts; 37 parser tests passed; container enforcement not exercised
Chromium PDF	Missing executable/crash/timeout, partial output	60s process timeout; no shared admission	Retain CV; explain download failed; retry only after cleanup/lookup; F03 bounds required	Active process/RSS/timeouts/files; no heavy live stress test
SMTP	Verification/reminder send fails or is suppressed	Configured timeout (dev 15s), synchronous send in Task.Run; direct auth/reminder attempts	Outbox/attempt state; do not mark suppressed as delivered; resend verification within token validity	Queue age/send failures/bounces; actual delivery not tested
Gmail / Graph / IMAP	Mail sync/calendar/send unavailable, revoked credentials	Provider-specific paths; send attempts preserve uncertain status	Manual tracking usable; reconnect action; do not auto-resend uncertain employer mail	Last sync success, auth/provider error, unresolved send attempt
Stripe	Checkout/portal unavailable; delayed webhooks, role mismatch	SDK calls; failed webhook returns non-2xx	Preserve current local state pending authoritative reconciliation; status banner; deletion waits for cancellation	Inbox age/divergence/cancel failures; sandbox required
Job site / NAV / translation	URL unavailable, oversized response, DNS fail	Job import client; redirects rejected, 4MB stream cap	Allow manual paste/edit; no empty replacement of a valid job	Timeouts/rejections/cache hit; real sites not called
File storage	Permission, missing bytes, full disk	I/O exceptions; staged attachment recovery; no universal quota	Roll back or preserve recoverable stage; do not create valid-looking broken artifact	Disk/inode pressure, orphan/missing counters, failed purge
Proxy / DNS / TLS	Site unreachable despite live backend	External infrastructure unknown	Correct route/cert/DNS, verify from outside; static maintenance response if necessary	External probe independent of host; not inspected live
Browser network	Offline/delayed/partial save/download	Axios has no shared timeout; no blanket retry; many local async handlers	Keep dirty state, bounded deadlines, retry by operation identity; distinguish ambiguous outcome	UI safe errors and request references; offline CV probe observed failure state
Logging / telemetry	Sink unavailable, disk growth, blocked script	Console logging; client errors best-effort	Drop/sample bounded optional telemetry; never block saves on a collector	Local dropped-event counter, disk alerts; failure injection pending
Backups / off-site copy	Missing/partial/inconsistent set	Script checks artifact shape; defaults 30 daily days/12 monthly sets	Preserve previous verified set; restore drill; do not prune based on unverified success	Backup failure and age from independent monitor; restore pending
Redis / broker / standalone document service	Not present	Not applicable	Do not add failure domains without a measured requirement	Reassess if introduced

Transaction and background-work review

Workflow	Existing transaction / state design	Partial failure that matters	Minimum change
Create CV + version 1	Two saves inside a relational transaction	Concurrent later edits still lack preconditions	Preserve creation transaction; add F01
Career Profile save	Relational children/profile/history transaction; controller later updates user projection	Canonical data committed but legacy projection/user update fails	One commit if same DB context can own all writes, or explicit repairable projection; do not tell user total failure after success
Job create/status	Job/opportunity/history saved together in normal EF unit	Client loses response and resubmits; no creation intent key	Return/retrieve result by idempotency key
Attachment upload/delete	Stage/quarantine files; DB commit; promote/purge; startup reconciliation	Commit uncertainty or crash leaves metadata/file transitions pending	Preserve existing protocol, add aggregate quota reservation and missing-file alert
Submitted package capture	Renders/reads before DB transaction; copies files during transaction; unique package version	Concurrent max-version selection collides; source can change during capture; cleanup race	Allocate package revision safely, use unique attempt storage then publish; freeze source revisions; test file ownership of cleanup
AI operation	Durable owner/idempotency/lease/deadline, result reference and attempts	Provider succeeds but result commit fails; worker crashes	Result persistence idempotency and lease recovery; F20 loop recovery; classify poison failures as terminal
Employer send	Durable attempt and payload hash; uncertain state preserved	Provider accepted send but response/DB completion lost	Reconcile by provider message ID when possible; explicit user decision before retry
Reminder / verification email	Direct send; reminder timestamps committed after loop	Duplicate/suppressed/unrecorded send F07	Durable intent/attempt ledger; SMTP cannot guarantee exactly-once
Stripe webhook	Re-fetch then user update then role mutation	Partial role update; overlapping subscription events	Per-customer transactional reconciliation + durable inbox
Account deletion	Persist stages; revoke sessions/share, quarantine, delete rows, purge, tombstone	In-flight files/results after inventory; Stripe still active; crash replay	F02 cancellation stage; coordinate active work and final inventory; exercise restore tombstones
Export	Account ZIP on disk; table JSON buffers; delete-on-close stream	Crash leaves ZIP; changes during export can span different logical moments	Consistency semantics/manifest; stale file cleanup; bound memory or background export

A transactional outbox is justified for required transactional mail and local billing work because an external side effect cannot join the database commit. It is not a reason to wrap every CRUD handler in infrastructure. Keep APIs/providers outside long DB transactions. Retries need operation identity, a payload hash and a known outcome; GET that triggers AI or writes should be examined before treating all GETs as harmless retry candidates.

Resilience test matrix / Break Jobjakt

Scenario	Test method	Expected behavior	Current result	Fix / next evidence
Two tabs save same CV	Read once, sequential stale PUTs; concurrent PUT pair	Conflict; newer state preserved	Observed fail: 200/200, stale state replaced newer edit	F01; extend to job/profile/restore
Duplicate create / Save double-click	Two identical job POSTs	Same intended action returns one resource	Observed: two 201s, distinct IDs	Introduce explicit intent-key contract; do not block legitimate repeat applications
Other user's CV IDs	Direct GET/PUT/DELETE as account B	403/404, no mutation	Observed pass: all 404	Keep permanent owner tests
Ordinary user admin call	GET admin audit	403	Observed pass: 403	Extend all admin verbs
Same-IP account export	A exports twice, B exports once	Independent account limits	Observed fail: B gets 429	F06 middleware integration test
Formula/Unicode/HTML input	Create synthetic job =1+1, Åse/company emoji, HTML note; export	Store safely, safe spreadsheet cells	Observed: storage works; formula preserved unsafe in CSV	F05; full stored-XSS surface still runtime needed
Offset/DST calendar	Four input instants through DB to ICS	Preserve UTC instant; repeated autumn hour distinct	Observed fail: all four return wrong UTC values, HTTP 200	F21; both database/server-zone matrices
Offline CV save	Browser context offline during name edit	Failure shown; no false saved state	Observed pass: failure message present	Preserve visible state
Refresh after failed save	Accept unload warning, reload	Recoverable draft if recovery supported	Observed gap: draft not recovered	F09; warning exists, recovery does not
Logout in second tab	Shared-cookie logout + auth-marker event	Clear private UI; deny future requests	Observed gap: other editor remains mounted	F08; distinguish visual residue from backend denial
EN/NB, light/dark, 375/1440	Static build + Chromium + axe + screenshots	No serious violations/overflow	Observed: outer overflow 0; contrast errors all 8 states; desktop scroll-focus errors	F14; manual assistive tech still needed
Malformed PDF/DOCX, ZIP bomb, parser hang	Existing pytest mocked/isolated child suite	Bounded rejection, no secret inheritance	Observed: 37 tests pass; not a live malware corpus	Run container boundary/fuzz corpus in disposable environment
DB down 30s / 1h	Disconnect disposable MariaDB or fault-inject repository	Bounded errors, live host, retry recovery	Needs runtime verification; F20 predicts host risk with AI worker	Add loop failure test then staging drill
Container restart mid-write/generation	Kill only a disposable deployment after known checkpoints	Durable work resumes; ambiguous writes recover	Needs runtime verification; unit lease/recovery tests pass	Exercise actual process/files/DB recovery, not mocks alone
API latency/timeout/partial response	Local reverse proxy delays/truncates selected responses	Dirty state kept; no blind POST retry	Needs runtime verification	Test stale search replies, file upload/download, proxy deadline
Browser storage corrupt/denied	Invalid JSON/unknown schema + throw on storage APIs	Safe defaults and retained form	Unit owner/theme tests pass; full route sweep Needs runtime verification	I18n reads/writes need denied-storage coverage
Screen reader/keyboard/200–400% zoom	Narrator/NVDA + actual browser zoom + text-spacing	Reach all controls, focus/announcements, usable reflow	Needs runtime verification beyond existing keyboard tests/axe	Do not equate viewport width or axe with screen-reader proof
PDF 1000 simultaneous / 100k jobs	Synthetic capped load/fake renderer first	Global admission, bounded queries/memory	Needs runtime verification, source gaps F03/F10	Never execute uncapped exhaustion on shared host
Disk/inodes/RSS exhaustion	Quota-mounted temp volumes/cgroup-limited staging	Fail before corruption; alert/recover	Needs runtime verification	Preserve backups; no filling the user's actual disk
Billing/mailer/OAuth outage	Fake gateways then sandbox provider callbacks	Correct pending/failed/retry state	Fake-gateway suites pass; live integrations Need runtime verification	F02/F07 and provider-specific contract tests
Backup corrupt/missing / migration halfway	Restore isolated backup, interrupt test migration	Fail deploy or forward-recover to a known checkpoint	Needs runtime verification beyond current migration/unit tests	F12 and signed drill record

The harness is observational: meetsExpectation=false records an unmet desired control, not a flaky test that should be retried until green. No PDF/CPU flood, actual malicious executable, payment, employer email or production outage was attempted. Logs distinguish deliberate offline/401/blocked-font errors from normal workflow errors. Full packet-loss simulation, clean-machine setup, production network/console/log cleanliness and exhaustive UI state testing are outstanding, not implied by this matrix.

What happens if…?

#	Scenario	Current answer and evidence boundary
1	Database offline for 30 seconds	Reads/writes and DB-backed session checks fail. /health is process-only; /ready checks DB. Enabled AI-worker claim/persist errors can stop the host F20. Forms need recoverable failure UX. Source; Needs runtime verification.
2	Database offline for an hour	No offline DB mode. Requests continue failing, leases age, worker/host restart behavior may recur; recovery requires DB availability and operation reconciliation. Alert early, preserve accepted work. Needs runtime verification.
3	Docker host restarts	Named DB/file/key/tombstone volumes should survive; unless-stopped services restart. In-memory caches/limits reset; durable leases recover. External MariaDB/proxy start order still matters. Needs runtime verification.
4	API dies halfway through request	Uncommitted DB transactions roll back; completed commits can be invisible to client. Staged attachments have recovery logic; synchronous PDF/email may be ambiguous or leave residue. Retry by operation ID, not blindly. Source; Needs runtime verification.
5	Frontend older than backend	Current unversioned DTOs have no negotiated compatibility/reload protocol. Additive changes usually survive; required/removed fields and enum changes can break open tabs. No automatic safe update guarantee. Needs runtime verification.
6	Migration fails halfway through deployment	Startup migration/reconciliation may prevent readiness; provider DDL rollback differs. Image rollback cannot unapply schema. Abort, preserve logs/checkpoint, determine forward fix versus isolated restore. Needs runtime verification.
7	Disk reaches 100%	DB/file/log/backup writes can fail; current API has no universal headroom admission policy. Staging helps attachment consistency, not host capacity. Stop costly admission and recover space safely. Needs runtime verification.
8	Memory exhausted	OS/container can kill API/renderer; normal process lifetime assumptions fail. Sidecar is constrained, API/PDF not globally bounded. F03/F10 apply. Needs runtime verification.
9	AI requests time out	Named client has 30s timeout; durable jobs classify/retry within deadlines, synchronous callers return failure/fallback. Manual data should remain usable. Fake timeout tests pass; live provider recovery Needs runtime verification.
10	Selected AI model removed	Configured route fails; privacy-approved fallback may apply. No evidence justifies silently choosing arbitrary replacement. Pin evaluated model and show retriable/unavailable state. Source; Needs runtime verification.
11	Email provider unavailable	Direct auth/reminder sends fail; some account changes already commit. Employer-send attempts can retain failed/uncertain state. Verification resend is necessary; outbox gap F07. Needs runtime verification.
12	Verification never delivered	User remains unverified where verification required; can request resend, subject to rate limits. No delivery guarantee, bounce proof or durable resend queue shown. Provide clear support/recovery path. Source; Needs runtime verification.
13	Stripe sends same webhook five times	Signature verified; each request re-fetches current subscription and reapplies roles. Sequential final state is generally idempotent; redundant work and concurrent writes remain. Handler does not charge five times. Source; sandbox Needs runtime verification.
14	Stripe events out of order	Current re-fetch mitigates sequential stale payloads. Concurrent handlers and multiple subscriptions can still conflict; no serialized customer reconciliation. F02. Source; Needs runtime verification.
15	Double-click Save	CV single-editor save queue serializes local changes; API does not dedup generic creation. Observed: identical job POSTs returned two IDs. A UI disabled button alone is insufficient.
16	Two tabs save same CV	Observed: stale and simultaneous PUTs both succeed; newer data can be overwritten. History is not conflict prevention. F01.
17	Refresh during CV generation	Queued extraction/AI can survive through operation records; synchronous preview/PDF request may abort/continue without recoverable browser result. Unsaved client state may be lost. Source; Needs runtime verification for generation refresh.
18	Session expires while editing	First 401 clears client auth and re-fetches identity; long-form state can unmount. No complete reauthentication-and-draft recovery contract found. Offline/unload warning behavior was tested separately. Needs runtime verification for actual expiry.
19	Logout in another tab	Observed: idle private CV editor stays mounted in the other tab. Backend session is revoked and subsequent requests fail. F08.
20	Delete CV referenced by application	Variant/history removed; job itself is not owned by CV. Already captured submitted package retains rendered content/copied bytes. Live selected-variant UI must handle missing resource. Source and package tests; exact multi-tab deletion Needs runtime verification.
21	Malformed PDF	Parser/upload bounds reject invalid data or return stable extraction failure; profile should not be replaced with invalid output. Observed existing parser tests pass; broader parser fuzzing remains.
22	Malicious uploaded document	Type/expansion/process limits reduce parser risk; no complete malware-scanning/quarantine-verdict service was found. Original download still requires safe headers. Source; real malware execution not performed. Needs runtime verification of container isolation.
23	Job URL redirects to localhost	Import HttpClient disallows auto-redirect; fetch rejects 3xx. Redirect is not followed. Source and importer tests. DNS rebinding is a separate remaining risk F11.
24	Job description contains prompt injection	It is untrusted prompt data; policy and review limit actions, but wording alone does not guarantee model obedience. No provider/network tools should be granted from model output. Synthetic eval fixtures exist; live adversarial model behavior Needs runtime verification.
25	AI returns invalid JSON	Parsing/normalization boundaries can return failure/fallback; durable status records category. Exact all-workload behavior is not proven by one parser. Source/fake-provider tests; full contract corpus Needs runtime verification.
26	AI invents experience	Reviewable suggestions and fact-preservation prompts exist; they cannot guarantee truth. Reject unsupported structured claims and require user review before export/send. Live English/Bokmål semantic evaluation Needs runtime verification.
27	1000 users request PDFs	No shared admission: many Chromium processes could exhaust API host. Per-public-IP+slug throttling does not bound aggregate authenticated load. Source F03; intentionally not stress executed. Needs runtime verification after bounding.
28	One user creates 100k jobs	Creation lacks total row quota; list pagination does not bound board/analytics/export scans. Expect latency/memory growth, not demonstrated capacity. Source F10; Needs runtime verification with synthetic data.
29	Export extremely large	ZIP file output is streamed on download, but generation loads collections/serialized buffers; jobs export materializes all rows. Can exhaust memory/disk or time out. Source; Needs runtime verification.
30	Quota reached during operation	AI reserves estimates before work; actual final usage may differ. Attachment check-before-write is racy and excludes other file categories. Need reservation/reconciliation and retained inputs. Source/unit tests; cross-pipeline load Needs runtime verification.
31	Subscription expires while logged in	Pro policy reads current DB roles and queued workers recheck; UI may remain stale until re-fetch. Correctness depends on billing sync updating roles. Source/entitlement tests; delayed-provider event Needs runtime verification.
32	Account deletion with pending work	Staged deletion revokes sessions/shares and cancels/removes owner work, purges files and records tombstone. In-flight external effects/billing require coordination F02. Source/deletion tests; concurrent runtime drill needed.
33	Backups silently fail seven days	File-age/catch-up logic exists locally and scheduled production workflow exists, but an independent delivered alert was not proven. Operator could remain unaware. Gate on independently measured backup age. Needs runtime verification.
34	Latest backup corrupt	Shape checks can miss logical corruption. Restore previous independently verified set and establish lost interval; do not overwrite live state before validation. Needs runtime verification F12.
35	TLS certificate expires	Browsers block/warn before application loads; backend health can remain green. Renewal belongs to external proxy operator. Independent certificate/HTTPS alert required. Needs runtime verification.
36	DNS breaks	Users/provider lookups fail before API workflows; cached clients may behave differently. Check authoritative and recursive resolution separately from health. Needs runtime verification.
37	Production secret leaked	Scope depends on JWT/provider/DB/CI/key material. Revoke/rotate and invalidate affected sessions/tokens; preserve evidence and identify exposed data. Key-ring rotation must preserve/re-encrypt required secrets. Runbook proposal; Needs runtime verification.
38	Administrator compromised	Can mutate users/roles and read global activity metadata/content; host admin reaches DB/files/keys. Minimal support access and audited privileged reauth are incomplete F13. Source; no live compromise simulated.
39	Another container compromised	Shared-network access can reach attached services; isolated AI network is helpful, but backend shares several networks and has high privileges. Actual firewall/volume/socket exposure Needs runtime verification.
40	Attacker requests another user's resource ID	Observed CV GET/PUT/DELETE 404 and admin 403; 753-test suite includes broader owner tests. This does not prove every future/new route; maintain route inventory/owner regression gate.
41	Browser cache contains previous user's data	Owner-scoped caches exist, but cross-tab UI residue and absent no-store remain. No cross-user HTTP cache exploit was demonstrated. Source/partial observed; complete Back/cache account-switch test Needs runtime verification.
42	localStorage corrupt	Theme/auth helpers catch many failures, jobColumns JSON is guarded; language storage and some writes are less defensive. Manual modifications never grant backend roles. Full corrupt/blocked-storage route sweep Needs runtime verification.
43	JavaScript error mid-form	Error boundaries can display fallback/reference, but local form state may be lost on remount. Test recoverable draft behavior and safe telemetry; no guaranteed content preservation. Source; Needs runtime verification.
44	200% zoom	Responsive layouts and tested narrow viewports help; actual browser zoom was not exercised. Needs runtime verification. Do not equate 375px viewport with zoom certification.
45	Keyboard only	Existing browser flows exercise selected buttons/navigation. New builder axe finds a scrolling-region focus issue; full tab/escape/reorder/return-focus journey remains. Partial observed; manual verification needed.
46	Screen reader	No live Narrator/NVDA run. Axe flags do not establish reading order, announcements or accessible generated PDFs. Needs runtime verification.
47	Norwegian text twice English length	Actual EN/NB builder states showed outer overflow 0 at 375/1440; not a doubled-string pseudolocale sweep. English template descriptions remain in NB. Partial observed; pseudolocale/all-route verification needed.
48	Hundreds-character company/job title	Normal browser suite has long-content workspace/CV cases and passed. Arbitrarily large inputs/100k rows are not covered; cap fields and test wrap/reflow. Partial observed; extreme boundaries need runtime verification.
49	Emoji/unusual Unicode	Observed: Åse/company emoji and profile content survive synthetic storage/render. Full combining/RTL/confusable corpus and generated glyphs need verification.
50	Paste formatted HTML	Plain inputs store text; React escaping/renderer encoding are intended barriers. Harness stored an HTML note but did not certify every rendering/email/Markdown sink. No rich-text sanitizer should be replaced by regex. Source/partial observed; persistent-XSS/paste corpus Needs runtime verification.

Time and compatibility policy

F21's four live HTTP/SQLite/ICS failures are concrete; the fix must distinguish calendar date, absolute instant, and local recurring time with timezone. Current use of DateTime, UTC timestamps and server-local reminders mixes those meanings. UTC is suitable for security events and leases. Date-only job deadlines must not shift with timezone. Recurring wall-clock behavior, if offered, needs an explicit zone and gap/fold policy. Do not add recurring reminders merely to justify a scheduling platform.

The production backup workflow checks for 02:00 Europe/Oslo at two UTC schedules. A spring-forward day has no 02:00 local hour; a fall-back day repeats it. Add a date-based idempotency/catch-up test rather than assuming the guard yields exactly one daily backup. Scheduler DST behavior was source-reviewed, not executed in Gitea.

Adopt expand/contract for externally visible DTO/schema changes: optional fields and readers first, dual-compatible writer period, measured old-client window, then removal. Keep stable IDs/routes and old enum meanings. Introduce explicit API versioning only when real independent clients require it. Build/version metadata already exists; a safe reload prompt may help, but must not discard dirty forms. Old-image compatibility is a precondition to automated rollback, not an assumption.