Files

T

cesnimda c38295d869 docs: add system overview, product research, and roadmap

Phase 1-3 deliverables: full architecture/security/tech-debt map,
2026 market research with feature matrix, and tiered execution roadmap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-07-02 21:24:03 +02:00

19 KiB

Raw Permalink Blame History

Jobbjakt (Job Tracker) — System Overview

Phase 1 deliverable: full-system map produced before any code changes. Last updated: 2026-07-02. Verified against commit eea327e1 plus local working-tree changes.

1. What the product is

Jobbjakt is a self-hosted, multi-user job application tracking platform with heavy AI assistance:

Track job applications end-to-end (status pipeline, follow-ups, deadlines, salary, tags, notes).
Company/recruiter CRM (pipeline stage, contact dates, recruiter details).
Correspondence log per application, including Gmail OAuth import with review workflow.
Attachments per application with purpose metadata and AI-inclusion toggles.
CV platform: upload → OCR/text extraction → structured CV parsing (Ollama-assisted block classification) → per-job tailored CV drafts → templated PDF export via Playwright.
AI drafts: cover letters, recruiter messages, follow-up drafts, job description summaries, translation (LibreTranslate optional).
Rules engine (auto-ghosting, follow-up "needs attention"), reminder emails, daily JSON export, history/event trail, encrypted backup (Windows/DPAPI).
Admin surface: user management, audit log, system readiness page.
Deployed to production at https://jobs.cesnimda.uk via Gitea Actions → SSH → Docker Compose.

2. Architecture overview

flowchart LR
    subgraph Client
        UI[React 19 SPA<br/>MUI 7, react-router 6<br/>CRA/react-scripts]
    end

    subgraph Frontend container
        NGINX[nginx 1.29-alpine<br/>serves build + proxies /api]
    end

    subgraph Backend container
        API[ASP.NET Core net9.0 API<br/>JobTrackerApi host]
        BG[Hosted services:<br/>Rules, FollowUpReminder,<br/>DailyExport, JobEnrichment,<br/>SummarizerProbe, CvProcessing]
        DB[(SQLite default<br/>or MariaDB/MySQL)]
        FS[/Data root:<br/>Attachments, CvArtifacts,<br/>exports, DP keys/]
    end

    subgraph AI stack
        AISVC[FastAPI ai-service :8001<br/>distilbart summarizer,<br/>OCR pytesseract/PyMuPDF,<br/>docx/pdf extraction]
        OLLAMA[Ollama :11434<br/>qwen2.5:7b<br/>CV classification + rewrite]
    end

    EXT1[Google OAuth / Gmail API]
    EXT2[Job sites: Finn, NAV,<br/>LinkedIn, Jobbnorge]
    EXT3[SMTP - Gmail app password]
    EXT4[LibreTranslate optional]

    UI --> NGINX --> API
    API --> DB
    API --> FS
    API --> AISVC --> OLLAMA
    API --> EXT1
    API --> EXT2
    API --> EXT3
    API --> EXT4
    BG --> DB

Solution layout (unusual — read this first)

Project	Role
`JobTrackerApi/`	Web host only: `Program.cs`, appsettings, migrations, Dockerfile. Its csproj excludes `Controllers/` and `Services/` from its own compilation.
`JobTrackerBackend/`	"Transitional shared-backend" library that compiles, via `<Compile Include>` links, the files physically located in `../Data`, `../Models`, `../JobTrackerApi/Controllers`, `../JobTrackerApi/Services`. Exists so tests can reference controllers/services without the web-entry project.
`JobTrackerApi.Tests/`	xUnit test project (~20 test classes incl. authorization/hostile-fixture tests).
`Models/`, `Data/` (repo root)	The real EF models and `JobTrackerContext`, compiled into JobTrackerBackend.
`Controller/` (repo root)	Legacy stub controllers (~1 KB each) — dead code, not referenced by any csproj.
`job-tracker-ui/`	React SPA.
`tools/summarizer/`	FastAPI AI service (own Dockerfile, pytest tests).
`deploy/`, `.gitea/workflows/`	Prod deploy script + CI/CD pipeline.
`docs/`	Session handoffs, security assessments (M013–M015), UAT notes.

3. Technology stack

Backend: ASP.NET Core net9.0, EF Core 9 (SQLite default; Pomelo MySQL/MariaDB switchable via Database:Provider), ASP.NET Identity Core (users/roles), JWT bearer auth (local + Google policy scheme), built-in RateLimiter, DataProtection (file-system keys), Playwright (CV PDF export).

Frontend: React 19, TypeScript 4.9, MUI 7 (+ x-data-grid, x-date-pickers, lab), axios, react-router-dom 6, @tanstack/react-table, CRA react-scripts 5 (build needs --max-old-space-size=4096), i18n EN + NB (custom provider), Jest/RTL tests.

AI: FastAPI + transformers (sshleifer/distilbart-cnn-12-6) for summaries; pytesseract/PyMuPDF/pypdf/python-docx for extraction/OCR; Ollama (qwen2.5:7b) for CV block classification and rewrite paths; TTL cache.

Infra: Docker Compose (4 services: backend, frontend/nginx, ai-service, ollama w/ GPU), Gitea Actions CI (build + backend tests + selected frontend tests + frontend build) → SSH deploy → deploy/deploy.sh on the prod host, external jobtracker_shared network.

4. Authentication & authorization

Smart policy scheme: inspects the bearer token issuer — Google-issued ID tokens (accounts.google.com) route to the google JWT handler (validated against Auth:GoogleClientId); everything else routes to local JWT (symmetric key Auth:JwtKey, issuer/audience validated, 2-min clock skew).
Cookie session support: local handler also reads the session cookie (AuthSessionOptions.SessionCookieName); CSRF double-submit middleware enforces cookie+header match for all mutating requests when a session cookie is present (login/register/reset/csrf endpoints exempt).
Auth:Require=true sets a fallback authorize-all policy (prod compose sets it). Dev without a JWT key generates an ephemeral key + warning; fails closed if auth required but no key.
Local tokens must carry a subject claim (LocalAuthIdentity), enforced in OnTokenValidated — hardened after finding M013-2.
Multi-tenancy: every tenant entity carries OwnerUserId; JobTrackerContext applies global query filters CurrentUserId != null && OwnerUserId == CurrentUserId (deny-on-null). Correspondence/JobEvents/CV entities filter through their parent's owner.
Roles via ASP.NET Identity: admin-only controllers (UsersController, AdminAuditController, AdminSystemController).
Password policy: min 8, digit + lowercase required. Password reset via emailed token (SMTP required). Registration disabled by default.
Rate limiting: auth-login (10/5 min/IP) and auth-email (5/15 min/IP) fixed-window policies.

5. Database schema (EF Core, 8 migrations)

erDiagram
    ApplicationUser ||--o{ Company : owns
    ApplicationUser ||--o{ JobApplication : owns
    ApplicationUser ||--o| UserRuleSettings : has
    ApplicationUser ||--o{ GmailConnection : has
    ApplicationUser ||--o{ CvUploadArtifact : owns
    ApplicationUser ||--o{ CvExtractionRun : owns
    Company ||--o{ JobApplication : "has jobs"
    JobApplication ||--o{ Correspondence : messages
    JobApplication ||--o{ Attachment : attachments
    JobApplication ||--o{ JobEvent : events
    JobApplication ||--o| TailoredCvDraft : "1:1 draft"
    CvUploadArtifact ||--o{ CvExtractionRun : "source of"
    ApplicationUser ||--o{ GmailReviewDecision : decides

Key notes:

ApplicationUser (IdentityUser) also stores profile CV text, structured CV JSON (ProfileCvStructureJson), avatar data-URL, Google link info, current CV artifact/run pointers.
JobApplication: status string (default "Applied"), soft delete (IsDeleted/DeletedAt), tags as JSON string, imported description + translation, persisted ShortSummary, tailored CV text, reminder bookkeeping. Cascade deletes to messages/attachments/events/draft.
RuleSettings (global, seeded Id=1) + per-user UserRuleSettings.
SystemEmailSettings: DB-stored SMTP override (resolved by EmailSettingsResolver).
Indexes: OwnerUserId on Company/JobApplication/GmailConnection; composite (OwnerUserId, UploadedAtUtc), (OwnerUserId, StartedAtUtc), unique (OwnerUserId, JobApplicationId) on draft, unique (OwnerUserId, GmailAddress).
SQLite file lives at DataRoot/jobtracker.db (WAL mode); migrations applied automatically at startup (StartupInitializationExtensions, 62 KB — also seeds admin, creates Identity tables where dotnet ef unavailable, ignores PendingModelChangesWarning).

6. API surface (all under `/api`, ~15 controllers)

Controller	Highlights
`JobApplicationsController` (151 KB!)	CRUD, paging/filtering/sorting, board, reminders, stats, history, unified timeline, status/follow-up PATCH, soft delete/restore, plus AI surface: application package material, follow-up drafts, cover-letter/recruiter drafts ("Maria" drafts), workflow signals.
`ProfileCvController` (117 KB)	CV upload artifacts, extraction runs, structure parsing, rebuild/improve, tailored CV generation via Ollama rewrite, template rendering + Playwright PDF preview/export, benchmark corpus harness.
`GmailController` (60 KB)	OAuth connect/callback, sync, message review queue, import decisions, job matching.
`AuthController` (22 KB)	login/register/me/config, Google exchange, password reset request/reset, session cookie + CSRF endpoints.
`CompaniesController`	CRUD, idempotent create by name, recruiter/pipeline fields.
`CorrespondenceController`	per-job messages CRUD.
`AttachmentsController`	multipart upload to disk, download, rename, delete, purpose/AI-inclusion metadata.
`RulesController`	global + per-user rule settings, clamped.
`ExportController`	JSON/CSV export.
`BackupController`	DPAPI-encrypted backup (Windows only).
`JobImportController`	URL preview via plugin parsers (SSRF-hardened).
`UsersController`, `AdminAuditController`, `AdminSystemController`	admin: user/role management, audit trail, system readiness (DB/Gmail/AI).
`ClientErrorsController`	frontend error intake → logs.

No OpenAPI/Swagger is wired up; the README is the de-facto API doc (already drifting).

7. Background services (6 hosted services)

Service	Function
`RulesHostedService` → `RulesEngine`	periodic auto-transitions (e.g., → Ghosted) from rule settings
`FollowUpReminderHostedService`	reminder emails for due/upcoming follow-ups (dedup via `LastReminderEmailSentAt`)
`DailyExportHostedService`	daily JSON export at configured local hour
`JobEnrichmentHostedService`	backfills summaries/enrichment for jobs
`SummarizerProbeHostedService`	probes AI service readiness
`CvProcessingHostedService` + `CvProcessingQueue`	in-memory queue for CV extraction/processing jobs

All state is in-process (IMemoryCache, in-memory queue) — single-instance assumption; no distributed locks; queue contents lost on restart.

8. AI pipeline (data flow)

Job import: URL → plugin parse (Finn/NAV/LinkedIn/Jobbnorge or universal JSON-LD parser) → optional LibreTranslate → language detect + skill tagging → preview → user accepts → stored on JobApplication.
Summaries: API → SummarizerService (31 KB) → FastAPI /summarize (distilbart, TTL-cached, GPU-if-available) → persisted ShortSummary.
CV ingest: upload (PDF/DOCX/image ≤ 8 MB) → FastAPI extract/OCR → block classification (Ollama-assisted, CvAiClassifier/CvAiNormalizer) → ProfileCvStructureJson on user.
Tailoring: job description + structured CV sections → Ollama rewrite path (recent commits: clamped lengths, hardened diagnostics) → TailoredCvDraft (JSON blocks) → CvTemplateRenderer (25 KB, template carousel) → Playwright → PDF.
Drafts: cover letter / recruiter message / follow-up drafts generated per job with attachment-aware context selection.

Degradation: if AI service or Ollama is down, core tracking still works (probe service + "AI is not a deploy gate" in CI).

9. Email

SmtpEmailSender with EmailSettingsResolver: config from env/appsettings or DB-stored SystemEmailSettings (admin-editable).
Uses Gmail SMTP + app password in prod. Flows: password reset, follow-up reminders. App:PublicBaseUrl builds links.

10. Configuration & secrets

.env (git-ignored) → docker-compose env → ASP.NET config. .env.example documents the shape. Real secrets currently present in local .env (JWT key, admin password, SMTP app password, Google client secret).
appsettings.Development.json contains only CHANGE_ME_* placeholders (good).
Key knobs: Database:Provider, ConnectionStrings:JobTracker, Data:Root, Cors:Origins, Ai:BaseUrl, Auth:*, Email:*, Exports:*, Translation:*, App:PublicBaseUrl, HttpsRedirection:* (TLS terminated at reverse proxy; HSTS/redirect off in-container).
ProductionConfigTests.cs exists to guard prod config shape.

11. Build, CI/CD, deployment

CI (.gitea/workflows/ci-deploy.yml): on PR + push-to-main → build backend (Release), run backend tests, npm ci, run an explicit whitelist of 10 frontend test files (not the whole suite), build frontend.
Deploy (push to main only): SSH to prod host → git reset --hard <sha> in /opt/job-tracker/app → deploy/deploy.sh (docker compose build/up with retry/cache-prune fallbacks) → verify containers; AI service health is non-blocking.
Frontend Dockerfile: node build stage → nginx 1.29-alpine (working-tree bump from 1.27 pending commit); nginx proxies /api to backend.
No staging environment; deploys go straight to prod after CI.

12. Testing strategy

Backend: xUnit integration-style tests via TestHostFactory; notable coverage: authorization (JobApplicationsAuthorizationTests, OwnershipGuardTests, hostile fixture DB project), auth/system, Gmail, CV corpus harness, summarizer, SQLite migration helper, production config.
Frontend: ~20 Jest/RTL test files (workspace flows, Gmail review, login, admin, attachments, drafts, trust-loop e2e-ish component tests). CI runs only the whitelisted subset.
AI service: pytest (tools/summarizer/tests/test_app.py).
No true end-to-end browser tests; no load/perf tests.

13. Logging & error handling

Console/debug logging; custom middleware logs every request (method, path, status, ms, traceId, sub claim). Unhandled exceptions logged then rethrown (500).
Client errors POSTed to /api/client-errors and logged server-side; React ErrorBoundary + route error page in UI.
No structured sink (Seq/OTLP), no log rotation policy in-app (container stdout), no correlation to frontend errorIds beyond log text, no ProblemDetails standardization.

14. Security posture (current)

Strong points (much already hardened via M013–M015 adversarial assessments in docs/security-assessments/):

SSRF on job import fixed & retested (DNS resolution check, private/loopback/link-local rejection, redirects disabled).
Subjectless-JWT / owner-filter bypass fixed & retested (fail-closed identity, deny-on-null query filters).
Cross-user job history leak fixed (81196374); authorization replay findings recorded (M015).
CSRF double-submit for cookie sessions; CORS allowlist; rate-limited login/email endpoints; ephemeral JWT key refused when auth required; Identity password hashing (PBKDF2); DataProtection keys persisted outside repo runtime path.

Open questions / watch areas (to verify in Phase 6):

AllowCredentials() combined with configurable Cors:Origins="*" wildcard mode (SetIsOriginAllowed(true) + credentials) — dangerous if ever enabled.
Attachment upload: file-type/size limits, path handling, content-type on download need re-audit.
Avatar stored as data-URL on user record (size/XSS considerations).
Gmail OAuth token storage encryption at rest; scopes; audit of GmailController (60 KB).
Global rate limiting only on 2 auth policies — AI/expensive endpoints unthrottled.
Backup endpoint Windows-only DPAPI — silently unavailable on Linux prod.
Dependency freshness (axios, react-scripts 5/CRA is deprecated upstream; transformers/torch pinning).
Secrets present in local .env (expected, git-ignored) — confirm no history leaks.

15. Technical debt report

God controllers: JobApplicationsController (151 KB), ProfileCvController (117 KB), GmailController (60 KB), StartupInitializationExtensions (62 KB). Massive single files mixing HTTP, business logic, AI prompt construction, and persistence. Highest-leverage refactor target — but high risk, needs test cover first.
Transitional project layout: JobTrackerBackend compiles files it doesn't own via glob includes; root Models//Data/ folders; dead root Controller/ folder; JobTrackerBackend/bin+obj artifacts and JobTrackerApi/jobtracker.db + bin_build/, CvArtifacts/, exports/, keys/ polluting the repo/working tree. .gitignore needs review.
CI runs a hand-maintained subset of frontend tests — new test files silently not run (already bit them once; profile-page.test.tsx had to be added manually).
CRA/react-scripts 5 is EOL-ish, slow builds (needs 4 GB heap), TS 4.9. Vite migration is the obvious path (medium effort).
Naming drift: Summarizer* vs AiService*; "Jobbjakt" vs "Job Tracker" branding split; EN/NB translation consistency flagged in handoff doc.
No OpenAPI; README endpoint list already drifts from code (e.g., Gmail/profile/admin endpoints missing there).
In-memory queue/cache single-instance coupling undocumented.
Root-level clutter: temp_job.json, temp_post_job.py, todo jobtracker.txt, test/, tmp/, vendor/, .venv/.
DaysSince compares DateTime.UtcNow with .Days truncation — timezone/UX edge cases; status is a free string, no canonical pipeline enum (README itself lists this as a wanted improvement).
Windows-only backup path.

16. Areas of concern

Single point of data: SQLite in a Docker volume; backups are manual/Windows-only; no automated off-host backup.
Deploy risk: git reset --hard + straight-to-prod with no staging and non-exhaustive CI test coverage.
AI coupling: prompt logic buried in controllers makes model/provider changes and testing hard.
Restart data loss: queued CV processing jobs are lost on restart (in-memory queue).
Uncommitted working tree: 3 modified files (Dockerfile nginx bump, useViewResource stale-closure fix, handoff doc) + untracked scripts/start-ollama-cv.ps1 and a stray JobTrackerApi/CvArtifacts/ data folder.

17. Opportunities for improvement (input to Phase 2/3)

Product (initial hypotheses, to be validated by market research):

Canonical pipeline model + customizable Kanban stages (already on README wish list).
Interview scheduling/prep hub (calendar integration, prep notes, question banks).
Salary/offer comparison and analytics dashboards (funnel conversion, response rates, time-in-stage).
Browser extension / bookmarklet for one-click job capture (plugins already exist server-side).
Saved searches/views, full-text search, date-range and tag filters.
Notifications beyond email (web push, digest).
Contact-level recruiter CRM (people, not just companies).
Mobile-friendly PWA pass.

Engineering:

Swagger/OpenAPI + generated TS client; ProblemDetails everywhere.
Split god controllers into feature services; move AI prompting behind interfaces.
Run full frontend test suite in CI (npm test -- --watchAll=false without whitelist) once flaky tests are addressed; add dotnet format/eslint gates.
Vite migration; dependency refresh.
Durable job queue (DB-backed) for CV processing; automated DB backup job.
Repo hygiene: delete dead Controller/, ignore build artifacts, remove committed DB files.

19 KiB Raw Permalink Blame History Unescape Escape