jobtrackingapp/docs/_archive/SYSTEM_OVERVIEW.md

# Jobbjakt (Job Tracker) — System Overview

> Phase 1 deliverable: full-system map produced before any code changes.
> Last updated: 2026-07-02. Verified against commit `eea327e1` plus local working-tree changes.

---

## 1. What the product is

Jobbjakt is a self-hosted, multi-user job application tracking platform with heavy AI assistance:

- Track job applications end-to-end (status pipeline, follow-ups, deadlines, salary, tags, notes).
- Company/recruiter CRM (pipeline stage, contact dates, recruiter details).
- Correspondence log per application, including **Gmail OAuth import with review workflow**.
- Attachments per application with purpose metadata and AI-inclusion toggles.
- **CV platform**: upload → OCR/text extraction → structured CV parsing (Ollama-assisted block classification) → per-job tailored CV drafts → templated PDF export via Playwright.
- AI drafts: cover letters, recruiter messages, follow-up drafts, job description summaries, translation (LibreTranslate optional).
- Rules engine (auto-ghosting, follow-up "needs attention"), reminder emails, daily JSON export, history/event trail, encrypted backup (Windows/DPAPI).
- Admin surface: user management, audit log, system readiness page.
- Deployed to production at `https://jobs.cesnimda.uk` via Gitea Actions → SSH → Docker Compose.

---

## 2. Architecture overview

```mermaid
flowchart LR
    subgraph Client
        UI[React 19 SPA<br/>MUI 7, react-router 6<br/>CRA/react-scripts]
    end

    subgraph Frontend container
        NGINX[nginx 1.29-alpine<br/>serves build + proxies /api]
    end

    subgraph Backend container
        API[ASP.NET Core net9.0 API<br/>JobTrackerApi host]
        BG[Hosted services:<br/>Rules, FollowUpReminder,<br/>DailyExport, JobEnrichment,<br/>SummarizerProbe, CvProcessing]
        DB[(SQLite default<br/>or MariaDB/MySQL)]
        FS[/Data root:<br/>Attachments, CvArtifacts,<br/>exports, DP keys/]
    end

    subgraph AI stack
        AISVC[FastAPI ai-service :8001<br/>distilbart summarizer,<br/>OCR pytesseract/PyMuPDF,<br/>docx/pdf extraction]
        OLLAMA[Ollama :11434<br/>qwen2.5:7b<br/>CV classification + rewrite]
    end

    EXT1[Google OAuth / Gmail API]
    EXT2[Job sites: Finn, NAV,<br/>LinkedIn, Jobbnorge]
    EXT3[SMTP - Gmail app password]
    EXT4[LibreTranslate optional]

    UI --> NGINX --> API
    API --> DB
    API --> FS
    API --> AISVC --> OLLAMA
    API --> EXT1
    API --> EXT2
    API --> EXT3
    API --> EXT4
    BG --> DB
```

### Solution layout (unusual — read this first)

| Project | Role |
|---|---|
| `JobTrackerApi/` | Web **host** only: `Program.cs`, appsettings, migrations, Dockerfile. Its csproj **excludes** `Controllers/**` and `Services/**` from its own compilation. |
| `JobTrackerBackend/` | "Transitional shared-backend" **library** that compiles, via `<Compile Include>` links, the files physically located in `../Data`, `../Models`, `../JobTrackerApi/Controllers`, `../JobTrackerApi/Services`. Exists so tests can reference controllers/services without the web-entry project. |
| `JobTrackerApi.Tests/` | xUnit test project (~20 test classes incl. authorization/hostile-fixture tests). |
| `Models/`, `Data/` (repo root) | The *real* EF models and `JobTrackerContext`, compiled into JobTrackerBackend. |
| `Controller/` (repo root) | **Legacy stub controllers (~1 KB each) — dead code**, not referenced by any csproj. |
| `job-tracker-ui/` | React SPA. |
| `tools/summarizer/` | FastAPI AI service (own Dockerfile, pytest tests). |
| `deploy/`, `.gitea/workflows/` | Prod deploy script + CI/CD pipeline. |
| `docs/` | Session handoffs, security assessments (M013–M015), UAT notes. |

---

## 3. Technology stack

**Backend**: ASP.NET Core net9.0, EF Core 9 (SQLite default; Pomelo MySQL/MariaDB switchable via `Database:Provider`), ASP.NET Identity Core (users/roles), JWT bearer auth (local + Google policy scheme), built-in RateLimiter, DataProtection (file-system keys), Playwright (CV PDF export).

**Frontend**: React 19, TypeScript 4.9, MUI 7 (+ x-data-grid, x-date-pickers, lab), axios, react-router-dom 6, @tanstack/react-table, CRA `react-scripts` 5 (build needs `--max-old-space-size=4096`), i18n EN + NB (custom provider), Jest/RTL tests.

**AI**: FastAPI + transformers (`sshleifer/distilbart-cnn-12-6`) for summaries; pytesseract/PyMuPDF/pypdf/python-docx for extraction/OCR; Ollama (`qwen2.5:7b`) for CV block classification and rewrite paths; TTL cache.

**Infra**: Docker Compose (4 services: backend, frontend/nginx, ai-service, ollama w/ GPU), Gitea Actions CI (build + backend tests + selected frontend tests + frontend build) → SSH deploy → `deploy/deploy.sh` on the prod host, external `jobtracker_shared` network.

---

## 4. Authentication & authorization

- **Smart policy scheme**: inspects the bearer token issuer — Google-issued ID tokens (`accounts.google.com`) route to the `google` JWT handler (validated against `Auth:GoogleClientId`); everything else routes to `local` JWT (symmetric key `Auth:JwtKey`, issuer/audience validated, 2-min clock skew).
- **Cookie session support**: local handler also reads the session cookie (`AuthSessionOptions.SessionCookieName`); **CSRF double-submit** middleware enforces cookie+header match for all mutating requests when a session cookie is present (login/register/reset/csrf endpoints exempt).
- `Auth:Require=true` sets a fallback authorize-all policy (prod compose sets it). Dev without a JWT key generates an ephemeral key + warning; **fails closed** if auth required but no key.
- Local tokens **must** carry a subject claim (`LocalAuthIdentity`), enforced in `OnTokenValidated` — hardened after finding M013-2.
- **Multi-tenancy**: every tenant entity carries `OwnerUserId`; `JobTrackerContext` applies global query filters `CurrentUserId != null && OwnerUserId == CurrentUserId` (deny-on-null). Correspondence/JobEvents/CV entities filter through their parent's owner.
- Roles via ASP.NET Identity: admin-only controllers (`UsersController`, `AdminAuditController`, `AdminSystemController`).
- Password policy: min 8, digit + lowercase required. Password reset via emailed token (SMTP required). Registration disabled by default.
- Rate limiting: `auth-login` (10/5 min/IP) and `auth-email` (5/15 min/IP) fixed-window policies.

---

## 5. Database schema (EF Core, 8 migrations)

```mermaid
erDiagram
    ApplicationUser ||--o{ Company : owns
    ApplicationUser ||--o{ JobApplication : owns
    ApplicationUser ||--o| UserRuleSettings : has
    ApplicationUser ||--o{ GmailConnection : has
    ApplicationUser ||--o{ CvUploadArtifact : owns
    ApplicationUser ||--o{ CvExtractionRun : owns
    Company ||--o{ JobApplication : "has jobs"
    JobApplication ||--o{ Correspondence : messages
    JobApplication ||--o{ Attachment : attachments
    JobApplication ||--o{ JobEvent : events
    JobApplication ||--o| TailoredCvDraft : "1:1 draft"
    CvUploadArtifact ||--o{ CvExtractionRun : "source of"
    ApplicationUser ||--o{ GmailReviewDecision : decides
```

Key notes:

- `ApplicationUser` (IdentityUser) also stores profile CV text, **structured CV JSON** (`ProfileCvStructureJson`), avatar data-URL, Google link info, current CV artifact/run pointers.
- `JobApplication`: status string (default "Applied"), soft delete (`IsDeleted`/`DeletedAt`), tags as JSON string, imported description + translation, persisted `ShortSummary`, tailored CV text, reminder bookkeeping. Cascade deletes to messages/attachments/events/draft.
- `RuleSettings` (global, seeded Id=1) + per-user `UserRuleSettings`.
- `SystemEmailSettings`: DB-stored SMTP override (resolved by `EmailSettingsResolver`).
- Indexes: `OwnerUserId` on Company/JobApplication/GmailConnection; composite `(OwnerUserId, UploadedAtUtc)`, `(OwnerUserId, StartedAtUtc)`, unique `(OwnerUserId, JobApplicationId)` on draft, unique `(OwnerUserId, GmailAddress)`.
- SQLite file lives at `DataRoot/jobtracker.db` (WAL mode); migrations applied automatically at startup (`StartupInitializationExtensions`, 62 KB — also seeds admin, creates Identity tables where `dotnet ef` unavailable, ignores `PendingModelChangesWarning`).

---

## 6. API surface (all under `/api`, ~15 controllers)

| Controller | Highlights |
|---|---|
| `JobApplicationsController` (**151 KB!**) | CRUD, paging/filtering/sorting, board, reminders, stats, history, unified timeline, status/follow-up PATCH, soft delete/restore, **plus** AI surface: application package material, follow-up drafts, cover-letter/recruiter drafts ("Maria" drafts), workflow signals. |
| `ProfileCvController` (**117 KB**) | CV upload artifacts, extraction runs, structure parsing, rebuild/improve, tailored CV generation via Ollama rewrite, template rendering + Playwright PDF preview/export, benchmark corpus harness. |
| `GmailController` (**60 KB**) | OAuth connect/callback, sync, message review queue, import decisions, job matching. |
| `AuthController` (22 KB) | login/register/me/config, Google exchange, password reset request/reset, session cookie + CSRF endpoints. |
| `CompaniesController` | CRUD, idempotent create by name, recruiter/pipeline fields. |
| `CorrespondenceController` | per-job messages CRUD. |
| `AttachmentsController` | multipart upload to disk, download, rename, delete, purpose/AI-inclusion metadata. |
| `RulesController` | global + per-user rule settings, clamped. |
| `ExportController` | JSON/CSV export. |
| `BackupController` | DPAPI-encrypted backup (Windows only). |
| `JobImportController` | URL preview via plugin parsers (SSRF-hardened). |
| `UsersController`, `AdminAuditController`, `AdminSystemController` | admin: user/role management, audit trail, system readiness (DB/Gmail/AI). |
| `ClientErrorsController` | frontend error intake → logs. |

No OpenAPI/Swagger is wired up; the README is the de-facto API doc (already drifting).

---

## 7. Background services (6 hosted services)

| Service | Function |
|---|---|
| `RulesHostedService` → `RulesEngine` | periodic auto-transitions (e.g., → Ghosted) from rule settings |
| `FollowUpReminderHostedService` | reminder emails for due/upcoming follow-ups (dedup via `LastReminderEmailSentAt`) |
| `DailyExportHostedService` | daily JSON export at configured local hour |
| `JobEnrichmentHostedService` | backfills summaries/enrichment for jobs |
| `SummarizerProbeHostedService` | probes AI service readiness |
| `CvProcessingHostedService` + `CvProcessingQueue` | in-memory queue for CV extraction/processing jobs |

All state is in-process (`IMemoryCache`, in-memory queue) — single-instance assumption; no distributed locks; queue contents lost on restart.

---

## 8. AI pipeline (data flow)

1. **Job import**: URL → plugin parse (Finn/NAV/LinkedIn/Jobbnorge or universal JSON-LD parser) → optional LibreTranslate → language detect + skill tagging → preview → user accepts → stored on `JobApplication`.
2. **Summaries**: API → `SummarizerService` (31 KB) → FastAPI `/summarize` (distilbart, TTL-cached, GPU-if-available) → persisted `ShortSummary`.
3. **CV ingest**: upload (PDF/DOCX/image ≤ 8 MB) → FastAPI extract/OCR → block classification (Ollama-assisted, `CvAiClassifier`/`CvAiNormalizer`) → `ProfileCvStructureJson` on user.
4. **Tailoring**: job description + structured CV sections → Ollama rewrite path (recent commits: clamped lengths, hardened diagnostics) → `TailoredCvDraft` (JSON blocks) → `CvTemplateRenderer` (25 KB, template carousel) → Playwright → PDF.
5. **Drafts**: cover letter / recruiter message / follow-up drafts generated per job with attachment-aware context selection.

Degradation: if AI service or Ollama is down, core tracking still works (probe service + "AI is not a deploy gate" in CI).

---

## 9. Email

- `SmtpEmailSender` with `EmailSettingsResolver`: config from env/appsettings **or** DB-stored `SystemEmailSettings` (admin-editable).
- Uses Gmail SMTP + app password in prod. Flows: password reset, follow-up reminders. `App:PublicBaseUrl` builds links.

---

## 10. Configuration & secrets

- `.env` (git-ignored) → docker-compose env → ASP.NET config. `.env.example` documents the shape. Real secrets currently present in local `.env` (JWT key, admin password, SMTP app password, Google client secret).
- `appsettings.Development.json` contains only `CHANGE_ME_*` placeholders (good).
- Key knobs: `Database:Provider`, `ConnectionStrings:JobTracker`, `Data:Root`, `Cors:Origins`, `Ai:BaseUrl`, `Auth:*`, `Email:*`, `Exports:*`, `Translation:*`, `App:PublicBaseUrl`, `HttpsRedirection:*` (TLS terminated at reverse proxy; HSTS/redirect off in-container).
- `ProductionConfigTests.cs` exists to guard prod config shape.

---

## 11. Build, CI/CD, deployment

- **CI** (`.gitea/workflows/ci-deploy.yml`): on PR + push-to-main → build backend (Release), run backend tests, `npm ci`, run an **explicit whitelist of 10 frontend test files** (not the whole suite), build frontend.
- **Deploy** (push to main only): SSH to prod host → `git reset --hard <sha>` in `/opt/job-tracker/app` → `deploy/deploy.sh` (docker compose build/up with retry/cache-prune fallbacks) → verify containers; AI service health is non-blocking.
- Frontend Dockerfile: node build stage → nginx 1.29-alpine (working-tree bump from 1.27 pending commit); nginx proxies `/api` to backend.
- No staging environment; deploys go straight to prod after CI.

---

## 12. Testing strategy

- **Backend**: xUnit integration-style tests via `TestHostFactory`; notable coverage: authorization (`JobApplicationsAuthorizationTests`, `OwnershipGuardTests`, hostile fixture DB project), auth/system, Gmail, CV corpus harness, summarizer, SQLite migration helper, production config.
- **Frontend**: ~20 Jest/RTL test files (workspace flows, Gmail review, login, admin, attachments, drafts, trust-loop e2e-ish component tests). CI runs only the whitelisted subset.
- **AI service**: pytest (`tools/summarizer/tests/test_app.py`).
- No true end-to-end browser tests; no load/perf tests.

---

## 13. Logging & error handling

- Console/debug logging; custom middleware logs every request (method, path, status, ms, traceId, sub claim). Unhandled exceptions logged then rethrown (500).
- Client errors POSTed to `/api/client-errors` and logged server-side; React `ErrorBoundary` + route error page in UI.
- No structured sink (Seq/OTLP), no log rotation policy in-app (container stdout), no correlation to frontend errorIds beyond log text, no ProblemDetails standardization.

---

## 14. Security posture (current)

Strong points (much already hardened via M013–M015 adversarial assessments in `docs/security-assessments/`):

- SSRF on job import **fixed & retested** (DNS resolution check, private/loopback/link-local rejection, redirects disabled).
- Subjectless-JWT / owner-filter bypass **fixed & retested** (fail-closed identity, deny-on-null query filters).
- Cross-user job history leak fixed (`81196374`); authorization replay findings recorded (M015).
- CSRF double-submit for cookie sessions; CORS allowlist; rate-limited login/email endpoints; ephemeral JWT key refused when auth required; Identity password hashing (PBKDF2); DataProtection keys persisted outside repo runtime path.

Open questions / watch areas (to verify in Phase 6):

- `AllowCredentials()` combined with configurable `Cors:Origins="*"` wildcard mode (SetIsOriginAllowed(true) + credentials) — dangerous if ever enabled.
- Attachment upload: file-type/size limits, path handling, content-type on download need re-audit.
- Avatar stored as data-URL on user record (size/XSS considerations).
- Gmail OAuth token storage encryption at rest; scopes; audit of `GmailController` (60 KB).
- Global rate limiting only on 2 auth policies — AI/expensive endpoints unthrottled.
- Backup endpoint Windows-only DPAPI — silently unavailable on Linux prod.
- Dependency freshness (axios, react-scripts 5/CRA is deprecated upstream; transformers/torch pinning).
- Secrets present in local `.env` (expected, git-ignored) — confirm no history leaks.

---

## 15. Technical debt report

1. **God controllers**: `JobApplicationsController` (151 KB), `ProfileCvController` (117 KB), `GmailController` (60 KB), `StartupInitializationExtensions` (62 KB). Massive single files mixing HTTP, business logic, AI prompt construction, and persistence. Highest-leverage refactor target — but high risk, needs test cover first.
2. **Transitional project layout**: `JobTrackerBackend` compiles files it doesn't own via glob includes; root `Models/`/`Data/` folders; **dead** root `Controller/` folder; `JobTrackerBackend/bin`+`obj` artifacts and `JobTrackerApi/jobtracker.db` + `bin_build/`, `CvArtifacts/`, `exports/`, `keys/` polluting the repo/working tree. `.gitignore` needs review.
3. **CI runs a hand-maintained subset** of frontend tests — new test files silently not run (already bit them once; `profile-page.test.tsx` had to be added manually).
4. **CRA/react-scripts 5** is EOL-ish, slow builds (needs 4 GB heap), TS 4.9. Vite migration is the obvious path (medium effort).
5. **Naming drift**: `Summarizer*` vs `AiService*`; "Jobbjakt" vs "Job Tracker" branding split; EN/NB translation consistency flagged in handoff doc.
6. No OpenAPI; README endpoint list already drifts from code (e.g., Gmail/profile/admin endpoints missing there).
7. In-memory queue/cache single-instance coupling undocumented.
8. Root-level clutter: `temp_job.json`, `temp_post_job.py`, `todo jobtracker.txt`, `test/`, `tmp/`, `vendor/`, `.venv/`.
9. `DaysSince` compares `DateTime.UtcNow` with `.Days` truncation — timezone/UX edge cases; status is a free string, no canonical pipeline enum (README itself lists this as a wanted improvement).
10. Windows-only backup path.

---

## 16. Areas of concern

- **Single point of data**: SQLite in a Docker volume; backups are manual/Windows-only; no automated off-host backup.
- **Deploy risk**: `git reset --hard` + straight-to-prod with no staging and non-exhaustive CI test coverage.
- **AI coupling**: prompt logic buried in controllers makes model/provider changes and testing hard.
- **Restart data loss**: queued CV processing jobs are lost on restart (in-memory queue).
- **Uncommitted working tree**: 3 modified files (Dockerfile nginx bump, `useViewResource` stale-closure fix, handoff doc) + untracked `scripts/start-ollama-cv.ps1` and a stray `JobTrackerApi/CvArtifacts/` data folder.

---

## 17. Opportunities for improvement (input to Phase 2/3)

Product (initial hypotheses, to be validated by market research):

- Canonical pipeline model + customizable Kanban stages (already on README wish list).
- Interview scheduling/prep hub (calendar integration, prep notes, question banks).
- Salary/offer comparison and analytics dashboards (funnel conversion, response rates, time-in-stage).
- Browser extension / bookmarklet for one-click job capture (plugins already exist server-side).
- Saved searches/views, full-text search, date-range and tag filters.
- Notifications beyond email (web push, digest).
- Contact-level recruiter CRM (people, not just companies).
- Mobile-friendly PWA pass.

Engineering:

- Swagger/OpenAPI + generated TS client; ProblemDetails everywhere.
- Split god controllers into feature services; move AI prompting behind interfaces.
- Run full frontend test suite in CI (`npm test -- --watchAll=false` without whitelist) once flaky tests are addressed; add `dotnet format`/eslint gates.
- Vite migration; dependency refresh.
- Durable job queue (DB-backed) for CV processing; automated DB backup job.
- Repo hygiene: delete dead `Controller/`, ignore build artifacts, remove committed DB files.