jobtrackingapp/docs/phase-2-1-cv-extraction-review.md

# Phase 2.1 — CV extraction: current implementation, weaknesses, proposal

> 2026-07-20. Investigation deliverable for Phase 2.1. Grounded in the code and in a real benchmark
> CV (Connor Babbington, Systems Developer). No code changed yet — this is the "explain → identify →
> propose" step before any implementation.

## 1. Current implementation (verified in code)

Upload flow (`ProfileCvController`, `CvProcessingQueue`, `tools/summarizer/app.py`):

```
Upload file  → CvUploadArtifact stored on disk, CvExtractionRun queued
   ↓ (background) ProcessQueuedRunAsync
Extract text (PDF/DOCX/image OCR)
   ↓
AI /cv/normalize   → LLM rewrites messy text into markdown-sectioned CV text
   ↓
split into blocks, AI /cv/classify-block per block → {section, fields, confidence}
   ↓
assemble StructuredCvProfile  (+ per-field metadata: confidence, method, sourceBlockId, reviewState)
   ↓
ApplyQueuedRunResultAsync  → SaveVersionAsync + user.ProfileCvStructureJson = <new>   ← REPLACE
   ↓
run.Status = "applied"   (no user confirmation)
```

The structured model (`JobTrackerApi/Models/StructuredCvProfile.cs`) already has `Contact, Summary, Jobs,
Education, Certifications, Projects, Skills, Languages` and per-field confidence metadata. Confidence
is **captured and displayed** (the review chips) but used for nothing else.

## 2. Weaknesses (code-verified, then benchmark-verified)

### Architectural (the big two)

- **A1 — Replace, not merge.** `ApplyQueuedRunResultAsync` overwrites the entire profile every run
  (`SaveVersionAsync(structuredCv)` + `user.ProfileCvStructureJson = structuredJson`). Re-importing a
  CV, or a slightly worse OCR pass, **discards** whatever the user curated. This is the exact failure
  the vision names: the profile oscillates instead of getting richer. There is no dedup, no
  field-level merge, no "keep the better value".
- **A2 — No review-before-apply.** The run auto-applies (`Status = "applied"`). The user never sees
  "we found 4 experiences, 15 skills, ⚠ 1 language" and never approves. The confidence metadata that
  would drive such a screen is already computed and then ignored.

### Extraction quality (verified against the benchmark CV)

- **Q1 — Projects and Certifications are dropped.** The `/cv/normalize` heading list
  (`app.py:566-573`) and the `/cv/classify-block` section enum (`app.py:622`) include only
  Contact / Summary / Work Experience / Education / Skills / Languages / Interests — **no Projects, no
  Certifications** — even though the data model and `StructuredCvProfileJson` fully support them. The
  benchmark CV's entire **Projects** section (JobTrack, InboxIntel, infra lab) is lost.
- **Q2 — Languages in prose are missed.** The benchmark states languages only inside the summary
  ("Native English speaker; Norwegian at B1"). There is no `# Languages` block for the classifier to
  pick up, and normalize won't synthesise one from prose, so **English/Norwegian + levels are lost**
  as structured languages.
- **Q3 — Grouped skills leak their category label.** "Development: C#, .NET, Python…" — the classifier
  turns skill lines into items but nothing strips the "Development:" / "DevOps & Infrastructure:" /
  "Practices:" prefixes, so a skill like "Development: C#" or a junk "Development" item can appear.
- **Q4 — Glued date/title runs.** Text extraction yields "2015–2023System Developer — Warwickshire
  County Council, UK" with no space between the date range and the title. The classifier expects clean
  `start`/`end`; a two-hop normalize→classify can mis-split or drop the date.
- **Q5 — Nested "Earlier roles (part-time)" list.** Three secondary jobs (Royal Vapes, The Hodcarrier,
  Nuffield Health) sit as sub-bullets under a heading, not as standard entries. They are likely
  mis-classified as bullets of the parent job or dropped.
- **Q6 — Two-hop LLM loses whole-CV context.** normalize (rewrites text, can hallucinate/omit) then
  per-block classify (no cross-block view) means duplicate or mis-sectioned entries and no dedup.
- **Q7 — Encoding.** PDF text extraction returns mojibake for `ø`, en-dashes and apostrophes
  (`T�nsberg`, `years�`, `2015�2023`) depending on the extractor. Downstream this corrupts company
  names, locations and dates. (The DB round-trip for `æøå` is already fixed; this is the *extraction*
  side.)

### Confidence

- **C1 — Confidence is display-only.** It is computed per field and shown as chips, but never used to
  (a) gate what auto-applies, (b) flag low-confidence items for review, or (c) decide merge-vs-keep.

## 3. Proposed architecture

Keep the strengths (text extraction, per-field confidence metadata, the structured model) and add the
two missing layers plus targeted extraction fixes.

```
Extract → AI structured extraction (improved) → Validate → Normalize
   → DIFF against current profile  (new)
   → REVIEW screen: "We found …"  (new, confidence-aware)
   → MERGE on accept  (new: field-level, dedup, confidence-gated)   ← never a blind replace
```

- **Merge engine (A1).** On accept, merge per entity: match experiences/education/projects by a stable
  key (company+title+dates / institution+qualification / project name), update a field only when the
  incoming confidence is high **and** differs, add genuinely new items, and never delete a curated
  item the import didn't mention. Skills/languages dedup case-insensitively; language levels update
  only on higher confidence.
- **Review screen (A2).** Reuse the diff: show counts and per-category adds/updates with ✓ (high) and
  ⚠ (low-confidence) markers, and Accept all / Review individually / Discard. Nothing writes until
  accept. (This is also Phase 2's "Import CV review" screen from the workspace refactor — one build
  serves both.)
- **Extraction fixes (Q1–Q7).** Add Projects + Certifications to both prompts and the block assembler;
  synthesise Languages from summary prose (or a dedicated language pass); strip skill-group prefixes;
  harden date/title splitting; handle the "earlier/part-time roles" pattern; fix extraction-time
  encoding.
- **Confidence gating (C1).** Drive the review markers and the merge rules from the existing
  per-field confidence — no new scoring needed to start, just *use* it.

## 4. Recommended increment order (small, verified, deployable each)

1. **2.1-a — Merge instead of replace + review gate** (highest value, the vision's core). Backend
   diff+merge engine with tests; frontend review screen. Nothing auto-overwrites again.
2. **2.1-b — Extraction coverage: Projects, Certifications, Languages-from-prose** (Q1, Q2). Prompt +
   assembler + parser, benchmarked on the CV. **— DELIVERED 2026-07-20.**
3. **2.1-c — Extraction cleanup: skill-group prefixes, glued dates, part-time roles, encoding**
   (Q3–Q5, Q7).
4. **2.1-d — Confidence-driven review markers and merge gating** (C1), once the review screen exists.

Each ships independently and leaves production green.

## 5. Benchmark as regression fixture

Save the benchmark CV's expected structured output as a test fixture: 5 experiences (2 primary + 3
part-time) or a documented decision on the part-time roles, 3 projects, grouped skills flattened,
English (Native) + Norwegian (B1) languages, education entry, contact with Norwegian location intact.
Extraction changes are measured against it — without overfitting (the rules must generalise).

## 2.1-b delivered — extraction coverage (2026-07-20)

**What changed.** The gap was upstream only: the C# assembler (`StructuredCvProfileJson.FromSections`
+ `BuildStructuredCvFromNormalizedMarkdown`) already maps `Projects`, `Certifications` and `Languages`
headings — the AI `normalize` prompt just never emitted them, so they were dropped.

- `tools/summarizer/app.py` `/cv/normalize`: added `# Projects` and `# Certifications` headings with
  shapes; added a **languages-from-prose** rule (extract "native English", "Norwegian B1" from the
  summary even without a Languages section; ignore programming languages); added **skill-group prefix
  stripping** ("Development:", "DevOps & Infrastructure:", "Practices:" are dropped, only the skills
  remain).
- `/cv/classify-block`: added `Projects` and `Certifications` to the section enum + rules (fallback
  path).
- `ProfileCvController.LooksLikeNormalizedMarkdownCv`: recognises `# Projects` / `# Certifications` so
  a CV whose structured content is mostly those sections still takes the markdown path.

**Verification.** 4 new backend tests (`CvExtractionCoverageTests`) lock the C# mapping of
Projects/Certifications/Languages; 1 new ai-service test (`test_classify_block_supports_projects_section`).
426 backend tests and 17 ai-service tests pass; `app.py` compiles. The LLM behaviour itself
(prompt → headings) could not be run here (no Ollama), but the C# side that consumes the headings is
proven, and the prompt change is additive/contract-safe.

**Deployment note.** These prompt changes live in the **ai-service container**, which `deploy.sh`
does **not** rebuild by default. Deploy with `DEPLOY_BUILD_AI_SERVICE=true ./deploy/deploy.sh` (or
rebuild `ai-service` manually) or the extraction change won't take effect. No database or backend
schema change.

**Not done here (moved to 2.1-c):** deterministic C# safety-nets for skill-prefix stripping and
glued-date splitting, and the "earlier/part-time roles" pattern. 2.1-b relies on the prompt for those;
2.1-c hardens them deterministically.

## 6. Open product decisions (need a call before building)

- **Merge matching keys** — how aggressively to treat two experiences as "the same" (company+title vs
  fuzzy). Conservative (fewer merges, some dupes) vs aggressive (cleaner, risk of wrong merges).
- **Part-time/earlier roles** — separate experience entries, or a sub-list on the primary role?
- **Auto-apply threshold** — does anything ever apply without review (e.g. an empty profile's first
  import), or is review always required?