jobtrackingapp/docs/phase-2-1-cv-extraction-review.md

Phase 2.1 — CV extraction: current implementation, weaknesses, proposal

2026-07-20. Investigation deliverable for Phase 2.1. Grounded in the code and in a real benchmark CV (Connor Babbington, Systems Developer). No code changed yet — this is the "explain → identify → propose" step before any implementation.

1. Current implementation (verified in code)

Upload flow (ProfileCvController, CvProcessingQueue, tools/summarizer/app.py):

Upload file  → CvUploadArtifact stored on disk, CvExtractionRun queued
   ↓ (background) ProcessQueuedRunAsync
Extract text (PDF/DOCX/image OCR)
   ↓
AI /cv/normalize   → LLM rewrites messy text into markdown-sectioned CV text
   ↓
split into blocks, AI /cv/classify-block per block → {section, fields, confidence}
   ↓
assemble StructuredCvProfile  (+ per-field metadata: confidence, method, sourceBlockId, reviewState)
   ↓
ApplyQueuedRunResultAsync  → SaveVersionAsync + user.ProfileCvStructureJson = <new>   ← REPLACE
   ↓
run.Status = "applied"   (no user confirmation)

The structured model (JobTrackerApi/Models/StructuredCvProfile.cs) already has Contact, Summary, Jobs, Education, Certifications, Projects, Skills, Languages and per-field confidence metadata. Confidence is captured and displayed (the review chips) but used for nothing else.

2. Weaknesses (code-verified, then benchmark-verified)

Architectural (the big two)

• A1 — Replace, not merge. ApplyQueuedRunResultAsync overwrites the entire profile every run (SaveVersionAsync(structuredCv) + user.ProfileCvStructureJson = structuredJson). Re-importing a CV, or a slightly worse OCR pass, discards whatever the user curated. This is the exact failure the vision names: the profile oscillates instead of getting richer. There is no dedup, no field-level merge, no "keep the better value".
• A2 — No review-before-apply. The run auto-applies (Status = "applied"). The user never sees "we found 4 experiences, 15 skills, ⚠ 1 language" and never approves. The confidence metadata that would drive such a screen is already computed and then ignored.

Extraction quality (verified against the benchmark CV)

• Q1 — Projects and Certifications are dropped. The /cv/normalize heading list (app.py:566-573) and the /cv/classify-block section enum (app.py:622) include only Contact / Summary / Work Experience / Education / Skills / Languages / Interests — no Projects, no Certifications — even though the data model and StructuredCvProfileJson fully support them. The benchmark CV's entire Projects section (JobTrack, InboxIntel, infra lab) is lost.
• Q2 — Languages in prose are missed. The benchmark states languages only inside the summary ("Native English speaker; Norwegian at B1"). There is no # Languages block for the classifier to pick up, and normalize won't synthesise one from prose, so English/Norwegian + levels are lost as structured languages.
• Q3 — Grouped skills leak their category label. "Development: C#, .NET, Python…" — the classifier turns skill lines into items but nothing strips the "Development:" / "DevOps & Infrastructure:" / "Practices:" prefixes, so a skill like "Development: C#" or a junk "Development" item can appear.
• Q4 — Glued date/title runs. Text extraction yields "2015–2023System Developer — Warwickshire County Council, UK" with no space between the date range and the title. The classifier expects clean start/end; a two-hop normalize→classify can mis-split or drop the date.
• Q5 — Nested "Earlier roles (part-time)" list. Three secondary jobs (Royal Vapes, The Hodcarrier, Nuffield Health) sit as sub-bullets under a heading, not as standard entries. They are likely mis-classified as bullets of the parent job or dropped.
• Q6 — Two-hop LLM loses whole-CV context. normalize (rewrites text, can hallucinate/omit) then per-block classify (no cross-block view) means duplicate or mis-sectioned entries and no dedup.
• Q7 — Encoding. PDF text extraction returns mojibake for ø, en-dashes and apostrophes (T�nsberg, years�, 2015�2023) depending on the extractor. Downstream this corrupts company names, locations and dates. (The DB round-trip for æøå is already fixed; this is the extraction side.)

Confidence

• C1 — Confidence is display-only. It is computed per field and shown as chips, but never used to (a) gate what auto-applies, (b) flag low-confidence items for review, or (c) decide merge-vs-keep.

3. Proposed architecture

Keep the strengths (text extraction, per-field confidence metadata, the structured model) and add the two missing layers plus targeted extraction fixes.

Extract → AI structured extraction (improved) → Validate → Normalize
   → DIFF against current profile  (new)
   → REVIEW screen: "We found …"  (new, confidence-aware)
   → MERGE on accept  (new: field-level, dedup, confidence-gated)   ← never a blind replace

• Merge engine (A1). On accept, merge per entity: match experiences/education/projects by a stable key (company+title+dates / institution+qualification / project name), update a field only when the incoming confidence is high and differs, add genuinely new items, and never delete a curated item the import didn't mention. Skills/languages dedup case-insensitively; language levels update only on higher confidence.
• Review screen (A2). Reuse the diff: show counts and per-category adds/updates with ✓ (high) and ⚠ (low-confidence) markers, and Accept all / Review individually / Discard. Nothing writes until accept. (This is also Phase 2's "Import CV review" screen from the workspace refactor — one build serves both.)
• Extraction fixes (Q1–Q7). Add Projects + Certifications to both prompts and the block assembler; synthesise Languages from summary prose (or a dedicated language pass); strip skill-group prefixes; harden date/title splitting; handle the "earlier/part-time roles" pattern; fix extraction-time encoding.
• Confidence gating (C1). Drive the review markers and the merge rules from the existing per-field confidence — no new scoring needed to start, just use it.

4. Recommended increment order (small, verified, deployable each)

1. 2.1-a — Merge instead of replace + review gate (highest value, the vision's core). Backend diff+merge engine with tests; frontend review screen. Nothing auto-overwrites again.
2. 2.1-b — Extraction coverage: Projects, Certifications, Languages-from-prose (Q1, Q2). Prompt + assembler + parser, benchmarked on the CV. — DELIVERED 2026-07-20.
3. 2.1-c — Extraction cleanup: skill-group prefixes, glued dates, part-time roles, encoding (Q3–Q5, Q7).
4. 2.1-d — Confidence-driven review markers and merge gating (C1), once the review screen exists.

Each ships independently and leaves production green.

5. Benchmark as regression fixture

Save the benchmark CV's expected structured output as a test fixture: 5 experiences (2 primary + 3 part-time) or a documented decision on the part-time roles, 3 projects, grouped skills flattened, English (Native) + Norwegian (B1) languages, education entry, contact with Norwegian location intact. Extraction changes are measured against it — without overfitting (the rules must generalise).

2.1-b delivered — extraction coverage (2026-07-20)

What changed. The gap was upstream only: the C# assembler (StructuredCvProfileJson.FromSections

• BuildStructuredCvFromNormalizedMarkdown) already maps Projects, Certifications and Languages headings — the AI normalize prompt just never emitted them, so they were dropped.

• tools/summarizer/app.py /cv/normalize: added # Projects and # Certifications headings with shapes; added a languages-from-prose rule (extract "native English", "Norwegian B1" from the summary even without a Languages section; ignore programming languages); added skill-group prefix stripping ("Development:", "DevOps & Infrastructure:", "Practices:" are dropped, only the skills remain).
• /cv/classify-block: added Projects and Certifications to the section enum + rules (fallback path).
• ProfileCvController.LooksLikeNormalizedMarkdownCv: recognises # Projects / # Certifications so a CV whose structured content is mostly those sections still takes the markdown path.

Verification. 4 new backend tests (CvExtractionCoverageTests) lock the C# mapping of Projects/Certifications/Languages; 1 new ai-service test (test_classify_block_supports_projects_section). 426 backend tests and 17 ai-service tests pass; app.py compiles. The LLM behaviour itself (prompt → headings) could not be run here (no Ollama), but the C# side that consumes the headings is proven, and the prompt change is additive/contract-safe.

Deployment note. These prompt changes live in the ai-service container, which deploy.sh does not rebuild by default. Deploy with DEPLOY_BUILD_AI_SERVICE=true ./deploy/deploy.sh (or rebuild ai-service manually) or the extraction change won't take effect. No database or backend schema change.

Not done here (moved to 2.1-c): deterministic C# safety-nets for skill-prefix stripping and glued-date splitting, and the "earlier/part-time roles" pattern. 2.1-b relies on the prompt for those; 2.1-c hardens them deterministically.

6. Open product decisions (need a call before building)

• Merge matching keys — how aggressively to treat two experiences as "the same" (company+title vs fuzzy). Conservative (fewer merges, some dupes) vs aggressive (cleaner, risk of wrong merges).
• Part-time/earlier roles — separate experience entries, or a sub-list on the primary role?
• Auto-apply threshold — does anything ever apply without review (e.g. an empty profile's first import), or is review always required?