feat(cv): Phase 2.1-b — extract Projects, Certifications, and languages-from-prose
The structured model and StructuredCvProfileJson.FromSections already map
Projects/Certifications/Languages headings, but the AI normalize prompt
never emitted them, so on the benchmark CV the entire Projects section and
the in-summary languages (English Native, Norwegian B1) were silently
dropped. This closes that gap upstream — no backend schema or data change.
ai-service (tools/summarizer/app.py):
- /cv/normalize: added # Projects and # Certifications headings; a
languages-from-prose rule (pull "native English", "Norwegian at B1" out
of the summary even with no Languages section; ignore programming
languages); and skill-group prefix stripping ("Development:",
"DevOps & Infrastructure:", "Practices:" dropped, only the skills kept).
- /cv/classify-block: Projects and Certifications added to the section
enum + rules (fallback path).
Backend:
- LooksLikeNormalizedMarkdownCv now recognises # Projects / # Certifications
so those CVs still take the markdown assembly path.
Tests:
- CvExtractionCoverageTests (4) lock the C# mapping of Projects,
Certifications and Languages sections into the structured profile.
- ai-service test_classify_block_supports_projects_section (1).
426 backend tests, 17 ai-service tests pass; app.py compiles.
The LLM behaviour (prompt -> headings) needs Ollama to observe and was not
run here; the C# side that consumes the headings is proven and the prompt
change is additive. Merge-not-replace + the review screen are the next
increment (2.1-a, approved: always-review, conservative merge).
Deployment: these prompts live in the ai-service container, which
deploy.sh does not rebuild by default -- deploy with
DEPLOY_BUILD_AI_SERVICE=true or the change won't take effect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
cesnimda
@@ -0,0 +1,160 @@
|
||||
# Phase 2.1 — CV extraction: current implementation, weaknesses, proposal
|
||||
|
||||
> 2026-07-20. Investigation deliverable for Phase 2.1. Grounded in the code and in a real benchmark
|
||||
> CV (Connor Babbington, Systems Developer). No code changed yet — this is the "explain → identify →
|
||||
> propose" step before any implementation.
|
||||
|
||||
## 1. Current implementation (verified in code)
|
||||
|
||||
Upload flow (`ProfileCvController`, `CvProcessingQueue`, `tools/summarizer/app.py`):
|
||||
|
||||
```
|
||||
Upload file → CvUploadArtifact stored on disk, CvExtractionRun queued
|
||||
↓ (background) ProcessQueuedRunAsync
|
||||
Extract text (PDF/DOCX/image OCR)
|
||||
↓
|
||||
AI /cv/normalize → LLM rewrites messy text into markdown-sectioned CV text
|
||||
↓
|
||||
split into blocks, AI /cv/classify-block per block → {section, fields, confidence}
|
||||
↓
|
||||
assemble StructuredCvProfile (+ per-field metadata: confidence, method, sourceBlockId, reviewState)
|
||||
↓
|
||||
ApplyQueuedRunResultAsync → SaveVersionAsync + user.ProfileCvStructureJson = <new> ← REPLACE
|
||||
↓
|
||||
run.Status = "applied" (no user confirmation)
|
||||
```
|
||||
|
||||
The structured model (`Models/StructuredCvProfile.cs`) already has `Contact, Summary, Jobs,
|
||||
Education, Certifications, Projects, Skills, Languages` and per-field confidence metadata. Confidence
|
||||
is **captured and displayed** (the review chips) but used for nothing else.
|
||||
|
||||
## 2. Weaknesses (code-verified, then benchmark-verified)
|
||||
|
||||
### Architectural (the big two)
|
||||
|
||||
- **A1 — Replace, not merge.** `ApplyQueuedRunResultAsync` overwrites the entire profile every run
|
||||
(`SaveVersionAsync(structuredCv)` + `user.ProfileCvStructureJson = structuredJson`). Re-importing a
|
||||
CV, or a slightly worse OCR pass, **discards** whatever the user curated. This is the exact failure
|
||||
the vision names: the profile oscillates instead of getting richer. There is no dedup, no
|
||||
field-level merge, no "keep the better value".
|
||||
- **A2 — No review-before-apply.** The run auto-applies (`Status = "applied"`). The user never sees
|
||||
"we found 4 experiences, 15 skills, ⚠ 1 language" and never approves. The confidence metadata that
|
||||
would drive such a screen is already computed and then ignored.
|
||||
|
||||
### Extraction quality (verified against the benchmark CV)
|
||||
|
||||
- **Q1 — Projects and Certifications are dropped.** The `/cv/normalize` heading list
|
||||
(`app.py:566-573`) and the `/cv/classify-block` section enum (`app.py:622`) include only
|
||||
Contact / Summary / Work Experience / Education / Skills / Languages / Interests — **no Projects, no
|
||||
Certifications** — even though the data model and `StructuredCvProfileJson` fully support them. The
|
||||
benchmark CV's entire **Projects** section (JobTrack, InboxIntel, infra lab) is lost.
|
||||
- **Q2 — Languages in prose are missed.** The benchmark states languages only inside the summary
|
||||
("Native English speaker; Norwegian at B1"). There is no `# Languages` block for the classifier to
|
||||
pick up, and normalize won't synthesise one from prose, so **English/Norwegian + levels are lost**
|
||||
as structured languages.
|
||||
- **Q3 — Grouped skills leak their category label.** "Development: C#, .NET, Python…" — the classifier
|
||||
turns skill lines into items but nothing strips the "Development:" / "DevOps & Infrastructure:" /
|
||||
"Practices:" prefixes, so a skill like "Development: C#" or a junk "Development" item can appear.
|
||||
- **Q4 — Glued date/title runs.** Text extraction yields "2015–2023System Developer — Warwickshire
|
||||
County Council, UK" with no space between the date range and the title. The classifier expects clean
|
||||
`start`/`end`; a two-hop normalize→classify can mis-split or drop the date.
|
||||
- **Q5 — Nested "Earlier roles (part-time)" list.** Three secondary jobs (Royal Vapes, The Hodcarrier,
|
||||
Nuffield Health) sit as sub-bullets under a heading, not as standard entries. They are likely
|
||||
mis-classified as bullets of the parent job or dropped.
|
||||
- **Q6 — Two-hop LLM loses whole-CV context.** normalize (rewrites text, can hallucinate/omit) then
|
||||
per-block classify (no cross-block view) means duplicate or mis-sectioned entries and no dedup.
|
||||
- **Q7 — Encoding.** PDF text extraction returns mojibake for `ø`, en-dashes and apostrophes
|
||||
(`T�nsberg`, `years�`, `2015�2023`) depending on the extractor. Downstream this corrupts company
|
||||
names, locations and dates. (The DB round-trip for `æøå` is already fixed; this is the *extraction*
|
||||
side.)
|
||||
|
||||
### Confidence
|
||||
|
||||
- **C1 — Confidence is display-only.** It is computed per field and shown as chips, but never used to
|
||||
(a) gate what auto-applies, (b) flag low-confidence items for review, or (c) decide merge-vs-keep.
|
||||
|
||||
## 3. Proposed architecture
|
||||
|
||||
Keep the strengths (text extraction, per-field confidence metadata, the structured model) and add the
|
||||
two missing layers plus targeted extraction fixes.
|
||||
|
||||
```
|
||||
Extract → AI structured extraction (improved) → Validate → Normalize
|
||||
→ DIFF against current profile (new)
|
||||
→ REVIEW screen: "We found …" (new, confidence-aware)
|
||||
→ MERGE on accept (new: field-level, dedup, confidence-gated) ← never a blind replace
|
||||
```
|
||||
|
||||
- **Merge engine (A1).** On accept, merge per entity: match experiences/education/projects by a stable
|
||||
key (company+title+dates / institution+qualification / project name), update a field only when the
|
||||
incoming confidence is high **and** differs, add genuinely new items, and never delete a curated
|
||||
item the import didn't mention. Skills/languages dedup case-insensitively; language levels update
|
||||
only on higher confidence.
|
||||
- **Review screen (A2).** Reuse the diff: show counts and per-category adds/updates with ✓ (high) and
|
||||
⚠ (low-confidence) markers, and Accept all / Review individually / Discard. Nothing writes until
|
||||
accept. (This is also Phase 2's "Import CV review" screen from the workspace refactor — one build
|
||||
serves both.)
|
||||
- **Extraction fixes (Q1–Q7).** Add Projects + Certifications to both prompts and the block assembler;
|
||||
synthesise Languages from summary prose (or a dedicated language pass); strip skill-group prefixes;
|
||||
harden date/title splitting; handle the "earlier/part-time roles" pattern; fix extraction-time
|
||||
encoding.
|
||||
- **Confidence gating (C1).** Drive the review markers and the merge rules from the existing
|
||||
per-field confidence — no new scoring needed to start, just *use* it.
|
||||
|
||||
## 4. Recommended increment order (small, verified, deployable each)
|
||||
|
||||
1. **2.1-a — Merge instead of replace + review gate** (highest value, the vision's core). Backend
|
||||
diff+merge engine with tests; frontend review screen. Nothing auto-overwrites again.
|
||||
2. **2.1-b — Extraction coverage: Projects, Certifications, Languages-from-prose** (Q1, Q2). Prompt +
|
||||
assembler + parser, benchmarked on the CV. **— DELIVERED 2026-07-20.**
|
||||
3. **2.1-c — Extraction cleanup: skill-group prefixes, glued dates, part-time roles, encoding**
|
||||
(Q3–Q5, Q7).
|
||||
4. **2.1-d — Confidence-driven review markers and merge gating** (C1), once the review screen exists.
|
||||
|
||||
Each ships independently and leaves production green.
|
||||
|
||||
## 5. Benchmark as regression fixture
|
||||
|
||||
Save the benchmark CV's expected structured output as a test fixture: 5 experiences (2 primary + 3
|
||||
part-time) or a documented decision on the part-time roles, 3 projects, grouped skills flattened,
|
||||
English (Native) + Norwegian (B1) languages, education entry, contact with Norwegian location intact.
|
||||
Extraction changes are measured against it — without overfitting (the rules must generalise).
|
||||
|
||||
## 2.1-b delivered — extraction coverage (2026-07-20)
|
||||
|
||||
**What changed.** The gap was upstream only: the C# assembler (`StructuredCvProfileJson.FromSections`
|
||||
+ `BuildStructuredCvFromNormalizedMarkdown`) already maps `Projects`, `Certifications` and `Languages`
|
||||
headings — the AI `normalize` prompt just never emitted them, so they were dropped.
|
||||
|
||||
- `tools/summarizer/app.py` `/cv/normalize`: added `# Projects` and `# Certifications` headings with
|
||||
shapes; added a **languages-from-prose** rule (extract "native English", "Norwegian B1" from the
|
||||
summary even without a Languages section; ignore programming languages); added **skill-group prefix
|
||||
stripping** ("Development:", "DevOps & Infrastructure:", "Practices:" are dropped, only the skills
|
||||
remain).
|
||||
- `/cv/classify-block`: added `Projects` and `Certifications` to the section enum + rules (fallback
|
||||
path).
|
||||
- `ProfileCvController.LooksLikeNormalizedMarkdownCv`: recognises `# Projects` / `# Certifications` so
|
||||
a CV whose structured content is mostly those sections still takes the markdown path.
|
||||
|
||||
**Verification.** 4 new backend tests (`CvExtractionCoverageTests`) lock the C# mapping of
|
||||
Projects/Certifications/Languages; 1 new ai-service test (`test_classify_block_supports_projects_section`).
|
||||
426 backend tests and 17 ai-service tests pass; `app.py` compiles. The LLM behaviour itself
|
||||
(prompt → headings) could not be run here (no Ollama), but the C# side that consumes the headings is
|
||||
proven, and the prompt change is additive/contract-safe.
|
||||
|
||||
**Deployment note.** These prompt changes live in the **ai-service container**, which `deploy.sh`
|
||||
does **not** rebuild by default. Deploy with `DEPLOY_BUILD_AI_SERVICE=true ./deploy/deploy.sh` (or
|
||||
rebuild `ai-service` manually) or the extraction change won't take effect. No database or backend
|
||||
schema change.
|
||||
|
||||
**Not done here (moved to 2.1-c):** deterministic C# safety-nets for skill-prefix stripping and
|
||||
glued-date splitting, and the "earlier/part-time roles" pattern. 2.1-b relies on the prompt for those;
|
||||
2.1-c hardens them deterministically.
|
||||
|
||||
## 6. Open product decisions (need a call before building)
|
||||
|
||||
- **Merge matching keys** — how aggressively to treat two experiences as "the same" (company+title vs
|
||||
fuzzy). Conservative (fewer merges, some dupes) vs aggressive (cleaner, risk of wrong merges).
|
||||
- **Part-time/earlier roles** — separate experience entries, or a sub-list on the primary role?
|
||||
- **Auto-apply threshold** — does anything ever apply without review (e.g. an empty profile's first
|
||||
import), or is review always required?
|
||||
Reference in New Issue
Block a user