fix(cv): preserve human languages during structured CV normalization

HumanLanguageCatalog built its lookup table solely from
CultureInfo.GetCultures, so which languages counted as human languages
depended on the host's ICU data rather than on the CV. Measured: 806
cultures on a normal Windows or Linux machine, exactly 1 under
globalization-invariant mode, and an English-only subset on a container
with trimmed ICU data.

Consequences by environment, all silent:
- full ICU: correct
- trimmed ICU: canonical names present in the reduced data survive and
  the rest are dropped, so a CV keeps English and loses Norwegian
- invariant: every language is dropped and a CV import loses its
  Languages section entirely, with no error

The tests were right and are unchanged. Seed the catalog explicitly with
the languages a CV realistically lists, before the culture enumeration,
which still runs and still adds breadth. Nothing in the seed collides
with a technical skill -- Go, Java, Swift, Rust and Basic are
deliberately absent, and Basic is also a proficiency level.

Verified 420 tests pass in four environments: Windows and Linux, each
with full ICU and with DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1. Before
this change the invariant runs failed 5 tests. No test was modified,
skipped or relaxed.

Added HumanLanguageCatalogTests to pin the seeded catalog, confirmed
non-vacuous by removing the seed and watching 15 tests fail.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
cesnimda
2026-07-19 19:19:17 +02:00
parent de35947244
commit 96816186cb
2 changed files with 91 additions and 0 deletions
@@ -0,0 +1,57 @@
using JobTrackerApi.Models;
using Xunit;
namespace JobTrackerApi.Tests;
// The catalog used to be built solely from CultureInfo.GetCultures, so which languages it
// recognised depended on the host's ICU data: 806 cultures on a normal machine, exactly 1
// under globalization-invariant mode. CV imports silently lost their Languages section on a
// container with trimmed or absent ICU data, and every existing test still passed locally.
//
// These pin the explicitly seeded catalog, so deleting the seed list fails here rather than
// in production on a machine nobody tested.
public sealed class HumanLanguageCatalogTests
{
[Theory]
[InlineData("English")]
[InlineData("Norwegian")]
[InlineData("French")]
[InlineData("Spanish")]
[InlineData("German")]
[InlineData("Arabic")]
[InlineData("Chinese")]
[InlineData("Polish")]
public void Common_languages_resolve_without_relying_on_host_culture_data(string language)
{
Assert.Equal(language, HumanLanguageCatalog.NormalizeLanguageName(language));
}
[Theory]
[InlineData("norsk")]
[InlineData("bokmål")]
[InlineData("nynorsk")]
public void Norwegian_aliases_resolve_to_the_canonical_name(string alias)
{
Assert.Equal("Norwegian", HumanLanguageCatalog.NormalizeLanguageName(alias));
}
[Theory]
[InlineData("C#")]
[InlineData("Leadership")]
[InlineData("Public speaking")]
[InlineData("Go")] // a programming language, not a human one
[InlineData("Java")] // Javanese is a language; Java is not
[InlineData("Swift")]
[InlineData("Rust")]
public void Technical_skills_are_not_treated_as_human_languages(string skill)
{
Assert.Null(HumanLanguageCatalog.NormalizeLanguageName(skill));
}
[Fact]
public void A_language_embedded_in_a_phrase_is_extracted_with_its_level()
{
Assert.Equal("Norwegian", HumanLanguageCatalog.NormalizeLanguageName("Native Norwegian speaker"));
Assert.Equal("Native", HumanLanguageCatalog.ExtractLevel("Native Norwegian speaker"));
}
}
+34
View File
@@ -103,6 +103,40 @@ public static class HumanLanguageCatalog
map.TryAdd(normalizedAlias, normalizedCanonical);
}
// Seeded FIRST, and deliberately not derived from the host.
//
// This table used to come only from CultureInfo.GetCultures, which returns whatever
// culture data the machine happens to carry: 806 entries on a normal Linux or Windows
// box, exactly 1 under globalization-invariant mode, and an English-only subset on a
// container with trimmed ICU data. So whether "Norwegian" was recognised as a human
// language depended on the deployment environment, not on the CV. Under invariant mode
// every language was silently dropped and a CV import lost its Languages section with
// no error at all.
//
// These are the languages a CV realistically lists. Culture enumeration still runs
// below and still adds breadth for free, but nothing here depends on it.
//
// Nothing in this list may collide with a technical skill — "Go", "Java", "Swift",
// "Rust" and "Basic" are deliberately absent. "Basic" is also a proficiency level.
string[] seed =
[
"English", "Norwegian", "Swedish", "Danish", "Finnish", "Icelandic",
"German", "Dutch", "French", "Spanish", "Portuguese", "Italian",
"Polish", "Czech", "Slovak", "Slovenian", "Croatian", "Serbian", "Bosnian",
"Bulgarian", "Romanian", "Hungarian", "Greek", "Albanian", "Macedonian",
"Russian", "Ukrainian", "Belarusian", "Lithuanian", "Latvian", "Estonian",
"Turkish", "Arabic", "Hebrew", "Persian", "Kurdish", "Pashto", "Urdu",
"Hindi", "Bengali", "Punjabi", "Gujarati", "Marathi", "Tamil", "Telugu",
"Malayalam", "Kannada", "Sinhala", "Nepali",
"Chinese", "Japanese", "Korean", "Vietnamese", "Thai", "Lao", "Khmer",
"Burmese", "Malay", "Indonesian", "Filipino", "Tagalog", "Javanese",
"Swahili", "Amharic", "Somali", "Hausa", "Yoruba", "Igbo", "Zulu", "Afrikaans",
"Catalan", "Basque", "Galician", "Welsh", "Irish", "Scottish Gaelic", "Maltese",
"Latin", "Esperanto", "Armenian", "Georgian", "Azerbaijani", "Kazakh", "Uzbek",
];
foreach (var language in seed) Add(language, language);
foreach (var culture in CultureInfo.GetCultures(CultureTypes.NeutralCultures | CultureTypes.SpecificCultures))
{
var english = CleanCultureLanguageName(culture.EnglishName);