fix(cv): preserve human languages during structured CV normalization

HumanLanguageCatalog built its lookup table solely from
CultureInfo.GetCultures, so which languages counted as human languages
depended on the host's ICU data rather than on the CV. Measured: 806
cultures on a normal Windows or Linux machine, exactly 1 under
globalization-invariant mode, and an English-only subset on a container
with trimmed ICU data.

Consequences by environment, all silent:
- full ICU: correct
- trimmed ICU: canonical names present in the reduced data survive and
  the rest are dropped, so a CV keeps English and loses Norwegian
- invariant: every language is dropped and a CV import loses its
  Languages section entirely, with no error

The tests were right and are unchanged. Seed the catalog explicitly with
the languages a CV realistically lists, before the culture enumeration,
which still runs and still adds breadth. Nothing in the seed collides
with a technical skill -- Go, Java, Swift, Rust and Basic are
deliberately absent, and Basic is also a proficiency level.

Verified 420 tests pass in four environments: Windows and Linux, each
with full ICU and with DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1. Before
this change the invariant runs failed 5 tests. No test was modified,
skipped or relaxed.

Added HumanLanguageCatalogTests to pin the seeded catalog, confirmed
non-vacuous by removing the seed and watching 15 tests fail.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
cesnimda
2026-07-19 19:19:17 +02:00
parent de35947244
commit 96816186cb
2 changed files with 91 additions and 0 deletions
+34
View File
@@ -103,6 +103,40 @@ public static class HumanLanguageCatalog
map.TryAdd(normalizedAlias, normalizedCanonical);
}
// Seeded FIRST, and deliberately not derived from the host.
//
// This table used to come only from CultureInfo.GetCultures, which returns whatever
// culture data the machine happens to carry: 806 entries on a normal Linux or Windows
// box, exactly 1 under globalization-invariant mode, and an English-only subset on a
// container with trimmed ICU data. So whether "Norwegian" was recognised as a human
// language depended on the deployment environment, not on the CV. Under invariant mode
// every language was silently dropped and a CV import lost its Languages section with
// no error at all.
//
// These are the languages a CV realistically lists. Culture enumeration still runs
// below and still adds breadth for free, but nothing here depends on it.
//
// Nothing in this list may collide with a technical skill — "Go", "Java", "Swift",
// "Rust" and "Basic" are deliberately absent. "Basic" is also a proficiency level.
string[] seed =
[
"English", "Norwegian", "Swedish", "Danish", "Finnish", "Icelandic",
"German", "Dutch", "French", "Spanish", "Portuguese", "Italian",
"Polish", "Czech", "Slovak", "Slovenian", "Croatian", "Serbian", "Bosnian",
"Bulgarian", "Romanian", "Hungarian", "Greek", "Albanian", "Macedonian",
"Russian", "Ukrainian", "Belarusian", "Lithuanian", "Latvian", "Estonian",
"Turkish", "Arabic", "Hebrew", "Persian", "Kurdish", "Pashto", "Urdu",
"Hindi", "Bengali", "Punjabi", "Gujarati", "Marathi", "Tamil", "Telugu",
"Malayalam", "Kannada", "Sinhala", "Nepali",
"Chinese", "Japanese", "Korean", "Vietnamese", "Thai", "Lao", "Khmer",
"Burmese", "Malay", "Indonesian", "Filipino", "Tagalog", "Javanese",
"Swahili", "Amharic", "Somali", "Hausa", "Yoruba", "Igbo", "Zulu", "Afrikaans",
"Catalan", "Basque", "Galician", "Welsh", "Irish", "Scottish Gaelic", "Maltese",
"Latin", "Esperanto", "Armenian", "Georgian", "Azerbaijani", "Kazakh", "Uzbek",
];
foreach (var language in seed) Add(language, language);
foreach (var culture in CultureInfo.GetCultures(CultureTypes.NeutralCultures | CultureTypes.SpecificCultures))
{
var english = CleanCultureLanguageName(culture.EnglishName);