MameLoshnLM: Yiddish Language Model and Evaluation Benchmark

The paper, published at COLM 2026, presents Mame Loshn LM, described as the first open-source 8B-parameter language model built specifically for Yiddish. The authors identify Yiddish as an important test case for multilingual NLP: Despite Yiddish's rich textual tradition, its limited digital presence and the scarcity of reliable evaluation resources have constrained progress in Yiddish language modeling. They argue that Existing multilingual corpora and benchmarks are often poor proxies for the language, containing substantial amounts of noisy, machine-translated, and misclassified text.

The paper introduces two supporting resources alongside the model: Oytser (אוצר, treasure), a high-quality Yiddish pretraining corpus, and Kashes (ק ַאשעס, difficult questions), a multi-task evaluation benchmark. Using these resources, the authors continue pretraining Llama 3.1 8B to obtain Mame Loshn LM, and show that it outperforms open baselines of similar scale across benchmark tasks.

A key motivating observation is that the problem extends beyond Yiddish: "For historically rich but digitally underrepresented languages, the main obstacle to language-model development may be not only data scarcity, but mismatch between authentic language use and the noisy public web data on which multilingual models are often trained."

The authors conducted a detailed audit of the Yiddish portion of mC4, which contains approximately 0.3B words and 0.1M pages tagged as Yiddish. Their analysis revealed that "fewer than half (42.2%) of the documents in the Yiddish mC4 split are genuine high-quality Yiddish from validated native sources. At least 29.8% of the documents appear to be machine-translated, and an additional 21.9% are Hebrew texts mistakenly identified as Yiddish, and the remaining 6% is a long tail of short fragments, multilingual pages, and small uncatalogued Yiddish sources. They conclude that the existence of a Yiddish split in a large multilingual corpus should not be equated with the availability of native Yiddish pretraining data."

Oytser was constructed from two source groups. The web-native Yiddish sources include Yiddish news websites and magazines, the Yiddish Wikipedia, and a Yiddish translation of the Hebrew Bible; in addition, we included several Yiddish forums, where users write in a freer style. The Yiddish Book Center Corpus (YBC) is the largest source: With more than 12K books and approximately 720M Yiddish words, this is, to our knowledge, the largest and most comprehensive digital source of Yiddish texts currently available, digitized via OCR using Jochre 3. In total, the corpus contains more than 915M words from 9 high quality online and literary sources (approximately 5.28B tokens). The authors also applied a de-identification pipeline for user-generated forum data, removing emails, phone numbers, URLs, and account metadata, retaining only decontextualized post and comment text.

Kashes is described as the first multi-task evaluation benchmark for Yiddish language models, spanning 9 tasks: four linguistic analysis tasks (POS tagging, lemmatization, dependency parsing, transliteration) drawn from the YiTB treebank (Andrews, 2025); named entity recognition evaluated on three datasets (EHRI-NER, WikiANN, and newNLP NER); and three language-understanding tasks from the Aya machine-translated collection (PIQA, WikiQA, PAWS-Wiki).

The authors also contribute Kashes-mt, a new parallel corpus for machine translation, described as the largest natively authored Yiddish–English benchmark to date. It was constructed from bilingual online publications (Forverts and In geveb), where the Yiddish texts are originally written by native speakers and the English translations are produced by Yiddish scholars and professional translators. Using a four-stage pipeline (document matching, sentence alignment with SentAlign, quality filtering, deduplication), they produced 5,287 high-quality sentence pairs (995 from Forverts, 4,292 from In geveb). To prevent benchmark leakage, we exclude every source document from the Oytser pretraining corpus.

Mame Loshn LM was produced by continued pretraining of Llama 3.1 8B on a Yiddish corpus using a causal language modeling objective, with bfloat16 precision, AdamW optimizer, learning rate 2×10−5, cosine schedule with 2% warmup, one epoch, and maximum sequence length of 1,024 tokens. To mitigate catastrophic forgetting, we include English data from CC100 during continued pretraining in a Yiddish-dominant mixture: Yiddish accounts for 72% of words (92% of tokens), and English for the remaining 28% (8% of tokens). The model was trained on approximately 5.7 billion tokens over 36,663 steps on a single NVIDIA H200 GPU for roughly 207 GPU-hours.

Evaluated against five open-weight baselines (Llama 3.1 8B, Qwen3 8B, BLOOMZ 7B, Gemma-2 9B, EuroLLM 9B) in a 5-shot setting, Mame Loshn LM achieves the best average score of 62.6, ahead of Gemma-2 9B (57.0), Llama 3.1 8B (56.8), and Qwen3 8B (54.7). Its gains are concentrated on Yiddish-centered tasks: it leads on POS tagging, dependency parsing, transliteration, EHRI and newNLP NER. Notably, The performance of Mame Loshn LM is especially strong in the English to Yiddish translation (more than 11 COMET points improvement over the closest competitor).

The paper probes whether general multilingual models actually learn native-like Yiddish or a flatter approximation shaped by noisy data, focusing on two phenomena:

Loshn-koydesh (LK) vocabulary underproduction: In English→Yiddish translation of the Kashes-mt benchmark, gold references contain LK words in 6.2% of content tokens; Mame Loshn LM produces 4.7% while Llama 3.1 8B produces only 1.6% (p < 10−229). When a gold translation contains an LK word, Mame Loshn LM matches it 52.4% of the time vs. 16.0% for Llama. The authors note that Llama's LK output is concentrated in proper nouns and culturally salient items, while it substitutes Germanic alternatives for common LK vocabulary (e.g., משפּחה family with פ ַאמיליע, מלחמה war with קריג). Importantly, Per-sentence LK recall correlates only weakly with COMET score (Spearman ρ = 0.23), meaning COMET is largely blind to loss of the loshn-koydesh lexical layer.

Yiddish-specific morphology: On lemmatization, the strongest contrast appears on ge- prefixed past participles (50.8% accuracy for Mame Loshn LM vs. 5.1% for Llama), Hebrew-origin plural patterns (30.2% vs. 2.3%), and the definite article paradigm (28.2% vs. 15.5%). On auxiliary verbs, the two models perform identically on regular forms of האָבן (84.7% vs. 84.7%), but on the suppletive verb זײַן whose inflected forms often look very different from the lemma, Mame Loshn LM achieves 20.0% vs. 8.9% for Llama. The authors conclude that the gap is therefore not uniform across morphology: it is largest when successful lemmatization requires knowledge of lexeme-specific inflectional system rather than simple regular transformations.

The authors tested whether historically related languages (Hebrew, German, Russian, Polish) improve Yiddish adaptation. Two variants were trained: Rebalanced (redistributing part of the English budget to Hebrew and German) and Expanded (adding all four languages while keeping English amount). However, "the multilingual variants remain competitive in translation and show isolated gains, they do not yield a consistently better Yiddish model, and on several Yiddish-centered tasks they underperform not only Mame Loshn LM but even the base Llama 3.1 model. They conclude that maintaining a strongly Yiddish-dominant training signal with a limited amount of English is a more effective strategy than reallocating that budget to related languages."

The paper positions this work as the first comprehensive LLM development effort for Yiddish, encompassing corpus construction, benchmark curation, model training, and evaluation. The authors note that several Kashes datasets were created by digital humanities scholars whose primary goal is extracting information from Holocaust testimonies and historical newspapers, so improved performance reflects not just benchmark gains but progress toward tools that these communities can actually use. Future work includes instruction tuning for interactive use, the creation of additional training and evaluation resources, and the application of Mame Loshn LM to large-scale digital humanities workflows. The model is released under a non-commercial license consistent with institutional agreements with the Yiddish Book Center and In geveb.

Improvements to AI systems:

1. Native-source-validated data curation. Replace blind reliance on multilingual web dumps (e.g., mC4) with provenance-based filtering: whitelist validated native sources, detect machine-translated text, and distinguish orthographically similar languages (Yiddish vs. Hebrew) via calibrated language identification. The improved system can build high-quality corpora for low-resource languages, rejecting the 58% noisy/misclassified content found in existing web splits.

2. Forgetting-aware continued pretraining. Continue pretraining with a strongly target-dominant mixture (72% Yiddish, 28% English by words) plus one epoch, short sequences, and cosine LR. The improved system can adapt to a new language while preserving general English capabilities, avoiding the degradation observed with aggressive target-only training.

3. Leakage-free benchmark design. Construct evaluation sets from natively authored, expert-translated sources and exclude every benchmark source document from pretraining. The improved system can be evaluated without inflated scores caused by benchmark contamination.

4. Lexical-layer-aware MT evaluation and training. Augment automatic MT metrics like COMET with per-sentence recall of culturally significant vocabulary (loshn-koydesh). The improved system can maintain and generate the appropriate literary/religious lexical layer instead of substituting generic Germanic alternatives, even when COMET fails to penalize the loss.

5. Morphology-targeted lemmatization. Train and evaluate on specific difficult inflectional phenomena: ge- prefixed participles, Hebrew-origin plural patterns, definite article paradigms, and suppletive verbs such as זײַן. The improved system can learn lexeme-specific inflectional systems, not just regular transformations, substantially improving lemmatization accuracy for morphologically complex languages.

6. Multilingual transfer with restraint. Avoid indiscriminately adding related languages to the training mixture. The improved system can determine when Hebrew/German/Russian/Polish transfer hurts target-language performance and instead rely on target-dominant plus English data, yielding consistently better results on language-centered tasks.

7. Privacy-preserving user-content incorporation. Include vernacular forum data through a de-identification pipeline that strips emails, phone numbers, URLs, and account metadata while keeping decontextualized text. The improved system can learn from informal register without memorizing or leaking personal information.

The resulting improved AI system can:

• Achieve the best average benchmark score (62.6) for Yiddish across POS tagging, dependency parsing, transliteration, NER, and language understanding, beating Llama 3.1 8B, Qwen3 8B, Gemma-2 9B, and EuroLLM 9B.

• Improve English→Yiddish translation by more than 11 COMET points over the closest open baseline.

• Match the correct loshn-koydesh word in 52.4% of cases (vs. 16.0% for Llama) when translating culturally rich sentences.

• Improve Yiddish lemmatization on ge- participles from 5.1% to 50.8% accuracy and on Hebrew-origin plurals from 2.3% to 30.2%.

• Provide usable NLP tools for digital humanities workflows, including information extraction from Holocaust testimonies and historical Yiddish newspapers, without compromising privacy or cultural-linguistic authenticity.

We present MameLoshnLM, the first open-source 8B-parameter language model built specifically for Yiddish. Despite Yiddish's rich textual tradition, its limited digital presence and the scarcity of reliable evaluation resources have constrained progress in Yiddish language modeling. Existing multilingual corpora and benchmarks are often poor proxies for the language, containing substantial amounts of noisy, machine-translated, and misclassified text. We address these gaps by introducing Oytser, a high-quality Yiddish pretraining corpus that combines contemporary web-native sources with literary materials, and Kashes, a multi-task benchmark spanning translation, linguistic analysis, information extraction, and language understanding. Using these resources, we continue pretraining Llama 3.1 8B to obtain MameLoshnLM. Across the tasks in the benchmark, MameLoshnLM outperforms open baselines of similar scale. Our analyses show that these gains are not only quantitative: relative to general-purpose multilingual models, MameLoshnLM better captures language-defining lexical and morphological patterns, pointing to a broader failure mode of noisy web-scale multilingual data for low-resource languages. Our results provide both a foundation for Yiddish NLP and a practical template for language model development in historically rich but digitally underrepresented languages.