Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains

The paper addresses the complete absence of Modern Greek from NVIDIA's Nemotron retrieval models and from major multilingual retrieval benchmarks (BEIR, MIRACL), despite Greek legal, energy, financial and clinical documents being exactly the long, jargon-dense text that retrieval-augmented generation is meant to serve. The authors adapted the Nemotron family to Greek end to end: mining the corpus, training the retrieval stack, synthesising the reader supervision and building an evaluation benchmark, then measured what each stage actually buys.

The headline findings are uncomfortable: on all five of our domains a BM25 lexical baseline, with no learned parameters, outscores every off-the-shelf multilingual embedder we tested, including an 8B one. Fine-tuning a 1B embedder on 65,773 Greek retrieval pairs lifts nDCG@10 from 0.362 to 0.835, and the competence it gains generalises to general-domain Greek (+0.399 over its unadapted base), but Its advantage over BM25 does not generalise in the same way. We lead in the domains we adapted on and lose on general Greek, and we report both. A cross-encoder whose off-the-shelf contribution is inconsistent across domains becomes a reliable gain once adapted. LoRA-tuning a 30B-A3B mixture-of-experts model as a grounded reader raises judged answer correctness from 29.4% to 66.9%. The paper also reports the four ways our own instruments lied to us along the way, including two claims of ours that a second, larger evaluation did not reproduce.

Greek-adapted generation models exist — Meltemi-7B, Llama-Krikri-8B, GR-NLP-TOOLKIT — but that work targets general Greek generation; ours targets retrieval and grounded reading, so the artifacts do not overlap. Greek grounded reading resources like Belebele and XQuAD provide only passage-grounded multiple choice or extractive items over a gold passage. What the authors could not find was a Greek benchmark carrying multi-passage context, mined distractors, citation targets and unanswerable items. On the retrieval side: BEIR is English-only and MIRACL excludes Greek, so the multilingual embedders we test were trained and measured without it.

Scale and reduction. The authors started from 407,053 raw (query, document, label, domain) pairs and ended with 65,773 clean per-query records across five domains: energy, legal, finance, medical and a general foundation slice. Most of the reduction was regrouping flat pairs by query, but one cleanup step is highlighted: positive-wins: a document mined as a hard negative for a query, which is in fact that query's own positive, gets dropped.

Synthetic query generation. Queries were synthesised under strict constraints because Left to itself, an instruct model writes short, keyword-like queries, which look nothing like what real users type: long, formal, fact-seeking questions after a specific decision number, date, capacity or amount. Real users average 176 characters per query (68 for naive synthetic ones). Few-shot style-matching on real production queries doubled the reranking lift over BM25, from +0.040 to +0.081 nDCG@10. However, on real user queries the margin over BM25 was roughly +0.04, against +0.080 on our synthetic held-out set, so under half the lift survived contact with production traffic. The authors report this honestly: it is an internal observation, not a measured result.

The 512 truncation trap. "A max len of 512, the most common default in retrieval tooling, truncates roughly 87% of our training pairs. This was the single most consequential thing the data told us. Greek document chunks are long: positives have a median of 1,196 characters (≈1,034 tokens), p95 of 1,288 and a maximum of 4,407. Both models were therefore trained at max len 4,096. Anyone reproducing this at 512 would train on truncated positives and conclude the method does not work."

Negative mining. Negatives were mined using an off-the-shelf Qwen3-Embedding-8B with a per-query cosine-kNN window over the query's own domain pool, skipping the top ranks because those are too likely to be genuinely relevant and dropping near-duplicates. Around six negatives per query survive. Mining with BM25 instead produced measurably worse negatives, so the lexical signal that makes BM25 a strong retriever on Greek does not make it a good negative miner. The result: query-to-positive similarity averaged 0.687 against 0.474 to the mined negatives, a margin of 0.213, with no query weakly grounded to its own positive.

Corpus composition. The corpus is deliberately bilingual, 87.1% Greek, with English retained at 12.9% rather than filtered out, because code-switched Greek/English is pervasive in Greek technical and legal writing.

There is essentially no commercially licensed, human-written Greek instruction data at scale. The Greek slice of the Aya collection runs to millions of rows that way [machine-translated or templated], while its human-annotated Greek portion is on the order of hundreds, and the open assistant corpora we surveyed carry no Greek at all. The assembled Greek instruction pool is 296,034 conversations, with 206,909 filtered training rows used downstream. "At this scale a Greek instruction pool is necessarily translation-heavy, which carries a translationese risk that supervised fine-tuning will shape directly into model behaviour. This is not a solved problem and we do not claim to have solved it. It is the honest reason we cannot cleanly separate a base-model ceiling from a data ceiling."

40,000 examples were built from the retrieval train split only. The teacher is Sophea-Titan-1, our Greek-adapted 27B model (a LoRA on Qwen3.6-27B). Each context is the gold passage plus k ∈ [2, 5] shuffled hard negatives, numbered, with the gold landing at a random position p, and the target is answer [p], an answer carrying an explicit citation. Gold position is shuffled specifically to counteract the lost-in-the-middle effect, and abstention is an explicit training target, with Roughly 20% of examples are abstentions built from negatives alone.

The audit caught their own judge. Grounded answer faithfulness measured 95.2% with 3 wrong citations in 30,903 and zero duplicate queries. But abstention label noise measured an alarming 20.4%. They ran a control "in which the gold passage was provably removed, rows that must be abstentions, and the same judge scored those at 23.5%. The control came in worse than the real data, which means 20.4% was mostly the judge's own false-positive floor rather than corpus noise. Real abstention noise is bounded near 5%." Three gaps are flagged: all 30,903 grounded answers cite exactly one chunk (no multi-document citation supervision), abstention targets use only two canned strings, and 8.7% of answers are ultra-terse.

HERA (Hellenic Retrieval-Augmented) is a 4,946-item Greek Wikipedia long-retrieval benchmark with citations, consisting of 3,712 answerable and 1,234 unanswerable (25%) items, with 1,145 multi-hop (23%) items, difficulty from L1 factoid to L5 synthesis, and contexts of 8 to 40 passages, 21.2 mean. Construction is two-stage: Qwen3.5-122B generates the question, answer and citation; GLM-5.2 then independently verifies that the item is answerable only from its source passage, correct, and well formed. Hard distractors are mined by hybrid BM25 plus dense retrieval fused with RRF and then reranked, so the context noise is the highest-scoring non-gold passage rather than a random one.

A declared confound. Our reader judge is a served Qwen3.5-122B-A10B-FP8, the same family as the model that authored HERA's items... family preference is a documented failure mode of LLM-as-judge. Mitigations: no item survives without independent GLM-5.2 verification, and the spot-check sampled the corpus by hand. Decontamination was verified rather than assumed: A 13-gram overlap check between the benchmark (77,411 distinct normalised 13-grams) and the full 206,909-row instruction pool returns zero overlapping rows.

Figure 1 is the result we did not expect. BM25 scores 0.757. Qwen3-Embedding-8B scores 0.680. The unadapted Nemotron embedder scores 0.362. BM25 wins on energy (+0.170), finance (+0.134) and general validation (+0.029), and loses only legal (−0.026). The practical reading is blunt: any Greek RAG system that replaced BM25 with an off-the-shelf multilingual embedder made its retrieval worse on these domains.

Scale does not rescue the family: "Within the same off-the-shelf family... 8B and 4B are indistinguishable (0.6801 vs. 0.6843, a gap of −0.004), and dropping all the way to 0.6B costs only 0.070 (0.6140). The entire family sits below BM25 across a 13× parameter range... doubling the parameter count buys nothing, because the missing ingredient is not capacity. It is exposure to the language."

However, the authors carefully delimit this claim. Out of domain on HERA, "the family is not flat at all. It is a clean monotone ladder, 0.5268 → 0.6182 → 0.6504, every step significant... and the 8B rung finally catches BM25: +0.0062, 95% CI [−0.0057, +0.0177], not separable. So the flatness above is a property of this corpus, not of Greek retrieval. They state the narrow version because we can only support the narrow version: one family, two query distributions."

Full fine-tuning of Nemotron-3-Embed-1B used InfoNCE contrastive loss with 7 × B mined hard negatives per query in the denominator, and GradCache to hold a logical batch of 256 at 4,096 tokens on a single GPU. Training loss curves show The Nemotron models start markedly higher and finish lower: Greek was already supported by the Qwen backbone before training, and out of distribution for Nemotron.

Results. "Macro nDCG@10 goes from 0.362 to 0.835... Every domain gains at least +0.42, and the gain is largest exactly where the base was weakest: finance +0.53, starting from 0.19. A base score under 0.21 does not mean degraded. It means nonfunctional on Greek financial text. Against BM25: +0.080, 95% CI [+0.072, +0.089]... A 1B model that has seen the language beats both a parameter-free baseline and an 8B model that has not."

Medical. On 650 queries over 24,812 passages, the adapted 1B scores 0.6867 vs BM25 0.6602: at +0.027, 95% CI [−0.0005, +0.054], the interval touches zero: on this set we cannot demonstrate an advantage over lexical search at this sample size.

Generalisation. Out of domain on HERA: "The adaptation transfers as language competence. Out of corpus, our 1B beats the unadapted Nemotron it was built from by +0.399, 95% CI [+0.387, +0.410]... That is the cleanest evidence in this paper for exposure, not capacity: the 1B base sat at 0.165, barely functional, and adaptation moved it to within 0.007 of a 0.6B model whose backbone already spoke Greek. But: The win over lexical search does not transfer. BM25 scores 0.6442 here and beats both our adapted models... the ordering we report in-domain reverses."

Fusion is the right move. A baseline that beats you is a component, not only a rival. Fusing dense and lexical with reciprocal rank fusion (RRF, k=60, equal weights, nothing tuned) scores 0.6715 out of domain vs BM25's 0.6442 (+0.027, CI [+0.019, +0.035]) and lifts Recall@10 from 0.733 to 0.782. In domain, where dense is the stronger component, the fusion reaches 0.6798 vs 0.6670 dense alone (+0.013, CI [+0.007, +0.019]). "The weight moves in the direction you would expect, favouring the dense side in domain and the lexical side outside it... the effect is a property of combining dense with lexical retrieval on Greek rather than of one checkpoint. The honest conclusion: neither system dominates, that they fail differently, and that the combination beats the better of the two in both regimes... For a Greek RAG system the practical recommendation follows directly, and it is not the one we expected when we started: run both."

Matryoshka truncation. Truncated to 512 dimensions the index is 4× smaller and still scores 0.823, which is 98.5% of full quality and still +0.066 over BM25.

The cross-encoder is trained pointwise with binary cross-entropy on 394,579 pairs at ~11% positive. Second stages are usually assumed to help. We measured that assumption against the right control: the no-rerank floor, i.e. the first stage's own ranking.

First evaluation (750 queries). "On this evaluation the off-the-shelf reranker's contribution is statistically indistinguishable from not reranking at all (−0.006, p = 0.52), while adapted, the same architecture is worth +0.047 (p < 0.001). Per domain it significantly degrades one set (validation, −0.050, p = 0.018) and significantly helps another (legal, +0.039, p = 0.003), which averages to nothing."

Second evaluation (2,580 queries). "We then measured it again, and one of those conclusions did not hold... On the larger evaluation the off-the-shelf reranker does clear the floor, by +0.032, CI [+0.023, +0.041]. We are no longer willing to say an unadapted cross-encoder contributes nothing. What we can say is that its contribution is inconsistent: it is significant on two of five sets, indistinguishable from zero on two more, and negative in point estimate on energy. The final ordering: adapted Nemotron-1B +0.061, adapted Qwen-0.6B +0.041, off-the-shelf Nemotron-1B +0.032, floor 0.8248. The methodological point is unchanged... No evaluation that omits the no-rerank floor can distinguish these cases. A 0.6B adapted cross-encoder nearly matches the 1B (not separable at 750 queries; +0.020 for the larger at 2,580). Parameter count separates the two adapted arms by a little; adaptation separates them from the unadapted one by three times as much. What the second stage was mostly missing was never architecture or scale. It was exposure to Greek."

Swapping both stages together lifts end-to-end nDCG@10 from 0.559 to 0.848, a 52% relative gain. More decisively, Recall@10... rises from 0.624 to 0.955. The off-the-shelf stack loses the answer entirely for about 38% of queries; ours loses it for under 5%. This bounds grounded answering: E[grounded correct] ≤ Hit@k, since A reader cannot ground an answer in a passage it never received. The off-the-shelf first stage therefore caps grounded end-to-end accuracy at 0.624 no matter how good the reader is, while ours raises that ceiling to 0.955.

The reader is Nemotron-3-Nano-30B-A3B, a mixture-of-experts model with 3B active parameters, LoRA-tuned (rank 16, α 32, dropout 0.05, 439M trainable parameters, 1.37% of the model) for one epoch at 8,192 tokens, 32.7 hours on four B200s. The 40k cite-and-abstain examples are only 16.2% of a 246,909-row blend — the remainder is the Greek instruction pool with a 7.2% English replay slice intended to limit catastrophic forgetting.

Results. Judged answer correctness rises 29.4% → 66.9%... Wilson 95% intervals over the judged items are [27.9, 30.9] and [65.3, 68.4]. Faithfulness rises 25.2% → 84.5%... a 3.4× gain against correctness's 2.3×. The base model does not merely answer wrongly, it answers ungroundedly, producing content the retrieved context does not support roughly three times in four.

Conditional scoring caveat. "Both arms are scored only on grounded items they attempted, and they attempt different numbers: the base declines 16 of the 3,712 answerable items, the adapted reader 128. Counting every refusal on an answerable item as wrong instead, correctness reads 28.7% → 64.3% over the full 3,712."

Position bias removed. "The base model degrades monotonically as the gold passage moves later in the context (31.9% → 29.4% → 26.1%), the effect reported by Liu et al. (2024). The adapted…

1. Train retrieval embedders at 4,096-token context, not 512. The 512 default truncates 87% of long-document training pairs and would make any adaptation appear to fail. An adapted 1B embedder at 4,096 raised nDCG@10 from 0.362 to 0.835. The improved system retrieves correctly from long legal/financial/medical chunks that dominate real corpora.

2. Mine hard negatives with a large dense embedder, not BM25. Use per-query cosine-kNN over the domain pool, skip the top ranks (too likely relevant), drop near-duplicates, and keep 6 negatives per query. This yields a query-to-positive similarity margin of 0.213 and measurable better training signal than BM25-mined negatives. The improved system trains contrastively with 7×B hard negatives in the InfoNCE denominator via GradCache (logical batch 256 at 4,096 tokens on one GPU).

3. Apply the positive-wins cleanup: any document mined as a hard negative that is actually the query's own positive gets dropped. This prevents the model from learning to push away its own gold document.

4. Always run hybrid dense + BM25 with reciprocal rank fusion (equal weights, k=60). A lexical baseline that beats you is a component, not only a rival. Fusion beats the better single component in both regimes: +0.027 over BM25 out-of-domain and +0.013 over dense alone in-domain. The improved system wins regardless of whether the query distribution matches the adaptation domains.

5. Deploy with Matryoshka truncation at 512 dimensions: 98.5% of full quality (0.823 vs 0.835 nDCG@10) with a 4× smaller index, still +0.066 over BM25.

6. Synthesize training queries with few-shot style-matching on real production queries. Real users type long, formal, fact-seeking questions (176 characters), not short keyword-like queries. Style-matching doubled reranking lift over BM25 (+0.040 → +0.081). Also validate on real user queries: only half the synthetic lift survived production traffic, so the system's reported margin should be discountable.

7. Fine-tune the cross-encoder pointwise (binary cross-entropy) on 395k pairs at 11% positive, and always measure it against the no-rerank floor — the first stage's own ranking. Off-the-shelf reranker contributions are inconsistent (from statistically zero to +0.032 depending on evaluation size); an adapted reranker is a reliable +0.047 to +0.061. The improved pipeline can distinguish reranker helps from reranker does nothing, which is invisible without the floor.

8. Adapt a 0.6B reranker instead of scaling up: an adapted 0.6B nearly matches an adapted 1B (+0.020 gap at 2,580 queries). What the reranker was missing was exposure to the language, not architecture or scale.

9. LoRA-tune a small-active-parameter MoE reader on cite-and-abstain supervision: gold passage plus 2–5 shuffled hard negatives, numbered, gold at random position p, target answer [p] (explicit citation), with 20% of examples being abstentions built from negatives alone. This produced: judged correctness 29.4% → 66.9%, faithfulness 25.2% → 84.5%, abstention on unanswerable items 1.2% → 30.5%, and complete removal of lost-in-the-middle position bias. The improved reader grounds answers in citations and refuses unanswerable items at 30× the base rate.

10. Add a 7% English replay slice during language adaptation to limit catastrophic forgetting; verify with a broad benchmark sweep (mean −0.004 across 13 benchmarks, Greek gains on 6 of 8).

11. Bound reader expectations by retrieval recall. A reader cannot ground an answer in a passage it never received, so optimize Hit@k as the hard ceiling on grounded correctness (0.624 → 0.955 in the chained stack, which caps end-to-end accuracy).

12. Treat abstention as unsolved and improve it deliberately: the 30.5% refusal rate is still a failing grade. Add multi-document citation supervision (the training set used only single-chunk citations), more than two abstention target strings, and supervision against ultra-terse answers.

13. Audit LLM judges with control sets where ground truth is provably known (e.g., rows with the gold passage removed must be abstentions). The judge's own false-positive floor was 23.5%, which explained most of the apparent 20.4% abstention label noise. The improved evaluation system quantifies its judge's error floor and subtracts it before trusting labels.

14. Build benchmark items with two different model families (one generates question/answer/citation, a second independently verifies answerability, correctness, and well-formedness), and mine hard distractors by fusing BM25 + dense with RRF then reranking. Verify decontamination empirically with 13-gram overlap checks against the full training pool (zero overlapping rows), rather than assuming it.

15. Score the model's actual output language. An MCQ evaluator that read only Latin answer letters recorded a Greek-letter-answering model as random guessing — a fake catastrophic-forgetting result. The improved evaluator accepts language-appropriate answer tokens.

16. Average, don't sum, distributed losses, or the effective learning rate scales silently with world size and runs become incomparable across GPU counts. Also verify validation curves with and without prompt prefixes — a missing prefix made the validation curve run backwards while true quality rose.

17. Evaluate second-stage components on at least two evaluation sizes. One conclusion (off-the-shelf reranker contributes nothing) was overturned when the evaluation grew from 750 to 2,580 queries. Report intervals and refuse single-run point estimates.

18. On hybrid Mamba-Transformer stacks, do not LoRA only the attention blocks — most capacity lives outside attention, and an attention-only LoRA was worse than no adaptation at all.