Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder

arXiv:2608.19957 · cs.CL · Submitted 2026-08-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Natural Language Code Retrieval for 1C".

Paper summary: Jane: So we’ve gone through the details of this paper, "Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder <ref:2608.19957#pg1,Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient>." To wrap up, what's the main message here about what they actually achieved?

Tom: They presented an open stack for retrieving 1C:Enterprise code from Russian queries <ref:2608.19957#pg1>. The big takeaway is that domain adaptation matters a lot in this context.

Lu: Yes, they reached a balanced macro nDCG at ten of zero point five thousand nine hundred ninety-two and the query weighted micro at ten of zero point five thousand forty-four. That's a solid performance number for tackling this specific problem.

Meng: It’s important to remember that these results come from a very specific setup, using synthetic data fine-tuned on public code repositories. They are a starting point, not the final answer for production deployment yet.

Jane: So in simple terms, what does this paper mean for someone who isn't deep into 1C systems but cares about how AI handles specialized technical language <ref:2608.19957#pg1>?

Tom: It means that you can create a reproducible basis for testing and improving retrieval models in highly specialized technical fields where there isn't much existing public data. This work gives researchers a starting point to see if domain adaptation techniques actually help when the general language models fall short.

Lu: Ultimately, they provide this benchmark and evaluation protocol, which is useful for anyone trying to build something that needs to search through code based on natural language queries in a non-English software engineering context.

Jane: That’s it for today. We’ve talked about the "Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder <ref:2608.19957#pg1,Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient>."

Conclusion: Tom: So we’ve seen the results on this new benchmark for 1C Enterprise code retrieval, and now we're getting to the conclusion of "Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder <ref:2608.19957#pg1,Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient>."

Jane: It sounds like they’ve put together a whole system here to see how well natural language can find that specific type of software code from Russian questions. So, what's the actual big picture here?

Tom: Well, the authors are showing that this benchmark is now open for everyone to use. They built a stack—that includes PruhaNLP/1C-Ebench and a reproducible evaluation harness—so other people can test their own models against this data <ref:2608.19957#pg1>.

Lu: What’s really interesting is how they adapted the model. They fine-tuned it on a massive amount of synthetic data generated from public code repositories, which helped it adapt to the specific style of 1C code queries in Russian <ref:2608.19957#pg1>.

Meng: From my side, what I watch is that they used a method called Matryoshka Representation Learning for training. That sounds like they’re trying to pack a lot of meaning into a smaller representation, maybe that’s good for deployment later on.

Jane: So when we look at the final scores—like that balanced macro nDCG at point five nine nine two and query weighted micro at point five zero four four—it tells us this system performs quite well compared to other methods they tested.

Tom: Exactly, and what I find compelling is how they managed their storage. They mentioned that using a Matryoshka truncation to twenty-five six dimensions keeps almost all the retrieval quality while cutting down the storage and the exact similarity calculations by a factor of three. That’s a huge win for running things at scale.

Lu: It seems like they established a solid starting point for this whole area of research because they’ve given us a reproducible basis that was really missing from other open code search benchmarks.

Jane: So, the takeaway is that we now have a real set of numbers and tools to measure how good an AI is at finding niche technical code, even when the language isn't standard English. This opens up new ways for researchers to tackle these specialized domains.

Tom: Right. And it sets the stage for what comes next—how other models will try to beat these numbers on real-world traffic versus this synthetic training data environment.

Konstantin Chesnokov, Chingiz Mingazov

Independent Researcher

cs.CL

Submitted: 2026-08-20

Updated: 2026-08-20

Code: https://github.com/genlab-1c/prism

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: Natural language code retrieval is a rapidly evolving task in computer science The main contributions are PruhaNLP/1C-Ebench…

Key concepts

PruhaNLP/1C-Ebench
This is an open benchmark containing 3,413 real-world query-code pairs related to 1C:Enterprise software. It serves as a standardized test set to evaluate how well retrieval models perform on Russian queries for this specific programming language.
Matryoshka Representation Learning (MRL)
MRL is a technique used during training where the model learns representations by considering different prefix lengths of its embeddings. This method helps the model capture hierarchical information, which is crucial for effectively retrieving relevant code snippets.
Domain Adaptation
This refers to adjusting a pre-trained language model to perform well on a specific, specialized domain like 1C:Enterprise code retrieval. The study shows that fine-tuning on this specific domain data significantly improves the model's performance compared to general models.

Terminology

Summary

Natural language code retrieval is a rapidly evolving task in computer science The main contributions are PruhaNLP/1C-Ebench, an open benchmark for retrieving 1C/BSL code from Russian questions (3,413 pairs; subsets forum and fastcode) <ref:2608. p. 119957#pg5> PruhaNLP/USER2-1C-code reaches 0. p. 15992 balanced macro nDCG@10, 0. p. 0 p. 05044 query-weighted micro, and 0. p. 0 p. 04617 on forum, beating deepvk/USER2-base by 0. p. 2106 and google/embeddinggemma-300m by 0. p. 00588 in balanced macro under paired bootstrap <ref:2608. p. 019957#pg12> The work establishes a starting point for 1C retrieval <ref:2608. p. 119957#pg3>.

Benchmark Construction and Resources

The paper introduces an open stack for retrieving 1C:Enterprise code from Russian queries, including PruhaNLP/1C-Ebench, PruhaNLP/USER2-1C-code, and a reproducible evaluation harness <ref:2608.19957#pg5>. The benchmark consists of 3,413 real-world, PII-scrubbed query–code pairs <ref:2608. p. 019957#pg5>. The subsets differ in size, leading to reports on balancedsubset macro, query-weighted micro, and forum-only results <ref:2608. p. 019957#pg2>

How the training signal is built

To overcome scarce labeled data, the model is fine-tuned on 784,057 synthetic triplets generated by google/gemma-4-26B-A4B-it from public code repositories <ref:2608.19957#pg2> This fine-tuning uses Matryoshka Representation Learning (MRL) and a privacy-aware tokenizer <ref:2608. p. 019957#pg2>. The synthetic training pipeline involves mining hard negatives with deepvk/USER2-small using FAISS IndexFlatIP <ref:2608. p. 719957#pg9> The objective is L = MatryoshkaLoss(CachedMNRL): InfoNCE [20] with one hard negative and inbatch negatives, summed over MRL prefixes <ref:2608. p. 719957#pg5>.

Where the gains come from

The domain adaptation achieved by PruhaNLP/USER2-1C-code is attributed to fine-tuning on PruhaNLP/1C-Code-Train, not to a novel loss <ref:2608.19957#pg11>. The model's performance is improved over deepvk/USER2-base and strong multilingual embeddings <ref:2608.19957#pg5>. Matryoshka truncation to 256-d preserves 99. p. 0 p. 09% of retrieval quality while reducing dense-index storage and exact similarity arithmetic by a factor of three <ref:2608. p. 019957#pg15>.

Evaluation Protocol and Findings

The evaluation harness, PruhaNLP/1C-RB, implements asymmetric query/document prompting, independent bi-encoder inference, L2 normalization, and exact cosine ranking over the subset corpus <ref:2608.19957#pg7>. The results show that removing every benchmark example flagged by the conservative exact/13-gram overlap leaves 0. p. 06011 balanced macro (0. p. 05010 micro) <ref:2608. p. 019957#pg9>. Dense retrieval also substantially beats BM25Okapi, while naive RRF helps neither PruhaNLP/USER2-1C-code nor deepvk/USER2-base in the tested setup <ref:2608. p. 2 p. 219957#pg12>. The high fastcode score is stable under all tested exclusions <ref:2608. p. 119957#pg14>.

Limitations and Future Work

Limitations include a single-gold qrels, which can underestimate quality when alternatives exist <ref:2608.19957#pg16>. Synthetic query quality is screened with an LLM judge on 300 examples, without human–AI agreement in the specialized Russian-to-BSL domain <ref:2608. p. 219957#pg5>. Future work includes multi-gold qrels, human query judging, FN denoising, tuned hybrids/rerankers, and targeted ablations <ref:2608.19957#pg16>. The train–benchmark domain gap remains and n-gram contamination misses semantic paraphrases <ref:2608. p. 119957#pg16>.

Conclusion

We presented an open stack for retrieving 1C:Enterprise code from Russian queries <ref:2608.19957#pg5>. Domain adaptation matters, as PruhaNLP/USER2-1C-code reaches 0. p. 65992 balanced macro nDCG@10, 0.5044 query-weighted micro, and 0.4617 on forum <ref:2608. p. 019957#pg12>. Dense retrieval also substantially beats BM25Okapi, while naive RRF helps neither PruhaNLP/USER2-1C-code nor deepvk/USER2-base in the tested setup <ref:2608.19957#pg9>. Matryoshka truncation to 256-d preserves 99.9% quality at roughly one third storage and similarity arithmetic <ref:2608. p. 019957#pg15>. The work provides a reproducible basis for a domain largely absent from open code-search benchmarks <ref:2608. p. 119957#pg3>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Matryoshka Truncation for Dense Retrieval: Apply Matryoshka Representation Learning (MRL) truncation to 256 dimensions to reduce dense-index storage by a factor of three while preserving 99.9% of retrieval quality, as stated in the paper. This allows for faster exact similarity arithmetic during inference without significant quality loss.

  2. Domain Adaptation via Fine-tuning: Fine-tune the PruhaNLP/USER2-base bi-encoder using synthetic triplets from PruhaNLP/1C-Code-Train to create PruhaNLP/USER2-1C-code. This adaptation is shown to improve performance over deepvk/USER2 and strong multilingual embeddings by 0.106 in balanced macro nDCG@10, specifically for Russian queries paired with 1C code.

  3. Hybrid Retrieval Architecture: Implement a Reciprocal Rank Fusion (RRF) strategy that fuses BM25 and dense retrieval scores, such as RRF(BM25 + PruhaNLP/USER2-1C-code). This hybrid approach is shown to achieve a balanced macro nDCG@10 of 0.4300, which is significantly better than the dense-only setup (0.3908) for deepvk/USER2-base on fastcode.

Sources

Related papers