JU'A -- A Benchmark for Information Retrieval in Brazilian Legal Text Collections

summary

The gist

Legal information retrieval in Portuguese remains difficult to evaluate systematically because available datasets differ widely in document type, query style…

In short

JUÁ is a public benchmark for evaluating information retrieval systems in Brazilian legal text collections, designed to address inconsistencies across different legal domains. It combines jurisprudence, normative text, and question-driven search into one protocol. Results show that domain-adapted dense retrieval excels on specific subsets like JUÁ-Juris, while BM25 remains strong where lexical cues are dominant.

Key concepts

JUÁ
A public benchmark for information retrieval in Brazilian legal text collections. It combines jurisprudence, normative text, and question-driven search settings under a unified protocol to allow for reproducible and comparable system evaluation across heterogeneous legal data.
JUÁ-Juris
A specific subset of JUÁ focused on jurisprudence from the Brazilian Federal Court of Accounts (TCU). It uses a simple exact positive-pair retrieval protocol where the query is the enunciado and the ground truth is the corresponding excerto, used to test domain adaptation.
BM25
A traditional ranking system that relies on lexical overlap between queries and documents. It performs well in settings where strong institutional phrasing and specific legal terminology are present, acting as a strong baseline against dense retrieval methods.
Domain Adaptation
The process of fine-tuning a dense retriever model using supervised query-document pairs from various sources like JUÁ train or SQuaD-pt. This adaptation helps the model perform better on specific legal domains by aligning its semantic understanding with the target collection's language and structure.

Terminology used across episodes

This episode discusses

The paper

JU'A -- A Benchmark for Information Retrieval in Brazilian Legal Text Collections · Read on arXiv

Jayr Pereira, Leandro Car໓sio Fernandes, Erick de Brito, Roberto Lotufo, Luiz Bonifacio

Centro de Ci§es e Tecnologia, Universidade Federal do Cariri (UFCA) · Faculdade de Engenharia Elétrica e da Computaγo, Universidade Estadual de Campinas (UNICAMP) · C˂mara dos Deputados · NeuralMind.ai

Legal information retrieval in Portuguese remains difficult to evaluate systematically because available datasets differ widely in document type, query style, and relevance definition. We present JUÁ, a public benchmark for Brazilian legal retrieval designed to support more reproducible and comparable evaluation across heterogeneous legal collections. More broadly, JUÁ is intended not only as a benchmark, but as a continuous evaluation infrastructure for Brazilian legal IR, combining shared protocols, common ranking metrics, fixed splits when applicable, and a public leaderboard. The benchmark covers jurisprudence retrieval as well as broader legislative, regulatory, and question-driven legal search. We evaluate lexical, dense, and BM25-based reranking pipelines, including a domain-adapted Qwen embedding model fine-tuned on JUÁ-aligned supervision. Results show that the benchmark is sufficiently heterogeneous to distinguish retrieval paradigms and reveal substantial cross-dataset trade-offs. Domain adaptation yields its clearest gains on the supervision-aligned JUÁ-Juris subset, while BM25 remains highly competitive on other collections, especially in settings with strong lexical and institutional phrasing cues. Overall, JUÁ provides a practical evaluation framework for studying legal retrieval across multiple Brazilian legal domains under a common benchmark design.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "JU'A -- A Benchmark for Information Retrieval in Brazilian Legal Text Collections".

Jane: Here's the gist of the paper. Legal information retrieval in Portuguese remains difficult to evaluate systematically because available datasets differ widely in document type, query style, and relevance definition We present JUÁ, a public benchmark for Brazilian legal retrieval designed to support more reproducible and comparable evaluation across heterogeneous legal collections More broadly, JUÁ is intended not only as a benchmark, but as a continuous evaluation infrastructure for Brazilian legal IR, combining shared protocols, common ranking metrics, fixed splits when applicable, and a public leaderboard The benchmark covers jurisprudence retrieval as well as broader legislative, regulatory, and question-driven legal search Results show that the benchmark is sufficiently heterogeneous to distinguish retrieval paradigms and reveal substantial cross-dataset trade-offs Domain adaptation yields its clearest gains on the supervision-aligned JUÁ-Juris subset, while BM25 remains highly competitive on other collections, especially in settings with strong lexical and institutional phrasing cues Overall, JUÁ provides a practical evaluation framework for studying legal retrieval across multiple Brazilian legal domains under a common benchmark design Benchmark Design and Datasets JUÁ is designed to evaluate information retrieval systems in the Brazilian legal domain under a unified and reproducible protocol The JUÁ benchmark combines complementary datasets spanning legal and regulatory domains, covering jurisprudence, normative text, and question-driven retrieval settings The benchmark can naturally be extended with new legal collections in future releases and is paired with a public leaderboard, which continuously ranks submissions under fixed evaluation settings and supports transparent comparison over time JUÁ-Juris is centered on jurisprudence drawn from the curated jurisprudence collection of the Brazilian Federal Court of Accounts (TCU) In our retrieval setup, the enunciado serves as the query, and the corresponding excerto is treated as the ground-truth positive passage The test-set construction is then performed via stratified sampling over difficulty bins derived from BM25 ranks The relevance judgments, also called qrels, are generated using a direct query–document correspondence (query-id == corpus-id) with binary relevance (score = 1), preserving a simple exact positive-pair retrieval protocol JUÁ combines two jurisprudence retrieval settings with broader legislative, regulatory, and question-driven retrieval scenarios under a common evaluation protocol How the training signal is built We fine-tune Qwen3-Embedding-4B as a domain-adapted dense retriever using supervised query–document pairs from four sources: JUÁ train, Ulysses train, Ulysses synthetic train, and SQuaD-pt The inclusion of SQuaD-pt is motivated by semantic regularization and query distribution diversification For adapting Ulysses to our training setup, we start from the legislative bills collection and exclude documents that overlap with the relevance-annotated split used in the original evaluation setup, preventing train–evaluation contamination Hard negatives are generated from precomputed first-stage BM25 retrieval runs For each query, we keep only candidates aligned with the training qrels and require that the gold document, i.e., the relevant positive document associated with the query in the supervision data, appear in the retrieved list; otherwise, the query is discarded from hard-negative construction The final composition is: JUÁ-Juris train (27,690), Ulysses train (42,580), Ulysses synthetic train (2,101), and SQuaD-pt (16,991), totaling 89,362 training instances Model adaptation follows the training protocol recommended for the Qwen Embedding family, implemented with ms-swift

24: and LoRA for parameter-efficient fine-tuning Where the gains come from The experimental results confirm this point.

Tom: Beyond ranking systems, JUÁ exposes meaningful trade-offs among lexical, dense, and reranking approaches across distinct legal retrieval regimes Domain-adapted dense retrieval yields its clearest gains on the supervision-aligned JUÁ-Juris subset At the same time, BM25 remains highly competitive on collections where lexical overlap, institutional phrasing, and domainspecific terminology remain strong signals Complementary analyses with MRR@10, MAP@10, and Recall@10 further show that these differences cannot be reduced to a single notion of retrieval quality JUÁ exposes substantial cross-dataset trade-offs and is therefore useful for distinguishing retrieval behavior beyond a single legal collection Benchmark-aligned adaptation is especially effective on the supervision-aligned jurisprudence setting represented by JUÁ-Juris, while broader cross-dataset evaluation remains necessary to assess generalization across Brazilian legal retrieval tasks Dataset-Specific Analysis The dataset-level results are easier to interpret when read alongside the design of the benchmark In practice, they point to two broad groups of retrieval tasks within JUÁ: jurisprudence search on one side, and broader legislative, regulatory, or question-driven retrieval on the other Within the jurisprudence group, however, the results are not uniform JUÁ-Juris is the subset where fine-tuning helps the most At the same time, BM25 remains a strong baseline on this subset because the enunciados are formulated by a specialized team within the TCU and capture a concise summary of the legal holding expressed in the decision from which they were extracted, which increases lexical consistency between queries and relevant jurisprudential passages and naturally benefits exactmatch retrieval On Ulysses-RFCorpus, BM25 is again highly competitive, which likely reflects the importance of exact term overlap, recurring institutional phrasing, and domain-specific lexical patterns in legislative retrieval NormasTCU occupies a more difficult middle ground where no approach is clearly dominant and where both sparse and dense methods must cope with long normative texts, article-level structure, and highly standardized regulatory language BR-TaxQA strongly favors dense retrieval, with all leading dense systems outperforming BM25 by a wide margin This contrast is useful for the benchmark: it shows that the nonjurisprudential portion of JUÁ is also heterogeneous, spanning regulatory retrieval tasks closer to document search and FAQ-style retrieval tasks closer to semantic answer matching Benchmark Limitations Although JUÁ is designed to improve comparability in Brazilian legal IR, it also has important limitations First, the benchmark is intentionally heterogeneous: it combines jurisprudence retrieval, normative retrieval, legislative retrieval, and question-answer retrieval under a common reporting scheme Second, the benchmark combines datasets with varying levels of naturalness and annotation procedures Third, some of the clearest gains from the domain-adapted retriever appear on JUÁ-Juris, which also contributes supervision to the fine-tuning mixture Future benchmark revisions would benefit from more explicit ablation protocols that separate these two effects Some benchmark design choices deliberately increase task difficulty, such as removing the ENUNCIADO field in JurisTCU and constructing JUÁ-Juris with difficulty-aware sampling These decisions improve discrimination among retrieval systems, but they also mean that the benchmark should be interpreted as a curated evaluation framework rather than as a direct proxy for all real-world legal search behavior We introduced JUÁ, a public benchmark for retrieval over Brazilian legal text collections, designed to support reproducible comparison across heterogeneous legal retrieval settings More broadly, we frame JUÁ not only as a benchmark, but as a continuous evaluation infrastructure for a legal IR landscape that is structurally diverse: jurisprudence retrieval, normative search, legislative retrieval, and question-driven legal retrieval do not reward the same modeling assumptions The experimental results confirm this point Beyond ranking systems, JUÁ exposes meaningful trade-offs among lexical, dense, and reranking approaches across distinct legal retrieval regimes Domain-adapted dense retrieval yields its clearest gains on the supervision-aligned JUÁ-Juris subset At the same time, BM25 remains highly competitive on collections where lexical overlap, institutional phrasing, and domainspecific terminology remain strong signals We view this heterogeneity as a central strength of the benchmark It makes JUÁ useful for leaderboard comparison, as well as for studying when semantic adaptation helps, when lexical retrieval remains difficult to beat, and how retrieval methods transfer across Brazilian legal domains The benchmark contributes both an evaluation infrastructure and an empirical lens on legal retrieval behavior in Portuguese Future work should focus on strengthening the benchmark as a resource rather than only extending the leaderboard In particular, promising next steps include expanding query diversity, refining protocols that separate benchmark-aligned adaptation from broader generalization, and incorporating additional legal collections and evaluation settings that further test transfer across institutions and retrieval regimes We hope that making JUÁ and its leaderboard public will support more transparent, cumulative, and analytically grounded progress in Brazilian LIR References

Title and authors: Tom: So we’re looking at the paper titled JUÁ -- A Benchmark for Information Retrieval in Brazilian Legal Text Collections. It sounds like they've tackled the problem where it's really hard to systematically evaluate how well we can find legal documents using Portuguese.

Jane: That makes sense because when you look at legal information retrieval, the datasets are so different—different document types, different ways people ask questions—that it’s tough to compare systems fairly.

Tom: Exactly. And the authors are Jayr Pereira and Leandro Carísio Fernandes and Erick de Brito and Roberto Lotufo and Luiz Bonifacio. They're coming from universities in Brazil, which gives them a good grounding in the specific legal context they're studying.

Lu: I see their background is in Computer Science at UNICAMP, which suggests they have the technical chops for setting up these kinds of retrieval evaluations.

Meng: I wonder if being based there helps them understand the nuances of Brazilian legal language and structure compared to, say, English legal texts.

The paper's summary: Jane: So, what’s the core idea here? Basically, they’re proposing JUÁ as this public benchmark for Brazilian legal retrieval because existing datasets don't match up well.

Tom: Right. They designed it to be a way to support more reproducible and comparable evaluation across these very different legal collections. It’s not just one set of documents; it covers jurisprudence, regulations, question-driven searches too.

Lu: They’re aiming for JUÁ to be more than just a static test; they want it to be a continuous evaluation infrastructure, meaning they have this public leaderboard and a shared protocol for submissions.

Jane: That's interesting because it moves beyond just testing one model once and lets researchers compare things over time against others by standardizing the data organization and the evaluation metrics.

The paper's improvements: Tom: They actually suggest some specific improvements to make JUÁ even better for evaluating retrieval systems. One big thing is combining different retrieval settings under a common protocol, which they call JUÁ standardizes data organization and evaluation metrics, allowing direct comparison across retrieval paradigms.

Lu: They also focus on how the training signal is built for their dense retriever. They fine-tune Qwen3 Embedding 4B using supervised pairs from four different sources—JUÁ train, Ulysses train, Ulysses synthetic train, and SQuaD-pt <ref:2604.06098#pg1>.

Meng: That sounds complicated because they’re mixing things up into one training set to adapt the model. I worry that if you mix those sources too much, the model might just memorize noise instead of learning how to retrieve well in any specific legal area.

Jane: It seems like they added SQuaD-pt for semantic regularization and query distribution diversification, which is smart because it helps make sure the retriever isn't just good at one narrow style of question.

Conclusion: Tom: So to wrap up, JUÁ is this public benchmark that combines multiple legal datasets under a shared design, and they're pushing for it to be a continuous infrastructure with that leaderboard. It shows the trade-offs between different retrieval methods in the Brazilian legal space.

Jane: Exactly. What’s really important is that it highlights how domain-adapted dense retrieval does best on the specific jurisprudence subset, but BM25 still holds up well where there are strong lexical cues or institutional phrasing.

Lu: I think the main contribution is showing these cross-dataset trade-offs clearly, which helps us understand when semantic adaptation really helps versus when a simpler method like BM25 is still competitive due to those specific linguistic signals.

Meng: From a deployment standpoint, I'm thinking about that domain adaptation gain on JUÁ-Juris subset. If you're only targeting jurisprudence, that dense retriever might be the right bet for production, but if you need something general across all legal domains, it’s not a silver bullet.

Lalam: My literature review confirms what they are saying; this benchmark is positioned as a tool to study when semantic adaptation helps and when lexical retrieval remains difficult to beat in Portuguese legal contexts.

More episodes

← Home