UniBuc at SemEval-2024 Task 2: Tailored Prompting with Solar for Clinical NLI
cs.CL
Submitted: 2026-09-30
Updated: 2026-09-30
Code: https://github.com/ClaudiuCreanga/sem-eval-2024-task-2
License: http://creativecommons.org/licenses/by/4.0/
The gist: This paper describes the approach of the UniBuc team in tackling the SemEval 2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials.
Terminology
Abstract
This paper describes the approach of the UniBuc team in tackling the SemEval 2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials. We used SOLAR Instruct, without any fine-tuning, while focusing on input manipulation and tailored prompting. By customizing prompts for individual CTR sections, in both zero-shot and few-shots settings, we managed to achieve a consistency score of 0.72, ranking 14th in the leaderboard. Our thorough error analysis revealed that our model has a tendency to take shortcuts and rely on simple heuristics, especially when dealing with semantic-preserving changes.
Sources
- A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Tasks
- Mistral 7B
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering