Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation".

Jane: Here's the gist of the paper. The gist The study presents a comparative analysis of training small language models (SLMs) in specialized biomedical datato-text generation tasks, exploring post-training methods like SFT, DPO, ORPO, and GRPO with Qwen-based SLMs on medication package leaflets to assess cross-dataset generalizability.

Tom: Dataset and Models The study uses a D2T dataset focusing on medication leaflets - BioLeaflets curated from a corpus of 1,336 package leaflets of medicines sampled from the European Medicines Agency website<ref:2607.13430#pg2> The dataset is split into six sections, including product overview, pre-administration information, dosage instructions, side effects, storage instructions, and package & other information identified by heuristic rules<ref:2607.13430#pg3> To test model generalizability, a new drug label dataset called FDALeaflets is constructed using drug labeling information collected from openFDA<ref:2607.13430#pg3> This new dataset reflects distinct regulatory conventions, section taxonomies, and linguistic patterns compared to BioLeaflets<ref:2607.13430#pg3> The base models employed are a family of Qwen2.5-based SLMs evaluated across three model sizes—0.5B, 3B, and 7B parameters<ref:2607.13430#pg3> They also compare against Meditron3-Qwen2.5-7B, a medically fine-tuned variant of Qwen<ref:2607.13430#pg3> Baselines included the BioLeaflet T5 model and GPT-5<ref:2607.13430#pg3>.

Jane: Post-Training Alignment Methods The researchers systematically evaluate four representative post-training alignment strategies: SFT, DPO, ORPO, and GRPO<ref:2607.13430#pg4> Supervised fine tuning (SFT) is the most widely used approach for domain adaptation, where models are trained on paired input–output examples by optimizing the negative log-likelihood of a reference text conditioned on the input<ref:2607.13430#pg5> Prompt loss weighting (PLW) is introduced to control the relative contribution of prompt tokens to the overall training objective, which is adopted for SFT and ORPO experiments<ref:2607.13430#pg6>.

Tom: Direct preference optimization (DPO) optimizes a language model using pairs of chosen and rejected responses without requiring an explicit reward model<ref:2607.13430#pg6> Odds ratio preference optimization (ORPO) extends preference-based alignment by unifying supervised likelihood maximization and preference learning within a single training objective<ref:2607.13430#pg6> Group relative policy optimization (GRPO) is a reinforcement learning–based method that optimizes a policy using relative preferences among groups of generated outputs rather than isolated preference pairs<ref:2607.13430#pg6>.

Paper summary: Tom: So looking at the full picture of "Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet," we see that the work systematically compared SFT, DPO, ORPO, and GRPO using Qwen2 point 5 models on medication leaflets <ref:2607.13430#pg1>

Jane: The main implication is that choosing the right alignment strategy is central to getting a model that actually performs well in this specialized biomedical data-to-text generation task <ref:2607.13430#pg1>

Lu: The paper establishes that ORPO provides the strongest performance among the RL-free methods for semantic fidelity, while GRPO demonstrates better robustness when moving from training data to new regulatory domains <ref:two thousand six hundred seven point one three four three zero#pg2

Meng: For those of us shipping models, it’s a clear signal that we need to be very deliberate about which fine-tuning objective we choose based on whether our primary goal is maximizing accuracy within the known domain or handling shifts in the regulatory environment <ref:two thousand six hundred seven point one three four three zero#pg2

Tom: Exactly, and it highlights that decoder-only small language models are performing well here, beating encoder–decoder baselines and proprietary LLMs constrained to prompt adaptation <ref:2607.13430#pg10>

Jane: It sets a foundation for how we approach advancing biomedical generative NLP beyond just simple supervised fine-tuning, emphasizing that alignment strategy selection is a central design consideration for deployable systems <ref:two thousand six hundred seven point one three four three zero#pg10

Conclusion: Tom: So we’re wrapping up on this paper about "Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet" and the main authors are from a group focused on applying these models to healthcare documentation tasks <ref:two thousand six hundred seven point one three four three zero#pg1

Jane: It looks like the core idea is testing four different ways—SFT, DPO, ORPO, and GRPO—to fine-tune these small models so they can actually write accurate patient instructions instead of just guessing <ref:two thousand six hundred seven point one three four three zero#pg1

Lu: The paper sets up a comparison between these methods on a standard dataset called BioLeaflets and then tests how well they generalize when the model sees a completely different set of regulatory rules in this new dataset called FDALeaflets <ref:two thousand six hundred seven point one three four three zero#pg2

Meng: I’m curious about the real deployment stuff here. If we’re shipping these models, which one of these alignment methods would I actually trust to handle a real-world scenario where the input data changes slightly <ref:two thousand six hundred seven point one three four three zero#pg2

Tom: Well, what they found is that ORPO performed best in terms of semantic similarity on the original data and it consistently beat SFT and DPO across most metrics <ref:two thousand six hundred seven point one three four three zero#pg7

Jane: That’s interesting because ORPO combines supervised learning with preference learning without needing an explicit reward model, which is a neat trick for making the models better at matching the specific style of medical text <ref:two thousand six hundred seven point one three four three zero#pg7

Lu: But then they look at the generalization—when you take those aligned models and try to use them on that new FDALeaflets set, GRPO actually showed the best performance there <ref:two thousand six hundred seven point one three four three zero#pg8

Meng: So ORPO is good for accuracy in a known setting, but GRPO seems better at handling the shift when regulatory styles change. That’s a big difference for us if we’re dealing with different drug labeling standards across different regions <ref:two thousand six hundred seven point one three four three zero#pg8

Tom: Exactly. They found that GRPO achieved the highest semantic similarity score on that new set, reaching zero point nine four zero seven, which is better than GPT-five and the others tested in this cross-dataset test <ref:two thousand six hundred seven point one three four three zero#pg8

Jane: What this means for us is that we can’t just rely on one alignment technique; you have to pick a strategy based on whether your primary concern is getting perfect text for one specific set of documents or ensuring the model stays robust when it encounters new types of medical paperwork <ref:two thousand six hundred seven point one three four three zero#pg8

Lu: The paper clearly shows that post-training alignment isn't just a small tweak; it’s what separates the models that are actually useful in this specialized biomedical area from the ones that fall short <ref:two thousand six hundred seven point one three four three zero#pg10

Tom: So while the numbers look promising for semantic quality and generalization, the paper makes it clear that getting a model ready for real-world medical use still involves a lot of careful human oversight on top of these alignment techniques <ref:two thousand six hundred seven point one three four three zero#pg10