An Agentic Generative Large Language Model for Treatment Planning of Colorectal Cancer

GatorOnco is an agentic generative large language model (LLM) developed for colorectal cancer (CRC) treatment planning. It was developed using a total of 282 billion tokens of biomedical text, including a healthcare system-scale clinical text with 166 billion tokens from the University of Florida (UF) Health. The model was built through a multi-stage training strategy: starting from Meta-Llama-3.1-8B, the authors performed continued pre-training on the 282-billion-token biomedical corpus to create GatorTronLlama, then merged this with its instruction-tuned counterpart using model merging (mergekit-evo) to mitigate catastrophic forgetting and preserve instruction-following capabilities. A two-stage post-training approach was applied, consisting of supervised fine-tuning (SFT) followed by reinforcement learning (RL) using Group Relative Policy Optimization (GRPO) under a KL-divergence constraint. An agentic retrieval-augmented generation (RAG) approach was implemented to dynamically integrate time-sensitive clinical guidelines (NCCN guidelines for colon cancer v4.2025 and rectal cancer v3.2025) into the reasoning process, enabling the model to initiate search actions during generation and use retrieved guidelines to guide treatment plan generation through reasoning. A Safety-Oriented Hierarchical Reward Function was co-designed with UF Health oncologists, which enforces that the model only generates specific treatment regimens (e.g., FOLFOX) if it correctly identifies the high-level therapeutic modality (e.g., Chemotherapy), prioritizing therapeutic correctness and patient safety.

The model was trained and evaluated using a dataset of 395 real-world UF Health CRC cases, which were manually annotated by three UF Health oncologists. The dataset was split into 316 cases for model development (276 for training, 40 for validation) and 79 held-out test cases. For single-therapy determination, GatorOnco achieved the best macro-F1 score of 0.924, outperforming other models (F1 scores from 0.748 to 0.910), including Llama-3.3-70B, a general-domain LLM almost 9 times larger. For high-stakes invasive treatments, GatorOnco achieved F1-scores of 0.979 for chemotherapy, 0.964 for surgery, and 0.933 for radiotherapy. For multi-therapy determination, GatorOnco achieved the best F1 score of 0.913 and suggested the correct multiple-therapy options for 73 of 79 cases (92.4%), compared with 55–68 cases (69.6–86.1%) by other LLMs. For narrative treatment plan generation, GatorOnco achieved the best average performance score (Overall = 0.6002), representing a 29.7% relative improvement over Llama-3.1-8B-Instruct without domain adaptation (0.4626), and outperformed substantially larger models including MedGemma-27B and Llama-3.3-70B-Instruct. The largest performance gains were observed on semantic and faithfulness scores compared with lexical overlap-based metrics, with GatorOnco achieving the highest AlignScore (0.4847), exceeding Llama-3.1-8B-Instruct (0.3820) and outperforming MedGemma-27B and Llama-3.3-70B (0.4689 and 0.4723, respectively).

In a blind, randomized clinical evaluation conducted by five UF Health oncologists on the 79 test cases, GatorOnco significantly outperformed open-source LLMs and achieved expert-level performance on par with UF Health oncologists. Specifically, compared with expert oncologists, GatorOnco was rated significantly higher for readability (4.46 versus 4.19, P < 0.01) and completeness (3.91 versus 3.52, p = <0.01); was rated statistically comparable (i.e., on par) for correctness (4.09 versus 4.11, p = 0.921), currency (4.04 versus 3.98, p = 0.478), and safety (4.22 versus 4.22, p = 0.999). In the mixed-effects model (Score System + (1Rater) + (1Sample)), GatorOnco showed the strongest fixed-effect estimate (β = 1.064, SE = 0.038, P < 0.01), followed by the expert gold standard (β = 0.928, SE = 0.038, P < 0.01), both significantly better than open-source Llama model (P < 0.01 for all pairwise contrasts). Compared with oncologist-authored ground-truth plans, GatorOnco received higher composite scores (β = 0.136, SE = 0.038, P < 0.01). Inter-rater agreement analysis showed overall consistency across evaluation dimensions, with the highest agreement observed for completeness (α = 0.849) and readability (α = 0.757), and moderate agreement for correctness (α = 0.626), currency (α = 0.679), and safety (α = 0.645).

The findings demonstrate that integrating agentic reasoning with large-scale domain adaptation can bridge the gap of generative AI in high-stakes cancer treatment planning. The authors conclude that general-purpose LLMs need to be adapted to the clinical context for high-stakes clinical tasks, and that deep involvement of oncologists at every step, from training to evaluation, is critical. They show that an 8-billion-parameter domain-adapted local LLM can significantly outperform much larger general-purpose models, suggesting that for specialized, high-stakes healthcare tasks, data quality and domain alignment are more critical than the size of AI models. The study also identifies a bottleneck in developing AI for medicine: the lack of a machine-friendly format of clinical guidelines, and advocates a paradigm shift toward AI-friendly clinical guideline releases, such as the Fast Healthcare Interoperability Resources (FHIR) standard. Limitations include findings derived from a single disease domain and a single institution, performance fluctuations with prompting structure, and the fact that GatorOnco remains a prototype tool that could assist oncologists with treatment planning for CRC, requiring further steps such as fail-safe mechanisms, uncertainty quantification, deferral policies, and integration with existing clinical workflows before deployment in real-world cancer care.

Improvements to AI Systems:

1. Domain-Adapted Continued Pretraining with Model Merging: Implement a multi-stage training pipeline that starts with a base LLM, continues pretraining on a large-scale domain-specific corpus (e.g., biomedical text), and then merges the domain-adapted weights with its instruction-tuned counterpart using evolutionary model merging (e.g., mergekit-evo). This mitigates catastrophic forgetting while preserving general instruction-following abilities, enabling the system to retain broad capabilities while gaining deep domain knowledge.

2. Agentic Retrieval-Augmented Generation (RAG) with Dynamic Search Actions: Enhance the LLM with an agentic loop that allows it to autonomously decide when to issue search queries during generation, retrieve time-sensitive clinical guidelines (e.g., NCCN v4.2025), and integrate the retrieved content into its reasoning chain. This enables the system to stay current with evolving standards and produce treatment plans grounded in the latest evidence, rather than relying solely on static training data.

3. Hierarchical Safety-Oriented Reward Function for Reinforcement Learning: Design a two-tier reward structure for RL fine-tuning (using GRPO with KL constraint) that first verifies the high-level therapeutic modality (e.g., chemotherapy vs. surgery) before rewarding specific regimen correctness (e.g., FOLFOX). This enforces clinical logic and patient safety, preventing the model from generating specific treatments without correctly identifying the overarching care strategy, and reduces risky hallucinations.

4. Multi-Stage Post-Training with SFT + RL: Apply a two-stage post-training approach—supervised fine-tuning on expert-annotated clinical cases, followed by reinforcement learning with a KL-divergence constraint to prevent reward hacking. This improves both lexical accuracy and semantic faithfulness, as evidenced by higher AlignScore gains over lexical metrics, making the system more reliable for narrative plan generation.

5. Clinician-in-the-Loop Evaluation and Iterative Refinement: Integrate a blind, randomized evaluation protocol with multiple domain experts (e.g., oncologists) scoring on dimensions like correctness, safety, completeness, readability, and currency. Use mixed-effects models to compare system outputs against expert gold standards, then iteratively refine the reward functions and training data based on inter-rater agreement (e.g., Krippendorff’s alpha) to target weak dimensions (e.g., correctness and safety).

6. Structured Output and Machine-Friendly Guideline Integration: Develop the system to output treatment plans in a structured, interoperable format (e.g., FHIR-based) and incorporate a mechanism to parse and reason over machine-readable clinical guidelines. This addresses the bottleneck of unstructured guideline text, allowing the AI to directly query and apply guideline elements (e.g., staging, regimens) without manual extraction, improving accuracy and reducing ambiguity.

7. Uncertainty Quantification and Deferral Mechanisms: Add a confidence-scoring layer that quantifies the model’s uncertainty for each treatment recommendation. When confidence falls below a threshold (e.g., for ambiguous or multi-therapy cases), the system automatically defers to a human oncologist or flags the case for review, enhancing safety in high-stakes decisions and preventing overconfident errors.

What the Improved AI System Can Do:

• Generate accurate, guideline-compliant treatment plans for colorectal cancer (and potentially other diseases) with F1-scores above 0.92 for single-therapy and 0.91 for multi-therapy, matching or exceeding expert-level performance on correctness, safety, and currency.

• Dynamically fetch and reason over the latest clinical guidelines (e.g., NCCN updates) during generation, ensuring recommendations remain current even as standards evolve post-training.

• Provide structured, machine-readable treatment outputs (e.g., FHIR) that integrate directly into electronic health records and clinical workflows, reducing manual data entry and errors.

• Explain its reasoning by first stating the therapeutic modality (e.g., “chemotherapy indicated”) before specifying the regimen (e.g., “FOLFOX”), making its logic transparent and auditable for clinicians.

• Operate locally with only 8 billion parameters, enabling deployment on institutional hardware without cloud dependency, while outperforming 70-billion-parameter general models on domain-specific tasks.

• Flag low-confidence or complex cases for human review, reducing liability and supporting oncologists as a decision-support tool rather than a replacement.

• Continuously improve through clinician feedback loops, with measurable gains in readability and completeness (e.g., 29.7% relative improvement in overall narrative quality) while maintaining safety parity with human experts.