FOCUS: Decoupling Expert Personas in LLMs to Enhance Domain Expert Capabilities
Guanyu Wang, Zidi Zhang, Xu Chu
cs.CL, cs.LG
Submitted: 2026-08-06
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Models (LLMs) can exhibit diverse personas, and activating expert personas has been shown to improve domain expertise and task accuracy.
Terminology
Abstract
Large Language Models (LLMs) can exhibit diverse personas, and activating expert personas has been shown to improve domain expertise and task accuracy. However, existing persona control methods often suffer from cross-domain coupling, which may lead to overly aggressive behavior in high-caution domains such as healthcare, or excessive conservatism in risk-sensitive domains such as financial trading. To address this issue, we propose FOCUS (ine-tuning with rthogonal ontrol for ncoupled persona). FOCUS first automatically extracts expert persona vectors from LLMs, then applies orthogonal decomposition to decouple domain-specific expert personas, and finally introduces an expert gating module to adaptively control persona activation according to task contexts. With a two-stage training strategy and a gated selection regularizer, the model learns to activate appropriate personas for both single-domain and cross-domain tasks. Experiments on financial, legal, medical, and cross-domain benchmarks show that FOCUS improves task accuracy and outperforms existing persona control methods. Our code is available at this url.
Sources
- The Llama 3 Herd of Models
- Constitutional AI: Harmlessness from AI Feedback
- Representation Engineering for Large-Language Models: Survey and Research Challenges
- Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models
- Qwen Look Again: Guiding Vision-Language Reasoning Models to Re-attention Visual Information
- GraphSOS: Graph Sampling and Order Selection to Help LLMs Understand Graphs Better
- Toy Models of Superposition
- Measuring Massive Multitask Language Understanding
- CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
- BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains
- Multitask Prompted Training Enables Zero-Shot Task Generalization
- Personas within Parameters: Fine-Tuning Small Language Models with Low-Rank Adapters to Mimic User Behaviors
- Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization
- Steering Language Models With Activation Engineering
- Finetuned Language Models Are Zero-Shot Learners
- MedEthicsQA: A Comprehensive Question Answering Benchmark for Medical Ethics Evaluation of LLMs
- TradingAgents: Multi-Agents LLM Financial Trading Framework
- Towards Best Practices of Activation Patching in Language Models: Metrics and Methods
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering