Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization

summary

The gist

Building general-purpose role-playing agents that faithfully portray any character from a natural-language profile remains challenging <ref:2606.27025#pg2> The dominant paradigm—supervised…

In short

The paper introduces Psy-CoT, a reasoning framework for role-playing agents that breaks down thinking into Interaction Perception, Psychological Empathy, and Logical Construction to move beyond simple mimicry. It pairs this with Role-Aware Policy Optimization (RAPO), which uses profile information to weight learning signals more effectively. This combination results in agents with significantly higher character fidelity on benchmarks.

Key concepts

Psy-CoT
A chain-of-thought framework that structures a model's pre-response reasoning into three sequential steps: Interaction Perception (understanding the scene), Psychological Empathy (feeling the character's emotions), and Logical Construction (deciding the character's next action). This forces the model to think dynamically from a psychological perspective rather than just copying surface patterns.
Role-Aware Policy Optimization (RAPO)
A technique used during reinforcement learning that adjusts how much importance is given to different parts of the model's training signal. It uses profile-token mutual information to amplify gradients for tokens specific to a character's profile when they are beneficial, effectively stopping reward hacking by ensuring role-specific choices have a disproportionately strong influence on learning.
Profile–Token Mutual Information
A mathematical measure used by RAPO to quantify how much the character's profile 'intervenes' in the model's decision-making process for a specific word or token. A positive value indicates that knowing the character's profile reduces uncertainty, meaning the model is more likely to generate that specific token when guided by the profile.

Terminology used across episodes

This episode discusses

The paper

Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization · Read on arXiv

Zhejiang University and Tencent Youtu Lab

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization".

Jane: Here's the gist of the paper. Building general-purpose role-playing agents that faithfully portray any character from a natural-language profile remains challenging <ref:2606.27025#pg2> The dominant paradigm—supervised fine-tuning—encourages behavioral mimicry without deep, human-like internal thought processes, resulting in poor out-of-distribution generalization <ref:2606.27025#pg2> Therefore, we propose Psy-CoT, a psychology-grounded chain-of-thought framework that decomposes preresponse reasoning into three role-specific steps—Interaction Perception, Psychological Empathy, and Logical Construction—so that the model thinks dynamically from the profile rather than merely mimicking surface patterns <ref:2606.27025#pg2> While structured reasoning provides a foundation, it alone is insufficient; reinforcement learning is essential to further align the model with character fidelity <ref:2606.27025#pg2> Psycho-Grounded Reasoning Framework Psy-CoT decomposes pre-response reasoning into three sequential steps: Interaction Perception, Psychological Empathy, and Logical Construction <ref:2606.27025#pg2> Interaction Perception requires the model to grasp the global scene and infer the interlocutor’s underlying intentions and mental state, grounded in Theory of Mind <ref:2606.27025#pg2> Psychological Empathy asks the model to filter the situation through the character’s unique background and flaws, identify its genuine emotional reaction, and decide how much to express, mask, or weaponize <ref:2606.27025#pg2> Logical Construction synthesizes facts through the character’s worldview and selects a short-term objective the character would naturally pursue <ref:2606.27025#pg2> Together, these steps constitute the character’s internal deliberation that precedes the final response <ref:2606.27025#pg2> Role-Aware Policy Optimization Mechanism To address reward hacking where generic and role-specific phrases receive identical gradient signals, we propose Role-Aware Policy Optimization (RAPO) <ref:2606.27025#pg2> RAPO uses profile–token mutual information to weight gradients asymmetrically—amplifying role-specific tokens under positive advantage while attenuating them under negative advantage <ref:2606.27025#pg2> Concretely, before computing gradients, RAPO computes the profile–token mutual information It = H(yt x) − H(yt P, x) by comparing the model’s entropy with and without the profile P <ref:2606.27025#pg2> The sign of It reveals how the profile intervenes: It > 0 indicates that P reduces uncertainty—the model, leveraging its understanding of the character, is more inclined to generate this token <ref:2606.27025#pg2> Experimental Validation and Results Experiments on CoSER, CharacterBench, and CharacterEval demonstrate that Psy-CoT outperforms existing role-playing CoT methods <ref:2606.27025#pg2> Furthermore, training with RAPO on Qwen3-4B-Instruct and Llama-3.1-8B-Instruct yields improvements of 40.1% over the untrained baseline on CoSER <ref:2606.27025#pg2> RAPO consistently achieves the best average score among all RL variants, with largest gains appearing on dimensions most indicative of role fidelity like Anthropomorphism on CoSER (+2.15 over GRPO) <ref:2606.27025#pg2> Training Dynamics and Human Evaluation Entropy dynamics show that RAPO maintains a stable exploration–exploitation balance throughout training, unlike GRPO and DAPO which suffer from monotonic entropy collapse <ref:2606.27025#pg2> Human evaluation confirms this effectiveness, with RAPO winning more often than it loses against every baseline, showing the improvements translate to perceptible gains in human judgment of role-playing fidelity <ref:2606.27025#pg2> Limitations and Future Work First, our experiments focus on the Qwen and Llama families; scaling to larger models remains to be verified <ref:2606.27025#pg2> Second, while RAPO shows consistent gains on the training datasets, its generalization to other role-playing datasets is not yet fully assessed <ref:2606.27025#pg2> Third, RAPO requires two forward passes per rollout sample to compute per-token mutual information, incurring additional computational cost and training time <ref:2606.27025#pg2> Overall, the main contributions of the work are as follows: (1) We propose Psy-CoT, a psychology-grounded chainof-thought framework that decomposes pre-response reasoning into Interaction Perception, Psychological Empathy, and Logical Construction <ref:2606.27025#pg2> (2) We propose RAPO, a role-aware policy optimizer that leverages profile–token mutual information to weight gradients asymmetrically, amplifying role-specific signals while maintaining exploration <ref:2606.27025#pg2> (3) Extensive experiments on three benchmarks and multiple model architectures validate the effectiveness of both Psy-CoT and RAPO <ref:2606.27025#pg2> Pruning redundant reasoning tokens The paper proposes Psy-CoT to move beyond behavioral mimicry by structuring reasoning into Interaction Perception, Psychological Empathy, and Logical Construction <ref:2606.27025#pg2> This framework ensures the model thinks dynamically from the profile rather than merely mimicking surface patterns <ref:2606.27025#pg2> RAPO is introduced to solve a critical issue where reward models treat generic and role-specific phrases identically, which leads to reward hacking <ref:2606.27025#pg2> RAPO uses profile–token mutual information It = H(yt x) − H(yt P, x) to quantify how much the profile shapes a token <ref:2606.27025#pg2> How the training signal is built The reward function R(y) is decomposed into three terms: R(y) = α Rfmt(y) + β Rthink(y) + γ Rans(y), where α + β + γ = 1 <ref:2606.27025#pg2> The profile–token mutual information It is used to compute asymmetric per-token weighting wt, which amplifies gradients on role-specific tokens when the advantage Aˆ ≥ 0 and attenuates penalties when A This asymmetry ensures that the model extracts more signal from character-specific choices while preserving exploration <ref:2606.27025#pg2> Where the gains come from The gains are observed across CoSER, CharacterBench, and CharacterEval benchmarks where Psy-CoT outperforms existing role-playing CoT methods <ref:2606.27025#pg2> RAPO consistently surpasses GRPO across multiple model scales <ref:2606.27025#pg2> The largest gains appear on dimensions most indicative of role fidelity, such as Anthropomorphism on CoSER (+2.15 over GRPO) <ref:2606.27025#pg2> RAPO-trained models also surpass proprietary models like GPT-5-Chat and DeepSeek-V4-Flash on certain metrics <ref:2606.27025#pg2> Training Dynamics and Human Evaluation The training dynamics of different policy optimization methods show that RAPO pulls ahead after approximately 40 steps, maintaining a clear lead thereafter <ref:2606.27025#pg2> This stability is due to RAPO's profile-aware loss reweight explicitly encouraging both exploitation and exploration on role-specific tokens, maintaining entropy within a healthy range throughout training <ref:2606.27025#pg2> Human evaluation confirms that RAPO wins more often than it loses against every baseline, with the largest margin over the untrained Base model being 70% win rate <ref:2606.27025#pg2> Limitations and Future Work The limitations include focusing experiments mainly on Qwen and Llama families, scaling to larger models remains unverified <ref:2606.27025#pg2> Generalization to other role-playing datasets is not yet fully assessed <ref:2606.27025#pg2> Additionally, RAPO requires two forward passes per rollout sample to compute profile–token mutual information, incurring additional computational cost <ref:2606.27025#pg2> The authors are actively conducting experiments on larger models and more diverse datasets to address these limitations <ref:2606.27025#pg2> The main contributions of the work are as follows: (1) We propose Psy-CoT, a psychology-grounded chainof-thought framework that decomposes pre-response reasoning into Interaction Perception, Psychological Empathy, and Logical Construction <ref:2606.27025#pg2> (2) We propose RAPO, a role-aware policy optimizer that leverages profile–token mutual information to weight gradients asymmetrically, amplifying role-specific signals while maintaining exploration <ref:2606.27025#pg2> (3) Extensive experiments on three benchmarks and multiple model architectures validate the effectiveness of both Psy-CoT and RAPO <ref:2606.27025#pg2> Where the gains come from The gains are observed across CoSER, CharacterBench, and CharacterEval benchmarks where Psy-CoT outperforms existing role-playing CoT methods <ref:2606.27025#pg2> RAPO consistently surpasses GRPO across multiple model scales <ref:2606.27025#pg2> The largest gains appear on dimensions most indicative of role fidelity, such as Anthropomorphism on CoSER (+2.15 over GRPO) <ref:2606.27025#pg2> RAPO-trained models also surpass proprietary models like GPT-5-Chat and DeepSeek-V4-Flash on certain metrics <ref:2606.

Title and authors: Tom: So we're looking at a paper here called "Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization". It sounds like they're trying to fix the problem where AI agents struggle to keep up with complex characters.

Jane: Exactly. The paper tackles that challenge by proposing a new way for these agents to think about a character's response, moving beyond just copying surface-level patterns.

Lu: They introduce this framework called Psy-CoT, which breaks down the thinking process into three distinct steps: Interaction Perception, Psychological Empathy, and Logical Construction. It's designed so the model thinks dynamically from the character's profile instead of just mimicking what it sees on the surface.

Meng: So, if I’m getting this right for a listener at home who might be driving or cooking, Psy-CoT is basically like giving the AI a little internal monologue before it speaks.

Lalam: Right, and that thinking phase is structured into those three steps: Interaction Perception to get the scene and intention, Psychological Empathy to feel the character's emotion and decide how to react, and Logical Construction to figure out what goal the character would naturally pursue based on their worldview.

Tom: That makes sense mechanically. So it starts with understanding the world around it, then layers in that emotional intelligence specific to the character, and finally determines what action is logical for that persona. It's a structured approach to pre-response reasoning.

Jane: But the paper points out that just having this structure isn't enough on its own; they argue you need reinforcement learning on top of it to really align the model with that character fidelity.

Lu: That’s a key point they make, because structured reasoning is just a foundation, and without reinforcement learning, the model can still drift away from the true character over time.

Meng: I worry about that alignment part though. If we use RL to fine-tune it, how do we stop it from just finding shortcuts that look good but aren't actually what the character would do?

Lalam: That’s where they bring in Role-Aware Policy Optimization, or RAPO. They propose using something called profile–token mutual information to weight the learning signals differently depending on whether a token is role-specific or not.

Tom: So instead of treating every response token the same way, RAPO amplifies the gradients for tokens that are specifically tied to the character's profile when things are going well, but it dampens them when things go poorly. It's about making sure role-specific choices get a much stronger push in training.

Jane: That sounds like a direct answer to the reward hacking problem they mentioned earlier, where generic responses might trick the system into getting a good score without actually being faithful to the character.

Lu: Precisely. They compute this profile–token mutual information by comparing the model's uncertainty with and without including that specific character profile in its input context. If that mutual information is positive, it means the profile helps reduce uncertainty, so the model is more likely to generate that token based on who the character is.

Meng: And from an engineering standpoint, I have to ask about the cost of this. The paper notes that RAPO requires two forward passes for every single rollout sample just to compute this mutual information, which adds computational overhead during training.

Tom: That’s a real trade-off. We get better fidelity and less reward hacking, but we pay for it with more training time and more compute per update step. It’s an explicit cost for that role-aware weighting mechanism.

Jane: So what are the results on this? The paper tests Psy-CoT on CharacterBench, CoSER, and CharacterEval to show how much better it is compared to existing chain-of-thought methods in role playing.

Lu: The experimental validation shows that Psy-CoT outperforms other role playing chain of thought methods across those three benchmarks. Specifically, training with RAPO on models like Qwen3-4B-Instruct and Llama3 point 1-8B-Instruct showed improvements of about forty point one percent over the untrained baseline on CoSER.

Tom: Forty point one percent is a solid number, especially when you look at the dimensions they measure. The largest gains they saw were on things that show role fidelity, like Anthropomorphism on CoSER, which was plus two point one five over a method called GRPO.

Meng: Plus two point one five is interesting because it’s tied to a dimension that actually matters for how believable the character feels. It’s not just about getting the structure right; it’s about getting the *feel* of the character right.

Jane: And they even found that RAPO consistently achieved the best average score among all reinforcement learning variants tested, and their human evaluation showed it winning more often than it lost against every baseline.

Lu: The training dynamics also look stable compared to other methods like GRPO or DAPO, which sometimes suffer from entropy collapse during training. RAPO maintains a healthy exploration–exploitation balance throughout the process.

Tom: Stability is good for production, I guess. So, to wrap up, this paper presents Psy-CoT for structured reasoning and RAPO for role-aware policy optimization using profile–token mutual information to guide learning.

Jane: It seems like they've built a system that forces the model to think through social cognition and character goals before it even tries to generate a response.

Lu: The core idea is that this structured reasoning, combined with the asymmetric gradient weighting from RAPO, allows for much deeper alignment with complex character profiles.

Meng: It’s ambitious work because it tackles two major problems at once: making the reasoning structure smarter and making sure the reinforcement learning process actually rewards character accuracy instead of just surface-level compliance.

Tom: So what does this mean for us on the ground? We're moving toward agents that don't just sound like a character, but genuinely think *as* that character within a given context.

Jane: It shifts the focus from simple pattern matching to grounding the response in psychological theory—specifically cognitive appraisal and emotion regulation theory—to understand what makes a character react in a certain way.

Lu: The implication is that for any future complex role playing, we need these kinds of multi-step reasoning frameworks layered with fine-grained, profile-aware optimization techniques.

Meng: For deployment right now, the cost of running this RL setup is high because of those two forward passes needed for the mutual information calculation. It’s not something you can just plug into a cheap inference pipeline easily yet.

Tom: Yeah, that's the reality for now. The gains are significant on fidelity metrics, but we still have to figure out how to make that training process more efficient before it's ready for mass deployment across all models.

Jane: Ultimately, this paper shows that layering structured reasoning with careful policy optimization can lead to significantly better character consistency in these agents.

Lu: The work on "Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization" lays a foundation for future models that exhibit more robust and nuanced persona coverage.

Meng: I just hope they scale this up to even larger models without making the training time prohibitive for real production use cases.

Tom: We'll have to keep an eye on those scaling experiments, but the results on Qwen and Llama show this direction is promising for getting agents closer to genuine character portrayal.

Jane: That's what we need to watch—the balance between deep reasoning and practical deployment cost is where the next big hurdle lies.

The paper's summary: Tom: So we've talked about Psy-CoT and RAPO, but what's actually going on in this paper? It boils down to two main ideas: first, using a three-step reasoning process called Psy-CoT to make the AI think better before it even talks, and second, using RAPO to make sure that thinking actually translates into character behavior during training.

Jane: That's right. The core problem they’re fixing is that current role playing agents just mimic surface patterns, like copying what a character *says*, instead of understanding what the character *means*. Psy-CoT tries to fix that by forcing the AI through three stages: Interaction Perception, Psychological Empathy, and Logical Construction.

Lu: Exactly. Interaction Perception is where the AI has to grasp the whole scene and figure out what someone is really trying to do based on theory of mind stuff. Then Psychological Empathy makes it filter everything through the character's specific background and flaws to decide how they should actually feel about it, or if they should hide that feeling or even use it as a weapon.

Meng: And Logical Construction is just the final step where the AI synthesizes all that understanding with the character’s worldview to pick a short term goal that makes sense for them. It's like giving the character an internal plan before they open their mouth.

Tom: But they don't stop there. They realize structure alone isn't enough, so they pair Psy-CoT with Reinforcement Learning to really lock in that character fidelity. That brings us to RAPO, which is their role aware policy optimizer.

Jane: RAPO’s job is to make sure the reinforcement learning process actually rewards the AI for being a good character rather than just rewarding it for using the right words, which they call reward hacking. They use profile token mutual information—basically measuring how much the character's specific profile helps or hurts when the AI chooses a word.

Lu: If that mutual information is positive, it means adding that token to the response actually reduces uncertainty for the model because it aligns with who we are trying to portray. So RAPO amplifies those role specific tokens while shrinking them down when things go negative.

Tom: That asymmetry is key, right? It makes sure that character-specific choices get a much stronger push during training than generic ones, which they say leads to these big gains across different benchmarks like CoSER and CharacterBench.

Jane: The results show that this combination doesn't just give them a little bump; it actually beats existing chain of thought methods and even some proprietary models on things like anthropomorphism, which is a big indicator of how well the character feels real.

Meng: I gotta say, the training dynamics look better than other RL setups they tested because RAPO manages to keep that exploration exploitation balance stable throughout the whole learning process instead of just collapsing into one thing.

Lu: That stability comes from how RAPO reweights things; it explicitly encourages both finding new ideas and sticking to what we know about the character's persona. It’s a more controlled way to learn.

Tom: So, when you look at this, the main shift is moving from models that just "sound" like characters to models that actually simulate the *internal deliberation* of those characters.

Jane: Exactly. This means future AI won't just be generating dialogue; it will be generating dialogue based on a whole internal psychological process we can ground in theory and optimize with this kind of targeted learning.

Lu: The real question is how much bigger models like these actually need to get before this works reliably, because they mentioned the experiments were mainly on Qwen and Llama families.

Tom: That's where the limitation comes in. It’s promising on those smaller models, but we still need to see if this scales up to the massive models we use in production right now.

The paper's improvements: Tom: So we've seen how they built the framework—Psy-CoT for reasoning and RAPO for optimization—but now let's talk about what these improvements actually mean for the agents themselves.

Jane: The main improvement is that these agents start thinking like real people, not just pattern matchers. Psy-CoT means the AI doesn't just spit out a response; it goes through a process where it perceives the scene, feels empathy for the character’s situation, and then constructs a logical action based on what that character would naturally want to do.

Lu: It moves them from just mimicking surface patterns to actually thinking dynamically from the character's profile. This grounding in social cognition and emotion theories is what makes it different.

Meng: For me, the practical improvement is that these agents become much more consistent when you push them into tricky situations. Instead of breaking character because they got confused, they follow a deeper logic path.

Tom: And RAPO makes sure that this internal thinking actually sticks during training. The optimization mechanism ensures that the reinforcement learning process is actually rewarding the AI for being faithful to the character's specific traits and not just for getting high scores on generic responses.

Jane: So, what does that look like in practice? It means we get outputs where every single part of a response is anchored to a specific attribute of the character's profile. It’s about deep character consistency, not just surface-level compliance.

Lu: If you look at the results, they show that RAPO training leads to gains on dimensions like anthropomorphism—how human-like the character feels—that are much larger than what we saw with previous methods like GRPO.

Tom: And I'm seeing numbers here that don't just say "it's better"; it says the gains are substantial, around twenty point one five over a baseline method on CoSER. That’s a measurable improvement in fidelity.

Jane: It changes how we evaluate these models because instead of just looking at if the dialogue makes sense, we start measuring if the character *feels* right in that moment. It's about capturing that nuance they’ve been struggling with for years.

Meng: From an engineering standpoint, this is what I want to see deployed: agents that are less prone to hallucinating character traits under pressure because their reasoning path is more robust.

Lu: The implication is that for complex role playing, we need this kind of multi-stage reasoning layered with profile aware optimization. It suggests a direction where the AI's internal deliberation becomes a measurable, trainable part of its architecture.

Tom: So, we’re moving toward agents that don't just sound like a character but genuinely simulate the internal thought process of that character during interaction.

Jane: And that moves us closer to having AI systems that can handle much more complex social scenarios with genuine character depth and consistency.

Conclusion: Tom: So to wrap up, we looked at Psy-CoT for structured thinking and RAPO for role aware policy optimization in this paper, "Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization."

Jane: The big implication is that we're moving past just surface level mimicry. We're building agents that actually simulate the internal deliberation of a character based on psychological grounding.

Lu: It suggests that for complex role playing, reasoning needs to be structured in these three specific steps before reinforcement learning even gets involved.

Meng: What I see is that this gives us a clearer path for debugging when models fail; we can trace whether the failure was in the perception step or the logical construction.

Lalam: I’m confident this paper lays solid groundwork because it tackles both the 'how' of reasoning and the 'why' of training alignment with one cohesive system.

Tom: It’s a lot to take in, but yeah, a solid piece of work that shows how to get better fidelity numbers without just throwing more data at a black box.

Jane: Exactly. It’s about making sure the model thinks like it knows the character's mind, not just like it memorized their lines.

Lu: The scaling challenge is still there though; they focused on smaller models, so we need to see if this blueprint holds up when we move to larger architectures.

Meng: And yeah, the computational cost of RAPO’s two forward passes per rollout sample is something that has to be figured out before I can think about deploying it at massive scale.

Tom: Still a hurdle for production, but the fidelity gains they showed on CoSER and CharacterBench are definitely worth exploring.

Jane: Absolutely. It shows that by focusing on the psychology behind the character’s response, we unlock a much deeper level of interaction between the AI and whatever it's talking to.

Lu: This work sets a high bar for how we should approach alignment in these kinds of generative tasks moving forward.

Tom: Definitely interesting stuff. Next up, we’re going to look at some papers dealing with how models handle misaligned character behavior when they're trained on specific data structures, which is a different kind of problem altogether.

More episodes

← Home