CALM-IT: Enhancing Synthetic Motivational Interviewing Dialogue Quality with Conversational Dynamics

arXiv:2601.10085 · cs.CL · Submitted 2026-01-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "CALM-IT: Enhancing Synthetic Motivational Interviewing Dialogue Quality with Conversational Dynamics".

Jane: As a fastidious and diligent researcher, I have meticulously analyzed both provided excerpts (A, B, and E) pertaining to the paper "CALM-IT:

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper CALM-IT, which is about improving how AI creates long-form motivational interviewing dialogues by tracking the changing states of both the client and the therapist Jane and that's what they claim to do in this paper.

Jane: Right, so the main idea is that current LLMs often fail because they don't track things like rapport or motivation over many turns, which means interventions might happen at bad times or goals might get settled too soon Tom so the thesis of CALM-IT is that explicitly modeling these evolving client and counselor states helps guide both how the AI chooses a strategy and what it says next Jane which is important because as we see in this paper, therapy isn't just a string of isolated responses, but goals evolve over time Tom

Lu: Exactly, they introduce this dual-actor framework where both agents maintain latent variables like rapport and inferred mental state and motivation and goals Jane those variables govern the strategy selection and response generation which is what Kim et al. 2025a pointed out as missing in previous work Tom <ref:2601.10085#pg2>

Meng: But how does it actually handle that tracking without just guessing or hallucinating the feeling of rapport or anxiety?

Lu: Well, they ground this in MI theory so it has a solid foundation for modeling those relational and motivational aspects Meng they use these dynamics to see how the broader relational landscape shifts over time which is what Kim et al. 2025a noted as necessary for replicating real-world failures like premature convergence Jane <ref:2601.10085#pg2>

Lalam: I’m seeing that in the transcript examples on page three you can actually see them tracking things like rapport and emotion and change state right before the inference step Tom it shows them inferring client relevant background cognitive states and then generating an utterance based on all that data Jane <ref:2601.10085#pg1>

Jane: That looks like they are successfully modeling that interaction, but the paper also talks about how they test this system Tom they evaluate CALM-IT on a large scale corpus of eight thousand two hundred thirty-two synthetic dialogues spanning multiple lengths and frameworks Lu so it's not just theoretical stuff; it's been tested on a big set of data <ref:2601.10085#pg1,evaluate CALM-IT on a large scale corpus of 8,232 synthetic>.

Paper summary: Tom: And the results are pretty strong because they compared it against all the baselines and CALM-IT achieved the best performance on the majority of MITI four point two global ratings Jane that’s a big number when you compare it to other models Tom

Jane: It also shows stability across dialogue lengths, which is pretty crucial because long conversations are where these dynamic tracking systems really show their worth Lu they found a minimal performance degradation when comparing short-form dialogues to longer ones like those generated using the OLMo-three point one-32B model Tom

Meng: But what about deployment? If I'm trying to run this on real traffic, how much overhead does this tracking cost compared to just using a standard LLM like DeepSeek-V3 point 2 <ref:2601.10085#pg1>?

Lu: The paper mentions that CALM-IT required an average wall-clock time of eight minutes per transcript, which is faster than CAMI+STAR's ten minutes and quicker than KMI's two minutes Tom

Jane: That speed is good, but the real win they point to in terms of client experience is the simulated client redirection acceptance rate, which hit sixty-four point three percent across different dialogue length conditions Lu that’s a high number for how well it handles steering the conversation when things get tricky Tom

Lalam: And they specifically found that their strongest redirections were followed by increased change talk and reduced sustain talk Jane so the dynamics tracking actually leads to better therapeutic outcomes in terms of what kind of dialogue happens Lu

Tom: We also saw component ablation experiments where removing the conversational dynamics components actually reduced dialogue quality, which proves those tracking elements are necessary for the performance boost Jane it’s not just a fancy layer on top, it’s essential to how it works Lu

Jane: So what does this mean for someone who just listens to this show? It means that when we talk about AI generating therapy, we need to be more careful because we need systems that actually understand the flow and the feeling behind a long conversation Tom

Paper summary: Lu: The implication is that if you want synthetic dialogue to actually model real-world therapeutic interactions well, you have to move beyond just sequence prediction and start modeling these complex interpersonal dynamics explicitly Jane it suggests future work should focus on making these dynamic components even more robust and generalizable for different types of therapy Tom

Meng: For me, the limitation they mention is that while they track things well, the authors themselves flag that the system still has to be fine-tuned for specific MI frameworks Jane so it’s not a plug-and-play solution for every single counseling scenario right now Lu

Tom: So CALM-IT is a solid framework because it shows that by explicitly modeling these dynamics you can get higher quality, more consistent long-form synthetic MI dialogues Jane the authors are definitely pushing for this approach to be the standard way forward in this area Lu

Lalam: I’m confident in its answer based on the evidence presented; it’s showing that explicit state modeling is what unlocks better process fidelity in these kinds of interactions Tom

Jane: So we've seen how CALM-IT uses dual-actor tracking to track rapport and goals across eight thousand two hundred thirty-two dialogues and how it outperforms baselines on MITI ratings while maintaining stability in length Lu that’s what we covered with the summary section Tom <ref:2601.10085#pg1>

Tom: And for the conclusion of this paper CALM-IT is presented as a framework that successfully integrates explicit modeling of complex interpersonal and psychological states into the generative process Jane it’s not just another dialogue tool; it’s about making sure the AI understands how those dynamics change over time Lu

Meng: It suggests that for real-world applications, we need models that can handle this level of relational complexity if they are going to be reliable in a production environment Tom

Jane: So the implication is that we might start expecting synthetic therapeutic dialogues to have a much better sense of trajectory and emotional depth than what we see now Lu as models get better at these dynamics, the quality of those long conversations will improve too Tom

Conclusion: Segment: Conclusion**

Tom: So we've looked at CALM-IT, which is about modeling the client and therapist states to make long motivational interviewing dialogues better Jane but now we're wrapping up by talking about what this paper actually means for us Lu

Jane: Right. To summarize, this framework takes those complex ideas—rapport, emotion, goals—and builds a structure where the AI actively tracks them throughout the conversation Tom it shows that when you explicitly model those dynamics, you get better performance on established therapeutic metrics like MITI ratings across huge sets of synthetic dialogues Lu

Tom: I mean, looking at the numbers here, CALM-IT achieved the highest mean scores on all four global MITI four point two ratings Jane and it showed really small changes in quality when you went from short dialogues to long ones Lu that stability is something I'm usually allergic to in these models

Lu: It’s important because most current synthetic dialogue generators just predict the next word based on what came before, which doesn't account for how rapport or motivation actually shifts over a sixty-turn conversation Tom CALM-IT proves you need that explicit state tracking for long-form interaction fidelity

Jane: So what does this mean if we take this into the real world? It means future AI therapeutic dialogues won't just be technically fluent; they will have a much better sense of trajectory and emotional depth Tom they’ll be steering the conversation more effectively than what we see now

Tom: Exactly. And for someone who only listens to this show, it means you can start expecting synthetic therapy that actually models the process, not just the words Lu but we still have to talk about how much work is left before this hits production pipelines

Jane: Right. Before we look at where this stuff is actually going in terms of deployment and real-world use cases for these dynamic trackers, let's talk about what limitations the authors themselves point out in the paper Lu

Viet Cuong Nguyen, Nhi Yen Nguyen, Kristin A. Candan, Mary Conlon, Vanessa Rumie, Kristen Risola, Michael L. Birnbaum, Munmun De Choudhury

Georgia Institute of Technology · Northwell Health Department of Northwell Health Columbia University

cs.CL

Submitted: 2026-01-15

Updated: 2026-10-13

Comments: 55 pages

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 89/100

The gist: As a fastidious and diligent researcher, I have meticulously analyzed both provided excerpts (A, B, and E) pertaining to the paper "CALM-IT: Enhancing Synthetic Motivational Interviewing Dialogue

Key concepts

Conversational Dynamics Tracking: This involves using a dual-agent system to monitor how rapport, emotion, goals, and client states change throughout a conversation. The framework explicitly models these evolving properties for both the client and the therapist to inform strategic decisions during dialogue generation.
Motivational Interviewing (MI): MI is a counseling approach focused on eliciting change from the client through collaborative communication. CALM-IT is specifically designed to generate synthetic MI dialogues, ensuring that the generated interactions adhere to established MI theory by tracking motivational factors like readiness and goals.
MITI Ratings: Motivational Interviewing Treatment Integrity (MITI) is a rigorous evaluation code used to score the quality of therapeutic dialogues. CALM-IT achieved the highest mean scores on these ratings, indicating that its generated conversations are judged as more effective and therapeutically sound by human experts.
Dual-Agent Framework: CALM-IT uses two interconnected agents—one representing the client and one representing the therapist. These agents jointly track their respective states (e.g., rapport, motivation) to ensure that the generated responses are contextually relevant and strategically aligned with therapeutic goals.

Summary

As a fastidious and diligent researcher, I have meticulously analyzed both provided excerpts (A, B, and E) pertaining to the paper CALM-IT: Enhancing Synthetic Motivational Interviewing Dialogue Quality with Conversational Dynamics. My synthesis below aims to construct a comprehensive, detailed summary that captures the core methodology, key findings, comparative performance metrics, and technical specifics of the framework.

Comprehensive Research Summary: CALM-IT Framework for Long-Form Motivational Interviewing Dialogue Generation and Evaluation

The paper introduces CALM-IT, a novel framework designed to significantly enhance the quality of long-form Motivational Interviewing (MI) dialogues by explicitly modeling the evolving states of both the client and the counselor. The central innovation lies in its dual-actor conversational dynamics tracking, which incorporates latent variables such as rapport, inferred mental state (including background, emotion, and stage of change), motivation levels, and goals for both agents. This explicit modeling guides crucial aspects of counseling strategy selection and utterance generation.

Methodology and Architecture

CALM-IT is explicitly grounded in established MI theory to provide a theoretically sound foundation for its relational, motivational, and emotional modeling capabilities. The architecture is characterized by tracking interpersonal alignment via rapport, performing mental-state inference, and jointly tracking activity structure toward stated goals.

The framework's evaluation process is rigorous and multi-faceted:

Dialogue Evaluation: It utilizes metrics derived from the Motivational Interviewing Treatment Integrity (MITI) code, evaluating performance at turn-, agent-, and conversation-level granularities.
Synthetic Corpus: CALM-IT was rigorously evaluated on a large-scale synthetic corpus comprising 8,232 dialogues spanning various lengths and MI frameworks.
Reproducibility: The work releases a complete generation framework, a process-level evaluation protocol, and the large synthetic corpus to facilitate further study of therapeutic LLMs under realistic long-form interaction conditions.

Key Findings and Performance Analysis

The empirical results demonstrate that CALM-IT establishes a new state of performance in generating high-quality MI dialogues:

Superior Overall Performance: Compared against all baselines, CALM-IT achieves the best performance on the majority of MITI 4.2 global ratings.
Stability Across Length: A critical strength highlighted is its minimal performance degradation as dialogue length increases, contrasting with other methods. This stability is further supported by Figure D1, which shows positive or near-zero percentage change in performance across metrics when comparing short-form (30 turns) to long-form (100 turns) dialogues generated using the OLMo-3.1-32B model.
Client Engagement: Despite initiating fewer change-directed prompts than some competitors, CALM-IT achieves the highest client acceptance rate, averaging 64.3% across different dialogue length conditions.
Therapeutic Efficacy: The framework produces more effective therapist redirection than all baseline frameworks tested. This is evidenced by larger and more consistent post-redirection reductions in client sustain talk and an increase in change talk, indicating superior therapeutic outcomes.

Comparative Analysis and Ablation Studies

The study benchmarks CALM-IT against several established baselines, including KMI, CAMI+STAR (C+S), and a version of CALM-IT without conversational dynamics (CI-NC).

Metric Comparison (Table D3): Table D3 provides turn-, agent-, and conversation-level evaluation scores. The results consistently show that CALM-IT outperforms all baselines across most measured metrics.
Component Contribution (Table D4): A component ablation analysis, conducted using DeepSeek-V3.2, isolates the specific contribution of the conversational mechanisms within CALM-IT, providing insight into which modeled dynamics (e.g., rapport tracking vs. mental state inference) are most critical to its success.

Technical Specifications and Computational Details

The experimental setup provides necessary context for reproducibility:

Model Usage: Dialogue generation was performed using DeepSeek-V3.2 without fine-tuning, with inference executed via sglang. Decoding parameters were kept constant across all tested frameworks.
Runtime Efficiency (Appendix E.2): The computational overhead varies by framework; for instance, CAMI+STAR required an average wall-clock time of approximately 10 minutes per transcript, while CALM-IT was faster at 8 minutes per transcript, and KMI was significantly quicker at only 2 minutes.
Compute Resources (Appendix E.3): All experiments were executed on nodes equipped with 8×H200 GPUs, resulting in a total compute usage of approximately 1650 GPU-hours across all transcripts.

In conclusion, CALM-IT is presented not merely as another dialogue generation tool, but as a theoretically grounded framework that successfully integrates explicit modeling of complex interpersonal and psychological states into the generative process. Its demonstrated superior performance, stability in long-form interactions, and enhanced therapeutic efficacy position it as a significant advancement for developing sophisticated therapeutic LLMs.

Improvements for AI systems

The CALM-IT framework explicitly models evolving client and counselor states, guiding both counseling strategy selection and utterance generation, which allows systems to avoid premature convergence by using markers like rapport (r) and stage of change (cˆ) as gates for strategy selection.
CALM-IT generates long-form dialogues by maintaining latent variables, including rapport, inferred mental state, motivation, and goals, enabling the agent to condition its responses on a jointly evolving state rather than isolated turn cues.
The system can perform process-level evaluation using metrics grounded in MI theory (MITI 4.2), specifically measuring directionality, goal alignment, and redirection effectiveness across multi-turn trajectories to assess how dialogue progresses over time.

Abstract

Conversational dynamics describe how rapport, emotion, and goals change over a dialogue. These changes are important in therapeutic conversations, where they can influence when and how therapists intervene. However, large language model (LLM)-based synthetic therapeutic dialogue generators rarely track how these properties evolve together, which may limit the quality of longer conversations. We introduce CALM-IT, a dual-agent framework that tracks client states and therapist-side inferences to guide strategy selection and utterance generation. We evaluate synthetic Motivational Interviewing (MI) dialogues using automated Motivational Interviewing Treatment Integrity (MITI) ratings, conversational metrics, and comparisons across dialogue lengths, with ratings from four licensed psychologists providing support for key findings. Across 8,232 dialogues covering 686 grounded scenarios, four frameworks, and three target lengths, CALM-IT achieves the highest mean scores on all four MITI 4.2 global ratings and shows the smallest aggregate change in quality between short- and long-form conditions. CALM-IT also achieves the highest simulated client redirection acceptance rate (64.3%), with its strongest redirections followed by increased change talk and reduced sustain talk. Ablation experiments show that removing conversational dynamics components reduces dialogue quality. Together, these results suggest that explicitly modeling conversational dynamics can improve the quality and consistency of long-form synthetic MI dialogues.