Do Personalized Skills Help Coding Agents? An Empirical Study of Developer Interaction Histories

This paper investigates whether personalized skills can be distilled from developer-agent interaction histories and whether they improve coding agent performance on subsequent tasks. The authors note that as human developers collaborate with LLM-powered coding agents over time, personalized preferences emerge that could potentially be used to adapt agent behavior. The core research question is: Can we distill personalized skills from developer-agent interaction histories, and do they improve coding agent performance and reduce pushback on subsequent tasks?

The authors propose a two-stage framework for generating personalized agent skills:

Personalized Skill Generation:

1. Bootstrap skill generation: Uses a fixed pattern analysis template to generate a task-independent bootstrap skill from the developer's evolution sessions. The template asks a backbone LLM to identify recurring patterns in communication, work style, follow-up handling, validation preferences, and commit behavior, consolidating them into a structured SKILL.md file while excluding task-specific details like repository names, file paths, and issue identifiers.

2. Evidence-grounded refinement: A second LLM refines the bootstrap skill using the original evolution sessions as evidence. The refiner verifies each candidate rule against the interaction history, retaining, revising, or removing existing rules, meanwhile adding recurring preferences missed during bootstrap generation. To reduce overfitting, the refiner retains a rule only when it is supported by at least two independent user turns from different evolution sessions.

Interactive Session Replay:

For evaluation, the authors design a reproducible replay framework using an LLM-based developer simulator. For each held-out test session, they generate a task summary from developer-authored messages only, then replay the interaction with a trajectory-conditioned developer simulator that reviews the agent's latest response and determines whether it has addressed the developer's explicit requests before issuing follow-ups or terminating.

The experiments use the SWE-chat dataset containing real coding-agent sessions. After strict filtering (removing incomplete interactions, inaccessible repositories, private dependencies, missing files, or no substantive code edits), the dataset contains only 206 sessions from 13 developers. Sessions are split into 80% evolution sets and 20% held-out test sets, resulting in 164 evolution sessions and 42 test sessions.

Four conditions are compared: (A) no skill, (B) the target developer's personalized skill, (C) a personalized skill from another random developer, and (D) a generic skill pooled across all developers. The experiments use Codex with GPT-5.5, with five random seeds for data splitting.

Generic skills provide the largest and most consistent gains. The generic skill achieves the highest average score (68.80), the largest improvement over the no-skill baseline (+3.78), and the highest win rate (50.95%). Although the improvement does not reach the conventional significance threshold (paired t-test, p =.063), the consistently stronger performance suggests that pooling interaction histories across developers is more effective.

Developer-specific skills do not show clear benefit from personalization. Personalized skills yield only a modest improvement of 0.97 over the no-skill baseline, with win and tie rates of only 41.43% and 14.76% respectively, indicating inconsistent improvement. The improvement is not statistically significant (p =.399). Similarly, a skill from another random developer achieves comparable performance (gain of 0.92, win rate 43.33%), showing limited evidence that developer-specific information contributes additional benefits beyond generic procedural guidance.

Impact of relevant interaction history: The effectiveness of personalized skills varies with the number of relevant evolution sessions. "When at least six relevant sessions are available, the personalized skill substantially outperforms both the no-skill baseline (B − A = +10.17) and the random developer skill (B − C = +8.92), and even surpasses the generic skill (B − D = 5.67)." However, with fewer relevant sessions, personalized skills show little advantage.

Developer simulator effectiveness: Among 171 replay instances with substantive follow-ups, 59.65% were exact matches, 29.82% were partial matches, and 10.53% were mismatches, meaning 89.47% were at least partially semantically consistent with real developer messages.

Interaction and execution behavior: Skills generally lead to more extensive execution rather than reduced interaction effort. All skill conditions produce more agent turns, follow-ups, generated tokens, tool calls, code changes, and validation attempts compared to the no-skill baseline. Personalized skills increase tool calls from 8.47 to 9.46 and execution time from 91.76 to 106.16 seconds, while also increasing unresolved-requirement follow-ups from 0.29 to 0.37.

Skill content analysis: The generic skill contains substantially more rules (25 vs. 14.15 on average) and words per skill (383 vs. 236.38) than personalized skills. The generic skill has a larger proportion of commit-related rules (16.0% vs. 7.6%). Personalized skills show substantial lexical variation across developers, with 64.7% of personalized rules unique to a single developer.

The authors conclude that skill conditioning can improve coding agents, but the current benefits arise primarily from reusable guidance shared across developers rather than from developer-specific personalization. They suggest that limited per-developer interaction histories make it difficult to identify stable and transferable developer preferences, though personalization appears more promising when multiple relevant historical examples are available. Future work directions include examining longer interaction histories, exploring other methods for integrating skills beyond prompt inclusion, and developing more adaptive mechanisms for skill utilization that combine task-relevant guidance with developer-specific preferences.

Improvements to AI systems:

1. Adaptive skill selection based on history availability: Implement a routing mechanism that detects the number of relevant historical sessions for a given developer-task pair. If ≥6 relevant sessions exist, inject the developer-specific skill; otherwise, fall back to a pooled generic skill. This prevents performance degradation when personalization data is sparse and maximizes gains when sufficient evidence exists.

2. Evidence-grounded skill refinement with minimum-support thresholds: Build a skill-generation pipeline that requires each candidate rule to be corroborated by at least two independent user interactions from distinct sessions before inclusion. This reduces overfitting to idiosyncratic single-turn behaviors and improves rule reliability.

3. Cross-developer skill pooling with deduplication: Aggregate interaction histories across all developers, then use a clustering or similarity-based deduplication step to merge redundant rules (e.g., commit-message conventions) while preserving rare but high-value rules. This yields a denser, more generalizable skill set than any single developer’s history can provide.

4. Dynamic skill content weighting by rule type: During skill generation, assign higher weight to commit-related and validation-related rules (which showed the largest generic-skill advantage) and lower weight to stylistic preferences that vary widely across individuals. This focuses the skill on behaviors with proven cross-developer utility.

5. Trajectory-conditioned developer simulator for evaluation: Use the replay framework (with exact/partial match scoring) to automatically test agent behavior against simulated developer follow-ups. This enables rapid, reproducible A/B testing of skill variants without requiring live human interaction, accelerating iteration on skill design.

6. Skill-conditional execution budget adjustment: When a skill is active, proactively increase the agent’s tool-call and execution-time budget (e.g., from 8.5 to 9.5 tool calls, 92 to 106 seconds) to accommodate the more thorough validation and commit workflows that skills encourage, while monitoring for diminishing returns.

7. Fallback to generic skill when personalized skill underperforms: Implement a real-time monitor that compares the agent’s task success rate with the personalized skill versus the generic skill on recent tasks. If the personalized skill’s win rate falls below the generic baseline for a rolling window, automatically switch to the generic skill for that developer.

8. Rule-count optimization: Generate skills with a target of 25 rules (matching the generic skill’s density) rather than the average 14 rules in personalized skills. This ensures sufficient coverage of procedural guidance without overwhelming the agent’s context window.

What the improved AI system can do:

• Achieve higher average task success (e.g., +3.78 points over no-skill baseline) by defaulting to pooled generic skills when personalization data is insufficient.

• Surpass generic-skill performance by up to +5.67 points when a developer has ≥6 relevant historical sessions, by switching to their personalized skill.

• Avoid performance cliffs caused by sparse personalization data (previously limited to +0.97 gain) through adaptive selection.

• Generate more reliable skills by filtering out unsupported rules, reducing false personalization.

• Automatically adjust its execution effort (more tool calls, longer runtime) when skills demand thorough validation, improving task completion rates.

• Continuously self-correct by monitoring skill effectiveness and reverting to the best-performing variant per developer.