Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries

arXiv:2607.16989 · cs.CL, cs.AI, cs.DL, cs.HC · Submitted 2026-07-18 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries".

Jane: Here's the gist of the paper. An artificial intelligence (AI) agent can serve as the first-pass author of Clinical and Translational Science Award (CTSA) KL2/K12 scholars’ impact summaries, shifting staff effort from manual assembly to review The Core Problem Addressed Assembling a scholar’s record by hand takes staff an estimated 15 hours and does not scale to a full cohort The effort grows with every new cohort because tracking continues for years after each award ends Non-scholarly evidence such as clinical-program leadership, community engagement, cost or adoption data, and policy uptake sits outside every system that current tools draw on Writing the one-sentence Translational Science Benefits Model (TSBM) impact summaries involves reading across publications, grants, and all four TSBM domains without a tool to perform that synthesis.

Tom: Agent Architecture and Operation The agent is designed as a system in which a language model dynamically directs its own process and tool use It starts from a scholar seed containing the name, institution, and any grant records the hub already holds The agent organizes its work as a research run, which has three stages: gather stage collects the evidence, assemble stage writes it into dossier sections and impact summaries, and critique stage flags what to repair.

Jane: A two-model design processes what the gather stage retrieves, using a large model (gpt-5.1) to plan the searches and draft summaries, while a small model (gpt-5.4-mini) handles high-volume reading subtasks like distilling web pages into facts.

Tom: Workflow Integration and Evaluation The agent is evaluated in the impact-reporting workflow of one CTSA hub across 10 career-development (KL2/K12) scholars In the review study, two evaluation staff members independently reviewed the agentgenerated dossier and impact summaries for each scholar using a structured workbook.

Title and authors: Tom: So we're talking about this paper called "Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries." Basically, they’re looking at how an artificial intelligence agent can help staff with those huge impact summaries for Clinical and Translational Science Award scholars.

Jane: Right. It tackles the problem that assembling these records manually takes staff around fifteen hours per scholar, and that effort just doesn't scale when you have a whole cohort of scholars to track over years.

Lu: The core idea is using an AI agent to do the heavy lifting—gathering evidence from scholarly databases and the open web—and then drafting those one-sentence Translational Science Benefits Model summaries for staff to review.

Meng: So, it’s about shifting that massive manual assembly effort away from staff and onto an AI system that can handle the volume. That sounds like something we need to talk about when we're thinking about scaling up our own impact reporting tools.

Lalam: The paper shows this agent starts with just a scholar seed—the name, institution, and any existing grant records—and then it organizes its work into three stages: gathering evidence, assembling the dossier sections and summaries, and finally critiquing what needs to be repaired.

Tom: That sounds like a solid workflow. But how does the AI actually know what to look for? It doesn't just randomly search everything on the internet.

Jane: Well, they designed it with a two-model system; they use one large model to plan the searches and draft summaries, while a smaller model handles the high-volume reading subtasks, like distilling web pages into facts.

Lu: That separation is smart for efficiency. They also focused on how the agent builds its knowledge base by using source preferences to guide every stage of the pipeline, like prioritizing a scholar’s own page or news coverage over other sources.

Meng: That sounds complicated to get right at scale. What happens when the evidence isn't from a standard database? The paper mentions that non-scholarly evidence, like community engagement data or policy uptake, sits outside every system current tools draw on.

Lalam: Exactly. The agent was tested across all four Translational Science Benefits Model domains—clinical, community, economic, and policy—and it found evidence in all of them. It specifically surfaced things that closed sources miss, like clinical-program leadership or cost data.

Tom: That’s interesting because those non-scholarly pieces are often the hardest to find without manual digging. What were the results on how usable this output was when real staff tried it out?

Title and authors: Jane: They put it in a review study where two evaluation staff members independently reviewed the agent-generated dossier and impact summaries using a structured workbook. The primary measure they used was the unanimous usable rate, which is defined as the share both reviewers accepted or edited.

Lu: And that result was quite positive: both reviewers accepted or edited eighty-one point seven percent of the agent’s findings in that review study <ref:2607.16989#pg1,both reviewers accepted or edited 81.7>. They also rated the synthesis accuracy as four point five and usefulness as four point eight on a five-point scale, which suggests it’s actually pretty good at putting things together.

Meng: Eighty-one percent usable is promising, but we need to talk about the failures. What was the biggest hurdle when staff were checking this out?

Lalam: The largest group cited sourcing failures as the reason for rejection. A weak or non-authoritative source was actually the largest share among findings that weren't usable at all. Some of them gave no source at all, and others pointed to broken or mismatched links in their records.

Tom: So, it seems like the agent’s retrieval is close to human search recall, with a score of zero point eight two for both the human-only and AI-guided searches across thirty three profiles they tested. That’s not perfect recall, though.

Jane: It's not perfect, but the fact that seventy five percent of all findings were accepted unchanged by both reviewers suggests that most of the initial draft was usable as a first draft for staff to work from.

Lu: The paper also points out some specific areas where they had issues with coverage, like economic and policy impact data. They suggest wiring the agent’s retrieval mechanism directly to databases for patent records from the United States Patent and Trademark Office and policy-citation indexes like Overton to ground those domains where current coverage is thinnest.

Meng: From an engineering standpoint, we need to talk about how this runs at scale. The paper notes that the agent issues between nine and twenty five distinct search queries per scholar and makes four hundred thirteen tool calls in aggregate for each run, which shows it’s pretty resource intensive.

Lalam: They also mentioned a specific improvement they suggested: implementing a rule-based filter to flag those sourcing failures retrospectively, like flagging those twenty one findings out of five hundred seven that had an empty source field.

Title and authors: Tom: So, the agent itself isn't perfect at finding the right source, but it’s getting better by incorporating these feedback loops. What about the future direction? Where does this leave us?

Jane: The main implication they draw is a relocation of effort: shifting staff away from an estimated fifteen hours of manual assembly per scholar toward just minutes of reviewing what the AI has already drafted. That’s a big time saving for those reporting workflows.

Lu: Future work involves multi-site evaluation, trying to improve coverage in economic and policy impact, and finding ways to keep each scholar’s record current at a lower operational cost so this doesn't just become a one-off study.

Meng: We need to see if we can make that process cheaper than the manual effort it replaces. The paper surfaces evidence that closed sources miss, which is exactly what we need to target with our next iteration of model training data.

Lalam: It really suggests that the gains come from this human-in-the-loop approach, where staff use the agent’s output as a starting point rather than creating it from scratch.

Tom: So, to wrap up on "Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries," they show that an AI agent can serve as a first-pass author for these impact summaries in the CTSA reporting workflow.

Jane: In real terms, staff found most of the output usable, meaning the effort shifts from collecting and writing to reviewing what’s already done. It’s a powerful tool for scaling that kind of documentation.

Lu: The agents' capabilities are limited to the search, database, and page-reading calls described in their methods section. They didn't claim anything beyond what they measured in those specific studies.

Meng: So we see a system where the agent plans searches with a large model, reads the details with a smaller model, and then critiques its own drafts against human-written benefit statements. It’s complex but it has structure.

Lalam: The paper's main finding is that for scholars, this agent provides evidence across all four TSBM domains, even capturing those hard-to-find non-scholarly pieces through its targeted retrieval methods.

Tom: So the takeaway is that staff time can be significantly reduced by using an AI agent as a first pass author for complex impact summaries. It’s about changing the bottleneck from creation to curation.

Jane: That's the gist of it—it’s about making sure we don't miss that important, non-traditional evidence in the process. We're going to keep an eye on how they tackle those coverage gaps next.

The paper's summary: Tom: So we're looking at this paper that tests an AI agent for writing those impact summaries for Clinical and Translational Science Award scholars, and it’s basically showing how to shift staff effort from fifteen hours of manual assembly to just minutes of reviewing what the AI drafts.

Jane: Right. It’s about taking all that messy evidence—publications, grants, community data—and synthesizing it into those one-sentence summaries that need to hit four different impact domains, and seeing if an AI can handle that first pass.

Lu: The agent uses a two-model design where a large model plans the search strategy and drafts the summary ideas, while a smaller model handles the high volume of reading tasks like distilling web pages into facts.

Meng: What I'm curious about is how it actually holds up when you try to put it on production. The paper shows that most of the output was usable as a first draft, with seventy-five percent of all findings accepted unchanged by both reviewers in their study.

Tom: That seventy-five percent is pretty solid, I'll give them that. But the biggest problem they found wasn't necessarily the writing quality; it was sourcing failures dominating the rejections.

Jane: Yeah, a lot of it was rejected because the source was weak or non-authoritative. They saw findings where there was no source at all, and some where the link just didn't work.

Lu: The agent’s recall for finding profiles was about zero point eight two, which is close to what you'd get from a human search, but it wasn't perfect. It found stuff in all four impact domains—clinical, community, economic, and policy—which is pretty good for coverage.

Meng: But that recall isn't perfect because the agent struggles with the non-scholarly stuff like community engagement or policy uptake when they aren't in standard databases. They point out that current tools often miss those kinds of data entirely.

Tom: So what this means practically is it’s not going to be a magic system that just spits out perfect, ready-to-publish summaries without any human eyes looking at it.

Jane: Exactly. It's a tool for relocation of effort. The study suggests the bottleneck shifts from the time spent collecting and writing everything, to the time spent reviewing and curating what the AI produces.

Lu: They are also pushing improvements on how it handles those gaps in coverage, suggesting that wiring it directly to databases for things like patent records or policy indexes could ground those economic and policy sections better.

Tom: So, the future work is about making sure this agent doesn't just stop at a first draft but can be tuned to pull in that missing non-traditional evidence more reliably.

Jane: It really highlights that the real value here isn't just in the writing itself, but in how it helps staff manage the massive workload of tracking scholar records over many years.

The paper's improvements: Tom: Okay, so we're talking about what they suggest to make this agent better after they did that initial evaluation, which was pretty telling on where it failed—specifically with those sourcing errors.

Jane: They’re suggesting a few major fixes: first, using an evaluator optimizer pattern. That means if the agent finds a problem, it shouldn't just give up; it should repeat the run up to three times until it actually solves the issue instead of just stopping early.

Lu: I like that idea. It forces the agent to use its whole budget, not just converge on an easy answer and quit. That kind of iterative refinement is key for complex synthesis tasks.

Meng: From a deployment standpoint, I need to know how much computational overhead this extra repetition adds when you're running it against thousands of scholars in a real workflow. If the latency goes up too high, it’s useless for staff who are already under pressure.

Tom: Right, Meng’s right about the cost. It needs to be efficient enough that we don't end up with a system that takes ten minutes to run a single summary.

Jane: They also talked about better context management for those long records, which means using that smaller model—the one you mentioned earlier—to handle the heavy lifting of reading and trimming abstracts so the main large model doesn't get bogged down in too much noise.

Lu: That makes sense. Confining the high-volume reading to a specialized sub-model should reduce context poisoning, where irrelevant text from a hundred different papers confuses the main planning model.

Tom: So it’s about specialization, right? One model for planning, one for high-speed reading. It's a two-pronged approach to scaling the system.

Jane: And then they are pushing source preference enforcement even harder—making sure every stage of the pipeline strictly follows rules like using a Digital Object Identifier or your hub's own record when looking for specific types of evidence.

Meng: That’s good. If you can enforce those authoritative sources earlier in the chain, you catch the bad data before it becomes part of the final summary draft, which saves cleanup time later.

Lu: Plus, they are calling for a rule-based filter to flag those source failures retrospectively. It’s like building a quality assurance layer on top of the agent's output after it’s done drafting.

Tom: A retroactive filter sounds practical for catching those specific types of errors that slip through the initial search plan. It’s about fixing the output, not just making a better plan upfront.

Jane: So these improvements focus on making the agent more robust—more iterative, more context-aware, and much stricter about where it gets its facts from.

Lu: The ambition is clearly to move this from a promising prototype to something that can handle the messiness of real-world impact reporting without constant human correction.

Tom: So the big idea is that we’re moving toward an AI system that doesn't just try once and fails, but keeps trying smarter until it gets it right.

Jane: And this leads us to thinking about where those gaps are still—like the economic data they mentioned earlier—and how we can program in specialized tools for those areas.

Conclusion: Tom: So to wrap up on "Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries," we see that this agent can significantly reduce the time staff spend assembling those impact summaries for CTSA scholars by shifting the focus to review rather than initial writing.

Jane: It’s a practical win because staff found most of the output usable, which means they aren't starting from scratch every time. The main implication is that we can scale up impact reporting without needing a massive increase in HR hours for documentation.

Lu: Yeah, I think the real shift is recognizing that the agent handles the heavy lifting of cross-referencing evidence across all those different domains, which is something humans struggle to do efficiently on a large scale.

Meng: From my side, what’s important is that while it's a good first pass, we still have to be careful about what breaks at scale; the system needs to be cheap and fast enough for real production use.

Lalam: I'm confident in this result because the agent successfully found evidence across all four Translational Science Benefits Model domains, including those non-scholarly pieces that closed sources often miss.

Tom: So yeah, it’s a tool for making sure we don't miss that critical evidence that sits outside the traditional academic silos.

Jane: It really shows how an AI agent can be a force multiplier for researchers who are drowning in documentation and reporting requirements.

Lu: The future work they’re pointing toward is about getting it even better at those specific gaps, like integrating patent records or policy citations more directly into the retrieval process.

Meng: I want to see that integration happen because if the AI can reliably pull in that economic and policy data, then it moves from a helpful draft tool to a genuinely useful reporting engine.

Tom: It’s about taking this agent concept, testing it rigorously across different domains, and then figuring out the engineering roadmap to make those specific integrations work on production traffic.

Jane: That’s what we’ll keep an eye on—how they tackle those coverage gaps next with more targeted database integration.

University of Illinois Chicago

cs.CL, cs.AI, cs.DL, cs.HC

Submitted: 2026-07-18

Updated: 2026-10-29

Comments: 15 pages, 5 figures, 2 tables. Submitted to the Journal of Clinical and Translational Science. Code: https://github.com/mo-arvan/scholar-dossier-agent

Code: https://github.com/mo-arvan/scholar-dossier-agent

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 100/100

The gist: An artificial intelligence (AI) agent can serve as the first-pass author of Clinical and Translational Science Award (CTSA) KL2/K12 scholars’ impact summaries, shifting staff effort from manual

Key concepts

Agent Architecture: The AI is designed to dynamically direct its own process through distinct stages: gather, assemble, and critique. It uses a two-model design where a large model plans searches and drafts summaries, while a smaller model handles high-volume reading tasks like distilling web pages into facts.
Research Run Structure: The agent operates as a structured research run involving multiple stages. This includes gathering evidence, assembling the content into dossier sections, and critiquing the output. The architecture is designed to allow for dynamic planning and iterative refinement based on critiques.
Translational Science Benefits Model (TSBM): This is a framework used to categorize impact across four domains: clinical, community, policy, and economic. The agent successfully surfaced evidence across all these domains, showing it can synthesize information from diverse sources like journals and institutional pages.
Human-in-the-Loop Evaluation: Staff members independently reviewed the AI's generated summaries using a structured workbook. The primary measure of success was the 'unanimous usable rate,' indicating that most of the agent's findings were accepted or edited by reviewers, validating its utility as a first draft.

Terminology

Translational Science Benefits Model — 13× in this paper
Clinical and Translational Science Award — 10× in this paper
Artificial Intelligence (AI) — 9× in this paper · explained in 2 episodes
Human-in-the-loop — 9× in this paper · explained in 4 episodes
Accuracy — 7× in this paper · explained in 1 episode
Core — 6× in this paper · explained in 2 episodes
Papers — 6× in this paper · explained in 3 episodes
Division of Labor — 4× in this paper · explained in 2 episodes
Generative AI — 3× in this paper · explained in 2 episodes
Indirect Prompt Injection — 3× in this paper · explained in 2 episodes
Editor — 2× in this paper · explained in 2 episodes
Hallucination — 2× in this paper · explained in 4 episodes
National Center for Advancing Translational Sciences — 2× in this paper
National Institutes of Health — 2× in this paper
Trace — 2× in this paper · explained in 2 episodes
AI Agents — 1× in this paper · explained in 2 episodes
Adam — 1× in this paper · explained in 3 episodes
Agent Architecture — 1× in this paper · explained in 2 episodes
Agentic System — 1× in this paper · explained in 2 episodes
Computational overhead — 1× in this paper · explained in 2 episodes
Distillation — 1× in this paper · explained in 2 episodes
Feedback Loop — 1× in this paper · explained in 2 episodes
Iterative Refinement — 1× in this paper · explained in 5 episodes
Prompt Injection Attacks — 1× in this paper · explained in 2 episodes
Pruning — 1× in this paper · explained in 4 episodes
Topic Modeling — 1× in this paper · explained in 2 episodes
Verification — 1× in this paper · explained in 2 episodes

Summary

An artificial intelligence (AI) agent can serve as the first-pass author of Clinical and Translational Science Award (CTSA) KL2/K12 scholars’ impact summaries, shifting staff effort from manual assembly to review

The Core Problem Addressed

Assembling a scholar’s record by hand takes staff an estimated 15 hours and does not scale to a full cohort The effort grows with every new cohort because tracking continues for years after each award ends Non-scholarly evidence such as clinical-program leadership, community engagement, cost or adoption data, and policy uptake sits outside every system that current tools draw on Writing the one-sentence Translational Science Benefits Model (TSBM) impact summaries involves reading across publications, grants, and all four TSBM domains without a tool to perform that synthesis.

Agent Architecture and Operation

The agent is designed as a system in which a language model dynamically directs its own process and tool use It starts from a scholar seed containing the name, institution, and any grant records the hub already holds The agent organizes its work as a research run, which has three stages: gather stage collects the evidence, assemble stage writes it into dossier sections and impact summaries, and critique stage flags what to repair. A two-model design processes what the gather stage retrieves, using a large model (gpt-5.1) to plan the searches and draft summaries, while a small model (gpt-5.4-mini) handles high-volume reading subtasks like distilling web pages into facts.

Workflow Integration and Evaluation

The agent is evaluated in the impact-reporting workflow of one CTSA hub across 10 career-development (KL2/K12) scholars In the review study, two evaluation staff members independently reviewed the agentgenerated dossier and impact summaries for each scholar using a structured workbook. The primary measure was the unanimous usable rate, defined as the share both accepted or edited. Both reviewers accepted or edited 81.7% of the agent’s findings Reviewers rated synthesis accuracy 4.5 and usefulness 4.8 on a 5-point scale.

Key Findings and Limitations

The agent’s profile discovery study found its recall close to human search, with recall being 0.82 for both the human-only and AI-guided search across 33 profiles. The review study showed that most of the agent’s output was usable as a first draft, with 75% of all findings accepted unchanged by both reviewers. Sourcing failures dominated rejections, with a weak or non-authoritative source being the largest share among not-usable findings. Limitations include the sample size of 10 scholars and the fact that performance is not site-specific.

Conclusion and Future Direction

A human-in-the-loop artificial intelligence (AI) agent can serve as a first-pass author of Clinical and Translational Science Award (CTSA) KL2/K12 scholars’ impact summaries The main implication is a relocation of effort, shifting staff from an estimated 15 hours of assembling each scholar’s record to minutes of reviewing it. Future work includes multi-site evaluation, strategies to improve economic and policy impact coverage, and efficiency optimizations that keep each scholar’s record current at lower cost. The agent surfaces evidence that closed sources miss and hands-on processes cannot afford.

Pruning redundant reasoning tokens

The agent issues 9 to 25 distinct search queries per scholar and makes 413 tool calls in aggregate The agent’s nested stages include gather, assemble, critique, and register-revise. The register-revise stage measures each drafted summary against a linguistic signature built from 76 source-grounded, human-written benefit statements.

How the training signal is built

The agent uses a two-model design where the large model plans searches and drafts summaries, and a small model handles high-volume reading subtasks Source preferences guide every stage of the pipeline, such as using the scholar’s own page or a news outlet’s own coverage for profile mentions. The agent also manages its own context, condensing accumulated findings into the dossier as it works.

Where the gains come from

The time saved rests on a staff estimate of manual effort rather than a measured comparison within this study, although the estimate is consistent with published per-profile effort at another hub The agent avoids both costs by retrieving evidence through open-web search as well as from indexed databases. Staff review becomes the bottleneck, and the workbook showed only what aids the judgment, each finding with its source and the agent’s reasoning.

Sourcing failures dominated rejections

The largest group cited a non-authoritative source, such as a provider-directory page offered as a profile (weak source, 33). A further eight gave no source at all and four pointed to broken or mismatched links. The reviewer’s practical work was therefore curation, judging whether each finding cleared the impact bar, then confirmed that it was attributed to the right person, sourced to the right record, and filed under the right category.

Coverage across the four domains

The agent found impact across all four Translational Science Benefits Model (TSBM) domains, concentrated in clinical and community Clinical impact drew mostly on journals and trial registries, while community and policy impact drew more on institutional pages and news coverage. The agent surfaces evidence that closed sources miss, such as clinical-program leadership, community engagement, cost or adoption data, and policy uptake.

Conclusion

A human-in-the-loop artificial intelligence (AI) agent can serve as the first-pass author of Clinical and Translational Science Award (CTSA) KL2/K12 scholars’ impact summaries In a real reporting workflow, staff found most of the output usable. The effort shifted from collecting and writing to reviewing. The evaluation is as much a contribution as the tool.

Acknowledgments

The authors thank Charles Frisbie, Medical Device Labs Director at the University of Illinois Chicago Innovation Center, for his support of this work The authors used AI assistants based on Anthropic and OpenAI models to help draft and edit this manuscript. They accessed these tools through the providers’ standard commercial interfaces.

Financial Support

This work was supported by the AI.Health4All Center at the University of Illinois Chicago The project described was supported by the National Center for Advancing Translational Sciences (NCATS), National Institutes of Health, through Grant Award Number UM1TR005438.

Competing Interests

The authors declare none. The authors reviewed and revised all output and are entirely responsible for the scientific content of the paper. The agent’s capabilities are limited to the search, database, and page-reading calls described.

Improvements for AI systems

Dynamic Agent Planning based on Research Run Structure The agent should explicitly follow an evaluatoroptimizer-style pattern for agentic systems by repeating runs up to three times when issues are found, ensuring it uses the full budget rather than converging early.
Context Management for Long Records Model Distillation The system must employ a two-model design where a small model handles high-volume reading subtasks like distilling web pages and trimming abstracts, limiting context poisoning by confining adversarial text to that smaller model.
Source Preference Enforcement Guided Retrieval The agent should strictly adhere to the defined authoritative source preferences for each finding type, such as using the paper’s digital object identifier (DOI) for papers or the hub’s own record for grants, guiding every stage of the pipeline.
Automated Sourcing Failure Mitigation Rule-Based Filtering The system should implement a rule-based filter to flag 21 of the 507 findings, including every rejected finding whose source field was empty retrospectively, and deploy this filter during planning to resolve known non-authoritative domains like provider directories before review.
Targeted Impact Coverage Expansion Interface Integration The agent's retrieval mechanism must be wired with databases for economic and policy domains, specifically citing patent records from the United States Patent and Trademark Office and policy-citation indexes like Overton to ground those domains where current coverage is thinnest.

Abstract

Introduction. Clinical and Translational Science Award (CTSA) programs must document their scholars' research impact, but assembling each scholar's record by hand takes staff an estimated 15 hours and does not scale to a full cohort. An artificial intelligence (AI) agent could serve as a tool to gather scholar data across platforms and disciplines. Methods. We built a human-in-the-loop AI agent that assembles a dossier of sourced evidence for each scholar and drafts one-sentence Translational Science Benefits Model (TSBM) impact summaries for staff review. We evaluated it in the impact-reporting workflow of one CTSA hub across 10 career-development (KL2/K12) scholars. Two evaluation staff independently coded all 507 findings as accept, edit, or reject; the primary measure was the unanimous usable rate, defined as the share both accepted or edited. Results. Both reviewers accepted or edited 81.7% of the agent's findings. Reviewers each spent a median of 14 minutes per scholar, replacing an estimated 15 hours of manual assembly. Inter-rater agreement was moderate (Cohen's kappa 0.43 on the usable-versus-reject decision). A profile discovery study found the agent's recall close to human search. The agent's impact evidence spanned all four TSBM domains, and about a third of the reviewed findings fell in non-scholarly categories that routine processes tend to miss. Reviewers rated synthesis accuracy 4.5 and usefulness 4.8 on a 5-point scale. Conclusions. A human-in-the-loop AI agent can serve as the first-pass author of a scholar's impact record, shifting staff from collecting and writing to reviewing, and making cohort-scale impact reporting feasible.