Automated Survey Collection with LLM-based Conversational Agents

The gist The study introduces an end-to-end survey collection framework driven by conversational Large Language Models (LLMs) to overcome the limitations of traditional, costly, and labor-intensive phone surveys in biomedical research;.

Objective

The objective of the proposed framework is to create an end-to-end survey collection system driven by conversational LLMs to address the high costs and labor intensity associated with traditional phone surveys in biomedical and healthcare applications;. This approach aims to reduce the workload on human interviewers while enhancing the accuracy and reliability of large-scale data collection through AI-powered methods;.

Materials and Methods

The framework consists of five main components: a researcher responsible for designing the survey, a conversational phone agent powered by an LLM that calls participants, a second LLM (GPT-4o) that analyzes conversation transcripts, and a database for storing and organizing the results;. The study utilized the Covid Impact Survey as the existing survey format to validate the framework;.

The system overview involves preparing a survey, instantiating REDCap for secure data management, collecting participant phone numbers, and creating an instruction prompt for the AI-based phone agent;. This prompt is then provided to an LLM-based agent developed by BlandAI which calls participants and stores conversation transcripts and audio recordings;. Subsequently, the transcripts are retrieved and prompted to GPT-4o to analyze the conversations and deduce answers to each survey question;. Finally, the LLM outputs are processed and uploaded into REDCap;.

Study Details

The study recruited 8 participants, consisting of 5 native and 3 non-native English speakers, and administered 40 surveys in total;. Each participant completed the survey five times before each session;. To simulate survey collection and evaluate data quality, fictitious personas were generated using GPT-4o based on probabilistic selection strategies;. These personas were created through a two-step process: fictitious survey synthesis followed by fictitious persona synthesis;.

The persona generation involved three sampling strategies for creating the fictitious survey, including uniform sampling, sampling based on probabilities derived from real survey response distributions, and using actual respondent data to maintain logical consistency among related questions;. Participants were provided with a description of an imaginary persona containing all necessary details to answer the survey questions before speaking with the AI phone agent;.

Evaluations

The evaluation focused on three key aspects: LLM phone agent’s transcription of conversations from audio to text, GPT-4o’s performance in deducing survey responses from transcripts, and the usability of the phone agent and participant experience;.

Transcription accuracy was measured by calculating the Word Error Rate (WER) using the formula S + D + I / N to assess alignment between Bland-generated and correct reference transcriptions;. GPT-4o’s performance in response deduction was assessed by repeating the inference process five times per transcript and selecting the most common answer using a self-consistency prompting approach;.

Participant experience was gauged through a post-study questionnaire adapted from the Chatbot Usability Questionnaire (CUQ) to assess personality, intelligence, error handling, and navigation;.

Results

The resulting WER scores showed an average WER of 7.7% across all participants, with mean error rates for non-native speakers being slightly larger (9.6%) than for native speakers (6.4%);. Crucially, the reasoning LLM (GPT-4o) achieved an average response deduction accuracy of 98% from conversation transcripts across eight participants, despite the audio transcriptions exhibiting an average word error rate (WER) of 7.7%;.

The results indicated no definitive correlation between Word Error Rate (WER) and accuracy, as participants with high WER in their transcriptions still achieved high accuracy in response deductions;. GPT-4o deduced fully correct answers for about half of the questions and achieved word-level accuracy above 97% for most questions;.

Discussion

The study found that using AI agents to conduct and analyze health-related phone surveys was highly effective, demonstrating real-world feasibility for end-to-end AI-powered survey collection;. The framework showed potential to reduce interviewer bias by acting as an objective surveyor;.

The framework holds significant potential for scaling and efficiency at less cost across significantly more participants of greater diversity than traditional methods;. The total costs to conduct the experiments were only about 75 cents per survey;.

Limitations

Limitations included interactions with the AI agent feeling less natural than if participants had been sharing their own experiences, and some participants paused during calls to search for information in the persona description;. The phone agent sometimes failed to enforce the answer format, leading to ambiguous responses, and GPT-4o made mistakes when handling ambiguous information or non-matching responses;.

The study concludes that this LLM-based framework is highly effective for conducting phone surveys in healthcare and biomedical research;. Future plans include adding automatic adaptation of available survey instruments in Phen-X and improving performance with better LLMs;.

Conclusion

This study introduces an LLM-based framework for conducting phone surveys in healthcare and biomedical research, demonstrating 98% accuracy in identifying survey responses correctly;. The framework is envisioned as a complementary tool to existing methods, with the AI agent having the potential to reduce interviewer bias and simplify operations;. This approach holds great potential in widening participation opportunities and simplifying operations for research teams;. The total costs to conduct the experiments were only about 75 cents per survey;.

Acknowledgments

This research was supported by the Intramural Research Program of the National Human Genome Research Institute, National Institutes of Health;.

Author Contributions Statement

KK is the primary developer of the framework and wrote the majority of the manuscript;. NJD conducted the analysis and annotation of Bland-generated audio and transcripts, while KK handled all other analysis and annotation of the results;. All authors contributed to data interpretation, manuscript revisions, and provided significant intellectual input to enhance the manuscript;.

Conflict of Interests Statement

None declared;.

References

[1] Shaw M, Dent J, Beebe T, Junghard O, Wiklund I, Lind T, et al. The Reflux Disease Questionnaire: a measure for assessment of treatment response in clinical trials. Health and quality of life outcomes. 2008;6:1-6.

[2] Jenkinson C, Burton JS, Cartwright J, Magee H, Hall I, Alcock C, et al. Patient attitudes to clinical trials: development of a questionnaire and results from asthma and cancer patients. Health Expectations. 2005;8(3):244-52.

1. textbf Real-time Contextual Adaptation for Agent Persona Adaptation: The system should incorporate feedback from post-survey questionnaires to dynamically adjust its conversational style to be more empathetic and less robotic, addressing participant feedback like The AI agent did not start with a warm greeting or ask me how I was doing.

2. textbf Enhanced Error Handling for Ambiguity and Format Enforcement: Improve the agent's ability to handle ambiguous or non-matching responses by explicitly instructing the LLM to adhere strictly to predefined data types and response options, mitigating issues like the AI agent sometimes wouldn’t enforce the answer format or misinterpreting words like male as mail.

3. textbf Improved Conversational Flow and Turn-Taking: Implement better turn-taking mechanisms, perhaps using a keypad entry option for complex numerical inputs or allowing the agent to pause and explicitly request clarification when faced with confusing input, addressing participant feedback that the agent struggled with things like fifteen thousand or cut off responses.

4. textbf Contextual Question Pruning: Develop logic to skip irrelevant follow-up questions based on preceding answers, as suggested by participants wanting the system to be more efficient: If I tell the AI agent that I am living on my own it does not need to loop through all questions about the age of the persons living in my household.

5. textbf Enhanced Consistency in Persona Generation: Refine the GPT-4o persona generation process by strengthening constraints based on existing survey data to ensure greater logical consistency, addressing instances where GPT-4o did not choose the correct label for ambiguous answers like Good. Fair when forced into a single category.

Objective: Traditional phone-based surveys are among the most accessible and widely used methods to collect biomedical and healthcare data, however, they are often costly, labor intensive, and difficult to scale effectively. To overcome these limitations, we propose an end-to-end survey collection framework driven by conversational Large Language Models (LLMs). Materials and Methods: Our framework consists of a researcher responsible for designing the survey and recruiting participants, a conversational phone agent powered by an LLM that calls participants and administers the survey, a second LLM (GPT-4o) that analyzes the conversation transcripts generated during the surveys, and a database for storing and organizing the results. To test our framework, we recruited 8 participants consisting of 5 native and 3 non-native english speakers and administered 40 surveys. We evaluated the correctness of LLM-generated conversation transcripts, accuracy of survey responses inferred by GPT-4o and overall participant experience. Results: Survey responses were successfully extracted by GPT-4o from conversation transcripts with an average accuracy of 98% despite transcripts exhibiting an average per-line word error rate of 7.7%. While participants noted occasional errors made by the conversational LLM agent, they reported that the agent effectively conveyed the purpose of the survey, demonstrated good comprehension, and maintained an engaging interaction. Conclusions: Our study highlights the potential of LLM agents in conducting and analyzing phone surveys for healthcare applications. By reducing the workload on human interviewers and offering a scalable solution, this approach paves the way for real-world, end-to-end AI-powered phone survey collection systems.