LLMs for Qualitative Data Analysis Fail on Security-specificComments in Human Experiments

The gist: LLMs for Qualitative Data Analysis Fail on Security-specific Comments in Human Experiments shows that large language models cannot reliably replace human annotators for security-relevant annotations, even with detailed codebooks and conflicting examples.

Background and Problem

Thematic analysis of free-text justifications in human experiments provides significant qualitative insights, yet it is costly because reliable annotations require multiple domain experts <ref:2604.10834#pg4>. Large language models (LLMs) seem ideal candidates to replace human annotators <ref:2604.10834#pg4>. However, coding security-specific aspects (code identifiers mentioned, lines-of-code-mentioned, security keywords mentioned) may require deeper contextual understanding than sentiment classification <ref:2604.10834#pg4>. The research explores whether LLMs can act as automated annotators for technical security comments by human subjects <ref:2604.10834#pg4>.

Methodology and Experimental Setup

The study explored whether LLMs can generate annotations for a fraction of unannotated comments based on a dataset partially annotated by other humans <ref:2604.10834#pg6>. The human annotation procedure involves four steps: (0) Emergence of codes, (a) Annotators individually read some samples and make up their own codes (principle of emergence), (b) At this point, a formal definition of the codebook is agreed <ref:2604.10834#pg6>. Then, steps include (1) First annotation with shared codes (names), which involves annotators getting an initial understanding from the background information and the codebook <ref:2604.10834#pg6>, and (2) Completed annotation with codebook and first review, where annotators review annotations from another annotator <ref:2604.10834#pg6>.

Prompt Engineering Strategy

The researchers designed a prompting strategy to mimic the human coding procedure, involving emerging coding at first, followed by a detailed codebook with examples, and finally addressing conflicting examples <ref:2604.10834#pg10>. They tested different prompt types: P1 (only task description and code names), P2 (formal definition of codes), and P3 (examples of other annotators) <ref:2604.10834#pg8>. The most complex prompt, P3+, showed the LLM the conflicting examples obtained in a pilot interaction, as well as the researchers’ answers to those ambiguities <ref:2604.10834#pg8>.

Results and Limitations

The analysis compared LLM outputs against human annotators and reviewers using Cohen’s Kappa (chance-corrected accuracy) <ref:2604.10834#pg9>. The results showed that LLMs cannot reliably replace human annotators for security-relevant annotations, not even with a detailed code-book and examples from other annotators <ref:2604.10834#pg10>. Table 3 reports the summary with the average precision, recall, accuracy, and Cohen’s Kappa (chance-corrected accuracy) for experiments where they tried to replace a human annotator with an LLM for the best prompt P3+ <ref:2604.10834#pg10>. The overall performance is too low to replace a human annotator <ref:2604.10834#pg10>.

Conclusion and Future Work

The findings suggest that the role of LLMs in qualitative security coding should, at present, remain assistive rather than autonomous <ref:2604.10834#pg12>. The paper concludes that prompt refinements with increasing efforts in terms of tokens do not uniformly guarantee a good performance across LLMs for security-specific annotations <ref:2604.10834#pg10>. Future work should focus on structured prompting interfaces, model calibration, and the design of hybrid pipelines where LLMs complement rather than replace human insight in technically demanding qualitative analyses <ref:2604.10834#pg12>. The paper notes that Prompt engineering lacks well-established, universally optimal procedures <ref:2604.10834#pg8>.

References

[1] Toufique Ahmed, Amiangshu Bosu, Anindya Iqbal, and Shahram Rahimi. 2017. SentiCR: A customized sentiment analysis tool for code review interactions <ref:2604.10834#pg14>.

[2] Anthropic. 2025. Claude 4: Claude Opus 4 & Claude Sonnet 4 Large Language Models <ref:2604.10834#pg14>.

[3] Andreas Armborst. 2017. Thematic proximity in content analysis <ref:2604.10834#pg14>.

[4] Anthony G Barnston. 1992. Correspondence among the correlation, RMSE, and Heidke forecast verification measures; refinement of the Heidke score <ref:2604.10834#pg14>.

[5] Avik Bhattacharjee. 2025. Introduction to Amazon Bedrock <ref:2604.10834#pg14>.

[6] David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation <ref:2604.10834#pg14>.

[7] Fabio Calefato, Filippo Lanubile, Federico Maiorano, and Nicole Novielli. 2018. Sentiment polarity detection for software development <ref:2604.10834#pg14>.

[8] Fabio Calefato, Filippo Lanubile, and Nicole Novielli. 2017. Emotxt: a toolkit for emotion recognition from text <ref:2604.10834#pg14>.

[9] Victoria Clarke and Virginia Braun. 2017. Thematic analysis <ref:2604.10834#pg14>.

[10] Guilherme A Maldonado da Cruz, Elisa Hatsue Moriya Huzita, and Valéria D Feltrim. 2016. Estimating Trust in Virtual Teams-A Framework based on Sentiment Analysis <ref:2604.10834#pg14>.

[11] Shih-Chieh Dai, Aiping Xiong, and Lun-Wei Ku. 2023. LLM-in-the-loop: Leveraging large language model for thematic analysis <ref:2604.10834#pg14>.

[12] DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu. 2025. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models <ref:2604.10834#pg14>.

[13] Zackary Okun Dunivin. 2024. Scalable qualitative coding with llms: Chain-of-thought reasoning matches human performance in some hermeneutic tasks <ref:2604.10834#pg14>.

[14] Denae Ford, Justin Smith, Philip J Guo, and Chris Parnin. 2016. Paradise unplugged: Identifying barriers for female participation on stack overflow <ref:2604.10834#pg14>.

[15] Robert Wayne Gregory, Mark Keil, Jan Muntermann, and Magnus Mähring. 2015. Paradoxes and the nature of ambidexterity in IT transformation programs <ref:2604.10834#pg14>.

[16] Greg Guest, Kathleen M MacQueen, and Emily E Namey. 2011. Applied thematic analysis <ref:2604.10834#pg14>.

[17] Wenbo Guo, Dongliang Mu, Jun Xu, Purui Su, G. Wang, and Xinyu Xing. 2018. LEMNA: Explaining Deep Learning based Security Applications <ref:2604.10834#pg14>.

[18] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford et al. 2024. Gpt-4o system card <ref:2604.10834#pg14>.

[19] Md Rakibul Islam and Minhaz F Zibran. 2018. DEVA: sensing emotions in the valence arousal space in software engineering text <ref:2604.10834#pg14>.

[20] Nishant Jha and Anas Mahmoud. 2018.

1. Improve LLM performance by incorporating emerging codes, a detailed codebook with examples, and conflicting examples into prompting strategies to mimic human annotation best practices. This addresses RQ2 by testing prompts P1 through P3+, aiming to find the optimal prompt structure that yields significant gain (Table 6).

2. Implement a system that uses Cohen’s Kappa (chance-corrected accuracy) as the primary success metric instead of traditional accuracy, as this metric is described as being identical to the definition of chance corrected accuracy and is more robust against unbalanced datasets.

3. Develop a model calibration layer that specifically targets distinguishing between general technical terminology from code-specific identifiers, addressing the qualitative analysis finding that models can get confused by the literal meaning of words and miss the high-level significance.

4. Design a hybrid pipeline where LLMs act as an assistant to generate a first draft of annotations, highlighting potential codes for human verification, or accelerating low-complexity tagging tasks, rather than attempting to be a full replacement for human annotators.

[Background:] Thematic analysis of free-text justifications in human experiments provides significant qualitative insights. Yet, it is costly because reliable annotations require multiple domain experts. Large language models (LLMs) seem ideal candidates to replace human annotators. [Problem:] Coding security-specific aspects (code identifiers mentioned, lines-of-code mentioned, security keywords mentioned) may require deeper contextual understanding than sentiment classification. [Objective:] Explore whether LLMs can act as automated annotators for technical security comments by human subjects. [Method:] We prompt four top-performing LLMs on LiveBench to detect nine security-relevant codes in free-text comments by human subjects analyzing vulnerable code snippets. Outputs are compared to human annotators using Cohen's Kappa (chance-corrected accuracy). We test different prompts mimicking annotation best practices, including emerging codes, detailed codebooks with examples, and conflicting examples. [Negative Results:] We observed marked improvements only when using detailed code descriptions; however, these improvements are not uniform across codes and are insufficient to reliably replace a human annotator. [Limitations:] Additional studies with more LLMs and annotation tasks are needed.