LLMs for Qualitative Data Analysis Fail on Security-specificComments in Human Experiments
The study tested if Large Language Models (LLMs) could automatically label technical security comments made by humans during code analysis experiments. Despite using detailed codebooks and conflicting examples, LLMs failed to reliably match human annotators for security-specific coding tasks. The conclusion is that LLMs cannot yet replace human experts in this domain.