Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study

arXiv:2505.21710 · cs.CL · Submitted 2025-05-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language".

Jane: The gist: ChatGPT performs well in detecting inappropriate content, showing notable improvements in accuracy through iterative refinements, particularly in Version 6,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let’s start by looking at the title and who wrote this stuff. The full title of this work is "Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study." It’s pretty straightforward, though I like the "Assessing" part because it tells you they aren't just claiming something works perfectly.

Jane: And the authors are Barbarestani, Maks, and Vossen. They’re from a place that did some solid work on content moderation annotation methods earlier, which gives them a good foundation for this kind of comparison. It suggests this isn't just an internal test; they’re trying to put ChatGPT against established benchmarks.

Lu: The implication here is that we need structured ways to evaluate these large language models in specific, nuanced tasks like targeting detection, not just general text generation. The authors are setting up a framework to systematically probe where the current technology succeeds and where it still falls short compared to human judgment.

Meng: So, what’s the immediate takeaway for us? It seems like the focus isn't just on building bigger models, but on making sure these existing models are reliable enough for real moderation work before we deploy them widely.

Lalam: I think it means we need to be very careful when we talk about AI catching something sensitive, because this paper points out that even advanced models have variability in how they spot specific targeting patterns.

The paper's summary: Tom: Now for the summary of what the authors actually found in this study. They compared ChatGPT’s outputs against crowd-sourced annotations and expert evaluations on a gold set of conversation threads. The core finding is that ChatGPT does perform well when it comes to detecting general inappropriate content, showing improvements as they refined it, especially with Version six <ref:2505.21710#pg1>.

Jane: But the summary also flags a specific issue: its performance on targeting language detection was variable, and it showed higher false positive rates when compared directly to expert judgments. That’s a key distinction they are making about the model's current limitations in that specific area.

Lu: They are looking at three parts of conversations—the beginning, the middle, and the final parts—to see how targeting behavior shifts across a thread. The study found that crowd workers noticed targeting initiation early on, with forty-six percent of subthreads starting with it, but ChatGPT detected a higher prevalence at the very beginning, about ninety-seven percent.

Meng: That difference between what the crowds saw and what the AI saw at the start is significant. If we rely too heavily on initial detection in a stream of comments, that variability could cause serious issues in how we triage incoming content.

Lalam: They also found that when it came to attributing targeting, attributed targeting became specific to an individual in twenty-three percent of cases by human annotators, but ChatGPT caught it twenty-eight percent of the time. That gap between human and AI attribution is what they are trying to quantify here.

The paper's improvements: Tom: The paper details how they improved the model through iterative prompting, moving from Version one up to Version six. They found that these prompt enhancements actually made a difference, leading to Version six achieving the highest Cohen’s Kappa score of zero point six six in identifying targeting language against those expert annotations on the gold set.

Jane: The key improvement they highlighted was making the model more consistent and less varied through prompt adjustments. Specifically, they changed the GPT model’s temperature setting from zero point three to zero for Version six which made the responses much more deterministic.

Lu: That adjustment to temperature is a technical detail that really matters because it moves the AI away from being overly creative and towards being more consistent with what’s actually present in the data, which is exactly what they need for reliable moderation.

Meng: Consistency at scale is everything for me. If a model can be made more deterministic, it means we can predict its output quality better when it hits high traffic volumes, instead of dealing with wildly fluctuating results from a less tuned model.

Lalam: So the improvement isn't just about better detection; it’s about aligning the AI’s behavior with human judgment by making its decision-making process more predictable and focused on what actually matters in those targeted comments.

Conclusion: Tom: So to wrap up this assessment of "Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study," the paper shows that while ChatGPT is a strong tool for spotting inappropriate language generally, we still need work on making its targeting detection more consistent.

Jane: The main implication is that continuous refinement through prompt engineering, like moving to Version six with a temperature of zero, can genuinely boost performance metrics when aligned with expert knowledge <ref:2505.21710#pg1>. It suggests that tweaking the input instructions matters as much as just increasing the model size.

Lu: I think it underscores that for complex tasks like identifying nuanced targeting across different parts of a conversation, we need these kinds of targeted refinement strategies to bridge the gap between raw capability and reliable application.

Meng: From an engineering standpoint, it tells us that tuning the inference parameters, like temperature, is a high-leverage way to improve reliability without necessarily needing a completely new model architecture every time.

Lalam: And for me, this paper confirms that we can use these comparative studies to pinpoint exactly where the AI is struggling—in this case, specifically with the variability in targeting detection—so we know exactly what needs to be addressed next in our refinement pipeline.

Baran Barbarestani, Isa Maks, Piek Vossen

Vrije Universiteit Amsterdam

cs.CL

Submitted: 2025-05-27

Updated: 2026-10-13

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 78/100

The gist: The gist: ChatGPT performs well in detecting inappropriate content, showing notable improvements in accuracy through iterative refinements, particularly in Version 6, while its performance in

Key concepts

Targeting Language
This refers to comments or titles in online discussions that specifically direct attention toward individuals or groups with the intent to harass, criticize, or attack them. The study focused on identifying these specific instances within conversation threads.
Inappropriate Content Detection
This involves using AI to automatically flag language that violates community standards, such as offensive words or explicit material. ChatGPT was tested on its effectiveness in spotting this type of harmful language across different parts of online conversations.
Iterative Prompt Refinement
This is the process of repeatedly adjusting the instructions given to an AI model (the prompt) to get better results. The researchers found that changing specific settings, like simplifying labels or using a different model version (GPT-4), led to major accuracy gains in detecting targeting language.
False Positive/Negative Rates
These metrics measure the AI's errors. A false positive occurs when the AI incorrectly flags safe content as inappropriate. A false negative occurs when the AI misses actual inappropriate content. The study noted that targeting detection had higher error rates than expert judgments initially.

Terminology

Summary

The gist: ChatGPT performs well in detecting inappropriate content, showing notable improvements in accuracy through iterative refinements, particularly in Version 6, while its performance in targeting language detection showed variability, with higher false positive rates compared to expert judgments.

Motivation and Context

Social network sites facilitate online interaction but the prevalence of Internet trolling poses significant challenges for these platforms. Technical solutions within social network software are increasingly recognized as vital for identifying and mitigating harmful conduct. Artificial intelligence AI tools can aid human moderators, improving content filtering and efficiency. However, scalability remains a challenge in content moderation as the vast volume of online content exceeds human moderation capacity. ChatGPT represents a significant advancement in natural language processing NLP.

Objectives and Contribution

The primary objectives are twofold: (1) To assess ChatGPT’s performance by evaluating how well it identifies targeting and inappropriate language in online comments compared to crowd-sourced and expert annotations. (2) To improve model accuracy by iteratively refining the model’s prompts and configurations to enhance alignment with human judgments. This study contributes by providing a detailed analysis of ChatGPT’s capabilities in detecting and classifying targeting and inappropriate language. It also compares the model’s performance with crowd-sourced and expert annotations to gauge its effectiveness in real-world scenarios.

Data Set Description

The study utilized an already existing data set of English conversation threads created by Barbarestani et al. (2024) from 28 banned subreddits on Reddit, comprising 67,677 submissions and 4,017,460 tokens. Subthreads were filtered based on comments (3 to 17 per subthread), tokens (51 to 1,276 per subthread), and toxicity scores calculated using three different lexicons. The gold data subset of 39 conversation threads was chosen through stratified sampling based on toxicity scores and the number of comments. Annotators reviewed comments and associated context, focusing on explicitly offensive language for inappropriateness and identifying instances where comments or titles targeted individuals or groups.

Methodology for Evaluation

The methodology involved comparing ChatGPT annotations to the crowd annotations on the gold set, which provided a benchmark. Analysis of conversation threads was segmented into three bins: beginning, middle, and final parts for inappropriate content and targeting behavior. ChatGPT’s capability to identify inappropriate and targeting behavior using contextual cues was evaluated across four scenarios. A prompt improvement process was conducted across six versions, focusing on refining prompts to enhance ChatGPT’s ability to identify targeting language more accurately.

Results and Iterative Refinement

In the gold set analysis, crowd workers consistently identified the early stages of conversations as critical for targeting initiation, with 46% of subthreads beginning with targeting. ChatGPT detected a higher prevalence of targeting at the beginning (97%) and maintained a consistent detection rate of 85% in both the middle and final parts of the conversations. Regarding targeting development, attributed targeting became specific to an individual in 23% of cases by crowd annotators and 28% by ChatGPT. The iterative refinement process showed that Version 6 achieved the highest Cohen’s Kappa score of 0.66 in targeting language identification by ChatGPT compared to the expert annotations on the gold set.

Limitations and Conclusion

The study found that ChatGPT demonstrated significant capabilities in detecting inappropriate content, with high agreement rates with expert annotations, though its targeting detection showed variability and higher false positives. The reliance on crowd-sourced annotations introduces variability in labeling, which can affect the consistency of performance comparisons. Future research should focus on improving targeting detection and reducing false positives to optimize the model’s effectiveness in real-world applications.

Appendix Details

The appendix details the various versions of prompts used to guide ChatGPT in annotating Reddit comments for targeting behavior, illustrating the iterative process of enhancing the model’s performance. The prompt Version 6 primarily differs from Version 5 in its adjustment of the GPT model’s temperature setting, which has been changed from 0.3 to 0, for more consistent and less varied responses from the model by making the output more deterministic. The final prompt version tasks the language model to classify each comment or title as either TARGETING or NOT TARGETING based on the provided context.

References

Barbarestani, B., Maks, I., and Vossen, P. T. (2024). Content moderation in online platforms: A study of annotation methods for inappropriate language. In Proceedings of the Fourth Workshop on Threat, Aggression & Cyberbullying@ LRECCOLING-2024, pp. 96–104.

Bassignana, E., Basile, V., Patti, V., and others (2018). Hurtlex: A multilingual lexicon of words to hurt. In CEUR Workshop proceedings, volume 2253, pp. 1–6.

Bhattacharya, A., Chakrabarti, T., Basu, S., Knott, A., Pedreschi, D., Chatila, R., Leavy, S., Eyers, D., Teal, P. D., and Biecek, P. (2024). Towards a crowdsourced framework for online hate speech moderation-a case study in the indian political scenario. In Companion Publication of the 16th ACM Web Science Conference, pp. 75–84.

Demetis, D. S. (2020). Breaking bad online: A synthesis of the darker sides of social networking sites. European Management Journal, 38(1):33–44.

Dinakar, K., Jones, B., Havasi, C., Lieberman, H., and Picard, R. (2012). Common sense reasoning for detection, prevention, and mitigation of cyberbullying. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(3):1–30.

Goar, V., Yadav, N. S., and Yadav, P. S. (2023). Conversational ai for natural language processing: An review of chatgpt. International Journal on Recent and Innovation Trends in Computing and Communication, 11:109–117.

Gongane, V. U., Munot, M. V., and Anuse, A. D. (2022). Detection and moderation of detrimental content on social media platforms: current status and future directions. Social Network Analysis and Mining, 12(1):129.

Gorwa, R., Binns, R., and Katzenbach, C. (2020). Algorithmic content moderation: Technical and political challenges in the automation of platform governance. Big Data & Society, 7(1):2053951719897945.

Haque, M. A. and Li, S. (2024). Exploring chatgpt and its impact on society. AI and Ethics, pp. 1–13.

Horta Ribeiro, M., Jhaver, S., Zannettou, S., Blackburn, J., Stringhini, G., De Cristofaro, E., and West, R. (2021). Do platform migrations compromise content moderation? evidence from r/the donald and r/incels. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–24.

Kumar, D., AbuHashem, Y.

Improvements for AI systems

  1. Improved targeting detection accuracy via prompt iteration: The iterative refinement process, specifically moving from Version 1 to Version 6, resulted in a Cohen’s Kappa score of 0.66 for targeting language identification by ChatGPT compared to expert annotations on the gold set, demonstrating the enhancements of Version 6 contributed to a slight enhancement in the model’s performance.

  2. Enhanced contextual cue utilization: Providing contextual cues significantly boosts detection accuracy, as demonstrated by "Scenario 1, where ChatGPT had access to both previous comments and their targeting labels along with the label of the comment being processed, the agreement rate for inappropriateness is significantly higher (71.32%) compared to Scenario 2."

  3. Optimized model determinism: The final refinement in Version 6, which involved further adjusted the temperature setting from 0.3 to 0 for even more consistent and less varied responses, ensures more consistent and less varied responses from the model by making the output more deterministic.

  4. Scalable content moderation support: The improved system can provide a scalable solution to managing online interactions by leveraging ChatGPT's capabilities, which can potentially replace or complement human annotators in real-time through alert mechanisms and preventive interventions.

Abstract

This study evaluates the effectiveness of ChatGPT, an advanced AI model for natural language processing, in identifying targeting and inappropriate language in online comments. With the increasing challenge of moderating vast volumes of user-generated content on social network sites, the role of AI in content moderation has gained prominence. We compared ChatGPT's performance against crowd-sourced annotations and expert evaluations to assess its accuracy, scope of detection, and consistency. Our findings show that ChatGPT performs well in detecting inappropriate content, and that almost the entire gain from iterative prompt refinement came from a single change --- simplifying the label scheme and moving to GPT-4 --- after which further adjustments to prompt structure and sampling temperature had little effect. Its performance on targeting language was weaker and less stable, with higher false positive rates than expert judgments before refinement and higher false negative rates after a change of model. This study contributes to the field by demonstrating the potential of AI models like ChatGPT to enhance automated content moderation systems while also identifying areas for further improvement. The results underscore the importance of careful prompt design and model selection, and of contextual understanding, in supporting automated moderation and mitigating harmful online behavior.

Sources

Related papers