Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study
The study tested ChatGPT's ability to identify targeting and inappropriate language in online comments by comparing its results against human expert evaluations and crowd annotations. While ChatGPT performed well on general inappropriate content detection, its performance on targeting language was inconsistent. Iterative prompt refinement, especially moving to GPT-4 and simplifying labels, significantly improved accuracy for targeting but did not universally boost other metrics.