Comedic Fool's Gold: Reward Exploits and Countermeasures in Conversational Humor
Researchers trained a language model to be funny using automated rewards but found ways users could 'cheat' by exploiting reward shortcuts like word-shuffled replies or adding 'Haha!'. They developed defenses, including fluency filters and audience model normalization, which successfully blocked most attacks. However, some minor loopholes remained for specific tokens.