Beyond Detection: Evaluating Defensive LLMs Against AI-Generated Social Engineering in Live Turn-by-Turn Interaction

This paper introduces trust-chain localization for evaluating defensive LLMs beyond binary scam detection. Using a controlled 300-case online-housing benchmark, we compared five models under live turn-by-turn and static full-transcript protocols. The results show that intervention coverage, intervention timing, and correct localization are distinct capabilities: models may intervene while identifying the wrong failure mechanism or correctly localize a failure without recommending protective action. Performance also varies across actor-authority, asset-control, verification-sufficiency, and transaction-path failures, with asset-control cases forming a recurring bottleneck for strong hosted models. These findings show that safe-looking warnings alone are insufficient for evaluating defensive LLMs. Future systems and benchmarks should separately measure whether a model intervenes in time and whether it identifies the correct compromised trust-chain component.

Improvements to AI Systems:

1. Add a trust-chain localization module to defensive LLMs. Instead of only outputting a binary scam/not scam verdict, the system must explicitly identify which component of the trust chain failed (e.g., actor-authority, asset-control, verification-sufficiency, transaction-path). This requires a structured output layer (e.g., JSON with fields: intervention needed, timing, localized component, recommended action).

2. Decouple intervention timing from diagnostic accuracy in the model’s training objective. Use a multi-task loss function that separately optimizes (a) early intervention (e.g., reward for flagging before the user transfers funds) and (b) correct component localization (e.g., cross-entropy loss on the trust-chain component). This prevents the model from learning to produce generic warnings without deep reasoning.

3. Add a bottleneck-aware training curriculum for asset-control failures. Since these are the recurring weak point for hosted models, augment training data with adversarial asset-control scenarios (e.g., fake escrow services, third-party payment links, or requests to move funds to a safe account). Fine-tune the model specifically on these cases with contrastive examples that distinguish legitimate asset control from compromised ones.

4. Implement a dual-protocol evaluation harness in the system’s deployment. Before going live, run the model under both static full-transcript and live turn-by-turn conditions. If the model performs well on static but poorly on live, add a real-time intervention trigger that forces a check after each user message, rather than waiting for a final summary.

5. Add a post-hoc verification layer that cross-checks the localized component against the user’s actual transaction path. For example, if the model says asset-control failure, the system should automatically verify whether the payment recipient matches the verified landlord’s account. If mismatch, escalate to a human or block the transaction.

What the improved AI system can do:

• It can tell you why a conversation is risky (e.g., The landlord’s identity is unverified, but the payment link redirects to a third-party wallet) rather than just saying this is a scam.

• It can intervene at the right moment—e.g., immediately after the user is asked to wire a deposit—even if it hasn’t yet fully analyzed the entire transcript.

• It can handle asset-control cases that currently fool strong models, by explicitly simulating the flow of money and flagging any deviation from a trusted path.

• It can be evaluated more rigorously: you can measure whether it intervenes in time separately from whether it correctly identifies the compromised component, so you know which capability to improve next.

• It can provide actionable, component-specific warnings to the user (e.g., Do not share your bank routing number; the escrow service is not registered) instead of generic advice like be careful.