How Do Large Language Models Judge Social Attraction? Evidence from Theory-Grounded Persona Ratings Across Multiple LLMs and Humans

This paper examines whether large language models (LLMs) can assess social attraction from theory-grounded persona profiles constructed from ten psychological and relational constructs and organized into three tiers: socially attractive, socially mixed, and socially unattractive. The study examines LLM ratings in two studies and compares them with human judgments in a third study.

In Study 1, 34 LLMs rated 12 profiles across three repeated runs. The results showed that although some models tended to give higher or lower ratings overall, they showed strong stability across runs, consistent three-tier ordering, and high agreement in relative profile ordering. Specifically, the mean single-measure absolute-agreement ICC was.965, with a median of.970, indicating high test-retest reliability. The LLM ratings reproduced the expected tier-level ordering: Socially attractive profiles received the highest ratings (M = 6.16, SD =.26), followed by socially mixed profiles (M = 4.70, SD =.34) and socially unattractive profiles (M = 2.93, SD =.41). A one-way repeated-measures ANOVA showed a strong effect of profile tier, F(2, 66) = 800.94, p <.001, ηp2 =.960. Cross-model agreement was high, with Kendall's W =.948 and mean pairwise Spearman correlation of.951. However, model-level means ranged from 3.87 to 5.09, indicating differences in overall rating leniency.

Study 2 examined sensitivity to gender presentation using six matched name-and-pronoun profile pairs and a separate pronoun-only test with a gender-neutral name. The results found no significant effects in either analysis. The three-tier pattern remained clear in the Gender-swapped condition, with a significant effect of profile tier, F(2, 26) = 276.91, p <.001, ηp2 =.955, but no effect of gender presentation, F(1, 13) = 0.03, p =.876, ηp2 =.002, and no interaction. In the pronoun-only test using Alex Wilson, ratings were similar across male (M = 5.39), female (M = 5.28), and gender-neutral presentations (M = 5.32), with no significant effect of pronoun presentation, F(2, 26) = 1.52, p =.238, ηp2 =.105.

In Study 3, 198 human participants evaluated the six matched profiles from Study 2. Their ratings reproduced the three-tier structure and followed a profile ordering consistent with that of the LLMs. The five-item social-attraction scale demonstrated high internal consistency, Cronbach's α =.937. Human ratings followed the expected structure: in the Original condition, socially attractive profiles received the highest ratings (M = 5.69), followed by socially mixed (M = 4.53) and socially unattractive profiles (M = 2.72). All pairwise tier contrasts were significant in both conditions. A linear mixed-effects model showed profile tier had a significant effect, likelihood-ratio χ2(2) = 834.00, p <.001, but neither gender presentation, χ2(1) = 1.12, p =.290, nor its interaction with profile tier, χ2(2) = 3.28, p =.194, was significant.

However, the study found that LLMs rated attractive profiles more positively and unattractive profiles more negatively than humans, while neither group showed a significant overall effect of gender presentation. Profile-level alignment was strong in both conditions, with Pearson's r =.981 in the Original condition and r =.979 in the Gender-swapped condition. Direct comparisons showed significant interactions between rater type and profile tier in both conditions. Holm-adjusted contrasts showed that LLMs rated attractive profiles more positively than humans in the Original (∆ = 0.49, p <.001) and Gender-swapped (∆ = 0.66, p <.001) conditions, and rated unattractive profiles more negatively in the Original (∆ = -0.49, p =.018) and Gender-swapped (∆ = -0.57, p =.003) conditions. Differences for the mixed tier were not significant.

The paper concludes that LLMs can reflect broad patterns in human social-attraction judgments from controlled persona profiles, but humans and LLMs did not use the rating scale identically, and some profile-level judgments differed. The authors note that LLMs may serve as scalable comparative instruments for examining standardized social stimuli, but their use requires repeated evaluations, comparisons across models, and validation against human judgments because aggregate alignment can conceal differences in scale use and profile-specific judgments. The absence of significant gender-presentation effects applies only to the controlled manipulation of names and pronouns and should not be generalized to broader forms of gender bias. The paper emphasizes that LLMs should therefore complement, rather than replace, human judgment in social-evaluation research.

Improvements to AI Systems:

1. Add calibrated social-attraction scoring with tier-aware anchoring.

• The AI system can be fine-tuned to output social-attraction ratings on a 1–7 scale that explicitly anchors to the three-tier structure (attractive: 6.2, mixed: 4.7, unattractive: 2.9).

• It can apply a correction factor to reduce systematic leniency/severity bias, matching human mean ratings more closely (e.g., dampening extreme scores for attractive/unattractive profiles).

2. Implement multi-run consensus and reliability reporting.

• The system can automatically run each evaluation 3+ times and compute intra-class correlation (ICC) and Kendall’s W to flag unstable or unreliable outputs.

• It can reject or re-sample responses when single-measure ICC falls below a threshold (e.g., <0.95), ensuring consistent relative ordering across profiles.

3. Integrate gender-presentation invariance checks.

• The system can include a built-in test module that swaps names/pronouns across matched profile pairs (e.g., male/female/neutral) and verifies no significant main effect or interaction (p > 0.05, ηp2 < 0.01).

• It can automatically generate a bias report for any new profile set, warning users if gender cues alter ratings beyond random variation.

4. Add human-validation alignment metrics.

• The AI can compare its profile-level rankings against a stored human benchmark (e.g., Pearson r > 0.97) and flag any profile where its rating deviates by more than 0.5 scale points from human averages.

• It can then adjust its output for those specific profiles using a learned correction function, improving cross-rater agreement.

5. Develop a comparative evaluation mode for standardized stimuli.

• The system can serve as a scalable screening tool to rank-order large numbers of persona profiles (e.g., thousands) by social attraction, using the three-tier structure as a prior.

• It can output not just ratings but also tier classifications with confidence intervals, enabling researchers to pre-filter stimuli before costly human studies.

6. Incorporate scale-use normalization across models.

• The system can automatically detect model-specific rating leniency (e.g., mean rating range 3.87–5.09) and apply a z-score or percentile normalization to make cross-model comparisons meaningful.

• It can provide both raw and normalized scores, with a note on model-level bias, to prevent misinterpretation of absolute values.

7. Add explicit uncertainty and boundary warnings.

• The system can flag profiles where its rating falls near tier boundaries (e.g., 4.0–5.0) as “low-confidence” and recommend human review.

• It can also refuse to generalize gender-neutral findings beyond name/pronoun manipulations, adding a disclaimer when users attempt to infer broader gender bias.

What the improved AI system can do:

• Provide reliable, human-aligned social-attraction scores for persona profiles with built-in quality checks (reliability, gender invariance, and human validation).

• Act as a pre-screening tool that ranks profiles by social attraction, reducing the need for large human samples in initial stages.

• Generate transparent reports that separate model bias from true profile effects, allowing researchers to trust relative comparisons while understanding absolute scale differences.

• Automatically detect and correct for systematic rating extremes, making LLM outputs more directly comparable to human judgments in social-evaluation research.