Embeddings for Preferences, Not Semantics
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Embeddings for Preferences, Not Semantics".
Jane: Here's the gist of the paper. Modern AI is opening the door to collective decision-making in which participants express their views as free-form text rather than voting on a fixed set of candidates A natural idea is to embed these opinions in a vector space so that the substantial literature on facility location problems and fair clustering can be brought to bear Standard text embeddings measure semantic similarity, whereas distances in facility location problems and fair clustering require what we call preferential similarity We formalize this as an invariance problem: text embedding models encode both a preference-relevant signal (stance and values) and semantic nuisance (style and wording), and the two are observationally correlated, so a geometry that relies on nuisance can appear preference-correct even when it is not We show that synthetic training data designed to break this correlation provably shifts the optimal scorer away from nuisance-dominated cosine and significantly improves preference prediction across 11 online deliberation datasets Diagnosing the Mismatch The authors diagnose the mismatch between semantic and preferential similarity as an invariance problem They formalize it via a decomposition of the cosine margin into a preference component and a nuisance component The utility of a candidate statement is modeled as the negative squared Euclidean distance to the participant’s ideal point within S Cosine similarity on off-the-shelf models decomposes additively as s(av, xj) = ⟨ψS(av), ψS(xj)⟩∆S + ⟨ψ⊥(av), ψ⊥(xj)⟩∆T > This decomposition shows that cosine captures the bilinear part of the utility margin but misses the item-norm difference ∆norm entirely, and contributes an out-of-subspace nuisance ∆T that has no utility counterpart How the Training Signal is Built The method involves synthesizing triplets consisting of an anchor, a preference match with different wording, and a semantic distractor with high surface overlap but opposite stance Training on these triplets forces cosine to down-weight nuisance variation that ordinary data leaves confounded with preference This procedure is called decorrelated preference tuning (DPT) and trains the encoder on them so that cosine learns to follow the preference-aligned item rather than the distractor that is similar in a superficial sense Theorem 1 proves that under this hard-triplet distribution, Bradley-Terry risk is strictly decreased by reducing the nuisance contribution below cosine’s unit weighting Where the Gains Come From Empirically, DPT substantially reduces hard-triplet failure identified in §4.3, lifting hard-triplet cosine accuracy from 48.3% to 80.0% for sentence-T5-XL The method is not specific to sentence-T5; applying it to three widely-used BERT-derived encoders produces similar patterns, with hard-triplet accuracy lifting by roughly 20 to 50 percentage points across models Furthermore, when per-topic votes are available, a low-rank projection of frozen embeddings outperforms full preference tuning The per-topic projected embedding reaches 77.6% macro-mean accuracy on the test split for the ideal-point scorer Validation and Deployment The DPT-tuned and per-topic projected embeddings provide a geometry in which clusters are coherent in approval behavior, beyond what the base ST5-XL geometry already provides The mean lift across all 12 configurations is 5.6% for the base, 6.7% for DPT-tuned, and 6.6% for the per-topic projected embedding This suggests that the DPT-tuned embedding provides a geometry which would be useful for the types of tasks identified in §8 The results indicate that val-selection on natural data is not an unalloyed good for downstream evaluation under distribution shift Limitations and Future Work The work focuses on within-participant rankings, not whether absolute distances are calibrated, and since utility is latent, direct verification is difficult A natural next step is to generate hard triplets conditioned on a target topic, producing a topic-specific embedding without per-topic votes An embedding aimed at capturing preferences is an empirical and imperfect representation of preferences, and it should not be treated as a perfect representation of any individual’s considered judgment The paper concludes that the two geometries are complementary, with DPT shifting the global encoder geometry to suppress the nuisance subspace, and the per-topic projected embedding extracting the dataset-specific preference subspace Pruning redundant reasoning tokens The authors show that when per-topic votes are available, a simpler method suggests learning a low-rank projection of the frozen embedding This projected embedding performs very well in practice despite its simplicity The ideal-point scorer commits to three main structural claims about ψ˜: the scorer is linear in ψ, tied between anchor and item (both projected by the same L), and a distance rather than an inner product Replacing the linear L⊤ with a shared nonlinear MLP while keeping the squared-distance form costs 4.1% on natural data and 14.6% on the hard evaluation triplets The per-topic scorer accuracy rises from r=1 and plateaus at r=20 How the training signal is built The authors formalize the direction in which hard-triplet training should move a scorer by defining the hard-triplet condition as E
∆T G: ≤ 0 almost surely, with strict inequality on a set of positive probability This condition formalizes the idea that, in hard triplets, the preference and nuisance components point in opposite directions The Bradley-Terry population risk R(B, λ) is shown to be strictly decreased by decreasing the weight on the nuisance when E
∆T G: ≤ 0 a.s.
Tom: Where the gains come from The DPT and the per-topic probe are complementary contributions, not redundant DPT shifts the global encoder geometry to suppress the nuisance subspace, while the per-topic projected embedding extracts the dataset-specific preference subspace The largest cosine averages 0.95 and the median averages 0.80 across datasets for the two probes The two probes share a core preference subspace where most directions are well aligned, and each has a small number of directions that are nearly orthogonal to anything in the other The paper shows that the same rank-20 budget extracts a stronger preference signal once the base directions have been cleaned by DPT Pruning redundant reasoning tokens The authors show that when stance and wording move together, the nuisance margin satisfies E
∆T G: > 0, which is the opposite of the hard-triplet condition used in Theorem 1 Normal-correlation training recovers a small lift over base cosine but is significantly worse than DPT The authors also show that val-selection on natural data is not an unalloyed good for downstream evaluation under distribution shift How the training signal is built The authors show that when stance and wording move together, the nuisance margin satisfies E
∆T G: > 0, which is the opposite of the hard-triplet condition used in Theorem 1 Normal-correlation training recovers a small lift over base cosine but is significantly worse than DPT The authors also show that val-selection on natural data is not an unalloyed good for downstream evaluation under distribution shift Where the gains come from The authors show that when stance and wording move together, the nuisance margin satisfies E
∆T G: > 0, which is the opposite of the hard-triplet condition used in Theorem 1 Normal-correlation training recovers a small lift over base cosine but is significantly worse than DPT The authors also show that val-selection on natural data is not an unalloyed good for downstream evaluation under distribution shift Where the gains come from
Paper summary: Tom: So, we're talking about this paper, "Embeddings for Preferences, Not Semantics." Basically, modern AI is letting people express their opinions as free-form text instead of just picking from a list of candidates.
Jane: And the core idea here is that we can embed these opinions in a vector space so we can use all that existing research on things like facility location problems and fair clustering. But standard embeddings measure how similar two pieces of text are semantically, which isn't what we need for preference-based tasks.
Lu: Exactly. We need preferential similarity, which means a person's agreement with a statement should be related to how close it is to their ideal point in that vector space.
Tom: Right, so the main problem they are tackling is that off-the-shelf embeddings mix two different things: the preference signal and the style or wording.
Meng: So, these standard models capture both what a person likes—their stance and values—and what a statement looks like—its style and how it's phrased. The paper says they are observationally correlated, but that correlation breaks when we need true preference accuracy.
Lalam: They formalize this as an invariance problem where the embedding model encodes both the preference-relevant signal and the semantic nuisance, and these two things are linked in a way that can be misleading <ref:2605.08360#pg1>.
Tom: And they show that this correlation can cause a geometry based on style to look preference-correct even when it isn't, which is why they need a new approach.
Jane: They tackle this by showing that cosine similarity, the standard way we measure similarity, breaks down because it captures the wrong part of the utility margin.
Lu: They break cosine down into two parts: one that captures the preference component and another one that's just a nuisance from style and wording <ref:2605.08360#pg2>.
Tom: This decomposition shows that cosine only captures the bilinear part of the utility margin, but it completely misses the item-norm difference, which is crucial for preference.
Meng: And this difference contributes an out-of-subspace nuisance, something that has no real utility counterpart when we're trying to score preferences <ref:2605.08360#pg2>.
Jane: To fix this, they synthesize training data called hard triplets designed specifically to break that correlation.
Paper summary: Lu: These triplets are built with an anchor statement, a preference match with different wording, and a semantic distractor that has high surface overlap but the opposite stance <ref:2605.08360#pg2>.
Tom: Training on these forces the cosine similarity to down-weight that nuisance variation that regular data usually lets get mixed up with preference.
Meng: They call this procedure decorrelated preference tuning, or DPT, and they train the encoder on these triplets so it learns to follow the preference-aligned item instead of a superficial distractor <ref:2605.08360#pg2>.
Jane: Theorem one shows that under this specific way of training, Bradley-Terry risk is strictly decreased by reducing the nuisance contribution below cosine's unit weighting <ref:2605.08360#pg1>.
Tom: So, where do we see these gains in practice? The results look pretty strong when they test it on those hard triplets.
Meng: Empirically, DPT substantially reduces the hard-triplet failure rate identified in section four point three of the paper. They lift that cosine accuracy from forty-eight point three percent up to eighty percent for the sentence T5 XL model <ref:2605.08360#pg2>.
Lu: And it isn't just specific to sentence T5 XL; applying this method to three different BERT-derived encoders shows similar patterns, with that hard-triplet accuracy lifting by roughly twenty to fifty percentage points across those models <ref:2605.08360#pg2>.
Jane: That sounds like a big win for robustness, especially since the authors noted that val-selection on natural data isn't always good for evaluation when there's a distribution shift <ref:2605.08360#pg1>.
Tom: And they also found something interesting when per-topic votes are available. A low-rank projection of frozen embeddings performs better than full preference tuning <ref:2605.08360#pg2>.
Meng: The per-topic projected embedding reaches a seventy-seven point six percent macro mean accuracy on the test split for the ideal-point scorer <ref:2605.08360#pg2>.
Lu: These DPT and per-topic probe methods provide a geometry where clusters are coherent in approval behavior beyond what the base sentence T5 XL geometry already offers <ref:2605.08360#pg1>.
Jane: The mean lift across all twelve configurations is about five point six percent for the base, six point seven percent for DPT-tuned, and sixty-six point six percent for the per-topic projected embedding <ref:2605.08360#pg1>.
Paper summary: Tom: That suggests that DPT provides a geometry useful for tasks they identified later in section eight of the paper.
Meng: It means that when you look at natural data, val-selection isn't always a good thing for seeing how well your model will actually perform down the line under distribution shift <ref:2605.08360#pg1>.
Lu: The authors also mentioned some limitations to keep in mind. They focus mostly on within-participant rankings, not whether absolute distances are calibrated, and since utility is latent, direct verification is difficult <ref:2605.08360#pg1>.
Jane: They also say that an embedding aimed at capturing preferences is just an empirical representation of preferences, so you shouldn't treat it as a perfect picture of any individual's judgment <ref:2605.08360#pg1>.
Tom: So, what’s the next step for them? What’s the future work they point toward?
Lu: They suggest generating hard triplets conditioned on a target topic to produce a topic-specific embedding without needing per-topic votes <ref:2605.08360#pg1>.
Meng: That would give you an embedding that targets preferences for one specific subject, rather than trying to model everything at once across all topics <ref:2605.08360#pg1>.
Jane: And they also mentioned that the two geometries they found, DPT and the per-topic projection, are complementary <ref:2605.08360#pg1>.
Tom: So one shifts the whole global encoder geometry to suppress that nuisance subspace, and the other extracts a dataset-specific preference subspace <ref:2605.08360#pg1>.
Lu: To wrap up, this paper, "Embeddings for Preferences, Not Semantics," is about moving past standard text embeddings when we need to model human preferences in collective decision-making.
Jane: The core of the idea is that off-the-shelf embeddings mix preference signals with style signals, and they show how to break that correlation using decorrelated preference tuning.
Tom: They found this method significantly improves performance on hard triplet testing, lifting accuracy from forty-eight point three percent up to eighty percent for models like sentence T5 XL <ref:2605.08360#pg2>.
Meng: And they also showed that when you have per-topic votes, a simpler method involving a low-rank projection of the frozen embeddings works very well in practice <ref:2605.08360#pg1>.
Lu: The implication is that we can create vector spaces where distances actually reflect what people prefer, which is something critical for things like fair clustering and facility location problems <ref:2605.08360#pg1>.
Conclusion: Tom: So we're wrapping up this look at "Embeddings for Preferences, Not Semantics." Basically, the whole point is that standard text embeddings are confusing preference with just style and wording on a computer.
Jane: Right, so instead of just measuring how similar two sentences are in meaning, we need to measure how close they are to what a person actually likes or dislikes.
Lu: The authors show that current methods break down because the way computers calculate similarity mixes the preference signal with the noise from the language itself, and they build training data specifically designed to clean that up.
Meng: From an engineering standpoint, this means we can try to get better results for things like grouping opinions or finding fair clusters without relying on models that just guess based on general text similarity.
Lalam: I’m confident in this direction because the math shows that by tuning the training data to force a specific pattern, we can reduce the error caused by that style noise significantly.
Tom: So it’s about taking those complex human opinions and putting them into a vector space where the distance between points actually reflects what people want.
Jane: It’s about making sure that when an AI looks at two statements, it doesn't just see they sound alike, but that it sees if one statement is genuinely preferred over the other based on stance and values.
Lu: The implication is that we can use these new geometries for problems like facility location or clustering where people need to make decisions based on their actual stances.
Meng: If this works reliably, it could mean deployment systems for collective decision-making get much smarter at understanding nuanced human input without us having to manually engineer every single preference feature.
Carter Blair, Ariel D. Procaccia, Milind Tambe
Harvard University
cs.AI
Submitted: 2026-05-08
Updated: 2026-11-06
Comments: Accepted at NeurIPS 2026. Model: https://huggingface.co/cartgr/embeddings-for-preferences-st5-xl
Code: https://github.com/cartgr/Embeddings-for-Preferences
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 99/100
The gist: Modern AI is opening the door to collective decision-making in which participants express their views as free-form text rather than voting on a fixed set of candidates A natural idea is to embed…
Key concepts
- Invariance Problem
- This is the core challenge: text embeddings capture both what people *say* (preference) and how they *say* it (style). The problem is that a geometry based on style noise can look preference-correct even when it isn't. The authors seek a representation that ignores the style nuisance to focus only on preference.
- Cosine Similarity Decomposition
- This breaks down how standard cosine similarity works by separating its effect into two parts: one related to the actual preference signal and another related to stylistic differences. This decomposition reveals that standard methods capture some aspects but miss crucial information about the item-norm difference, which is essential for accurate preference scoring.
- Decorrelated Preference Tuning (DPT)
- This is a training method using synthetic data made of triplets: an anchor, a preference match with different wording, and a semantic distractor. Training on these forces the embedding model to learn the preference direction instead of being confused by superficial stylistic similarities, effectively tuning the model away from nuisance variation.
Terminology
Summary
Modern AI is opening the door to collective decision-making in which participants express their views as free-form text rather than voting on a fixed set of candidates A natural idea is to embed these opinions in a vector space so that the substantial literature on facility location problems and fair clustering can be brought to bear Standard text embeddings measure semantic similarity, whereas distances in facility location problems and fair clustering require what we call preferential similarity We formalize this as an invariance problem: text embedding models encode both a preference-relevant signal (stance and values) and semantic nuisance (style and wording), and the two are observationally correlated, so a geometry that relies on nuisance can appear preference-correct even when it is not We show that synthetic training data designed to break this correlation provably shifts the optimal scorer away from nuisance-dominated cosine and significantly improves preference prediction across 11 online deliberation datasets
Diagnosing the Mismatch
The authors diagnose the mismatch between semantic and preferential similarity as an invariance problem They formalize it via a decomposition of the cosine margin into a preference component and a nuisance component The utility of a candidate statement is modeled as the negative squared Euclidean distance to the participant’s ideal point within S Cosine similarity on off-the-shelf models decomposes additively as s(av, xj) = ⟨ψS(av), ψS(xj)⟩∆S + ⟨ψ⊥(av), ψ⊥(xj)⟩∆T > This decomposition shows that cosine captures the bilinear part of the utility margin but misses the item-norm difference ∆norm entirely, and contributes an out-of-subspace nuisance ∆T that has no utility counterpart
How the Training Signal is Built
The method involves synthesizing triplets consisting of an anchor, a preference match with different wording, and a semantic distractor with high surface overlap but opposite stance Training on these triplets forces cosine to down-weight nuisance variation that ordinary data leaves confounded with preference This procedure is called decorrelated preference tuning (DPT) and trains the encoder on them so that cosine learns to follow the preference-aligned item rather than the distractor that is similar in a superficial sense Theorem 1 proves that under this hard-triplet distribution, Bradley-Terry risk is strictly decreased by reducing the nuisance contribution below cosine’s unit weighting
Where the Gains Come From
Empirically, DPT substantially reduces hard-triplet failure identified in §4.3, lifting hard-triplet cosine accuracy from 48. p. 6 0% and naturaltriplet mean accuracy from 65. Further, the recipe is not specific to sentence-T5:
applying the same training to three widely-used BERT-derived encoders, e5-large-v2, BGE-large,
and all-mpnet-base, produces similar patterns, with hard-triplet accuracy lifting by roughly 20 to 50
percentage points across models (Table 3). 0% and naturaltriplet mean accuracy from 65. Further, the recipe is not specific to sentence-T5:
applying the same training to three widely-used BERT-derived encoders, e5-large-v2, BGE-large,
and all-mpnet-base, produces similar patterns, with hard-triplet accuracy lifting by roughly 20 to 50
percentage points across models (Table 3).Where the paper says it
Where the paper says it
Validation and Deployment
The DPT-tuned and per-topic projected embeddings provide a geometry in which clusters are coherent in approval behavior, beyond what the base ST5-XL geometry already provides The mean lift across all 12 configurations is 5.6% for the base, 6. p. 1 The
semantic distractor uses the anchor’s wording but reverses its stance, while the preference match
preserves the stance but changes the wording. A base embedding model ranks the distractor closer to
the anchor. 6 Empirical Results
Fine-tuning substantially reduces the hard-triplet failure identified in §4. Our DPT-tuned sentenceT5-XL lifts hard-triplet (from the evaluation set) cosine accuracy from 48.Where the paper says it
Where the paper says it
Limitations and Future Work
The work focuses on within-participant rankings, not whether absolute distances are calibrated, and since utility is latent, direct verification is difficult A natural next step is to generate hard triplets conditioned on a target topic, producing a topic-specific embedding without per-topic votes An embedding aimed at capturing preferences is an empirical and imperfect representation of preferences, and it should not be treated as a perfect representation of any individual’s considered judgment The paper concludes that the two geometries are complementary, with DPT shifting the global encoder geometry to suppress the nuisance subspace, and the per-topic projected embedding extracting the dataset-specific preference subspace
Pruning redundant reasoning tokens
The authors show that when per-topic votes are available, a simpler method suggests learning a low-rank projection of the frozen embedding This projected embedding performs very well in practice despite its simplicity The ideal-point scorer commits to three main structural claims about ψ˜: the scorer is linear in ψ, tied between anchor and item (both projected by the same L), and a distance rather than an inner product Replacing the linear L⊤ with a shared nonlinear MLP while keeping the squared-distance form costs 4.1% on natural data and 14. p. 1 Generic embedding models do not have this invariance, because
they encode topical and stylistic features that are useful for retrieval and similarity tasks but unrelated
to whether two statements would be endorsed by the same person. On natural deliberation data, this
gap is partly hidden because semantic and preferential similarity are correlated: people who share a
stance often share wording.Where the paper says it
The authors formalize the direction in which hard-triplet training should move a scorer by defining the hard-triplet condition as E[∆T G] ≤ 0 almost surely, with strict inequality on a set of positive probability This condition formalizes the idea that, in hard triplets, the preference and nuisance components point in opposite directions The Bradley-Terry population risk R(B, λ) is shown to be strictly decreased by decreasing the weight on the nuisance when E[∆T G] ≤ 0 a.s.
The DPT and the per-topic probe are complementary contributions, not redundant DPT shifts the global encoder geometry to suppress the nuisance subspace, while the per-topic projected embedding extracts the dataset-specific preference subspace The largest cosine averages 0.95 and the median averages 0. p. 8 Since
platforms typically surface several comments rather than one, the relevant problem is selecting a slate,
which corresponds geometrically to clustering. Here k-median and k-center capture utilitarian and
egalitarian aggregation, while proportional fairness requires any large, aligned subgroup to have a
representative [Chen et al.Where the paper says it
The authors show that when stance and wording move together, the nuisance margin satisfies E[∆T G] > 0, which is the opposite of the hard-triplet condition used in Theorem 1 Normal-correlation training recovers a small lift over base cosine but is significantly worse than DPT The authors also show that val-selection on natural data is not an unalloyed good for downstream evaluation under distribution shift
The authors show that when stance and wording move together, the nuisance margin satisfies E[∆T G] > 0, which is the opposite of the hard-triplet condition used in Theorem 1 Normal-correlation training recovers a small lift over base cosine but is significantly worse than DPT The authors also show that val-selection on natural data is not an unalloyed good for downstream evaluation under distribution shift
The authors show that when stance and wording move together, the nuisance margin satisfies E[∆T G] > 0, which is the opposite of the hard-triplet condition used in Theorem 1 Normal-correlation training recovers a small lift over base cosine but is significantly worse than DPT The authors also show that val-selection on natural data is not an unalloyed good for downstream evaluation under distribution shift
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Develop embeddings that capture preference geometry by decomposing cosine margin into a preference component and a nuisance component to ensure invariance to style and wording, as described in
We formalize this as an invariance problem: text embedding models encode both a preference-relevant signal (stance and values) and semantic nuisance (style and wording), and the two are observationally correlated.
-
Implement decorrelated preference tuning (DPT) by training the encoder on synthetic triplets where
the nuance signal intentionally points in the wrong direction,
which forcescosine to down-weight nuisance variation that ordinary data leaves confounded with preference
andstrictly decreases the Bradley-Terry risk relative to standard cosine.
-
Utilize per-topic projected embeddings where a low-rank linear map is learned from participant votes to score candidates, allowing for a
stronger topic-specific deployment mode where the base embedding model is held fixed and a low-rank linear map is learned from participant votes.
-
Employ a rank-r projection of frozen embeddings to create ideal-point models, as this method shows that
the frozen encoder already contains preference signal and that L simply learns to read it out,
achieving high accuracy on hard evaluation triplets. -
Design user clustering algorithms using DPT-tuned or per-topic projected embeddings to form
user vectors by mean-pooling the encoder embeddings of their authored comments
and then compute within-cluster approval rates, which can revealhow well the geometry aligns with participants’ expressed approval preferences.
Abstract
Modern AI is opening the door to collective decision-making in which participants express their views as free-form text rather than voting on a fixed set of candidates. A natural idea is to embed these opinions in a vector space so that the substantial literature on facility location problems and fair clustering can be brought to bear. But standard text embeddings measure semantic similarity, whereas distances in facility location problems and fair clustering require what we call preferential similarity: a participant's agreement with a piece of text should be inversely related to their distance from it. Off-the-shelf embeddings inherit a coarse preference signal through a correlation between semantic and preferential similarity, but fail to capture preferences when the correlation breaks. We formalize this as an invariance problem: text embedding models encode both a preference-relevant signal (stance and values) and semantic nuisance (style and wording), and the two are observationally correlated, so a geometry that relies on nuisance can appear preference-correct even when it is not. We show that synthetic training data designed to break this correlation provably shifts the optimal scorer away from nuisance-dominated cosine and significantly improves preference prediction across 11 online deliberation datasets.
Sources
- A New Heuristic Algorithm for Balanced Deliberation Groups
- Generating Fair Consensus Statements with Social Choice on Token-Level MDPs
- Probably Approximately Consensus: On the Learning Theory of Finding Common Ground
- Finding Common Ground in a Sea of Alternatives
- Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models
- Opportunities and Risks of LLMs for Scalable Deliberation with Polis
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
- SparseCL: Sparse Contrastive Learning for Contradiction Retrieval
- Qwen2 Technical Report
- Jasper and Stella: distillation of SOTA embedding models
- Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents
- Can AI Agents Synthesize Scientific Conclusions in Health?