Multimodal Item Parameter Estimation using Simulated Response Probabilitie

The paper presents results from reconstructing multiple-choice model (MCM) and three-parameter logistic (3PL) model curves using a fine-tuned multimodal large language model (LLM) based on Qwen3.5. The model is prompted and fine-tuned to replicate choice probabilities across a large training corpus of multiple-choice items containing both image and text stimuli, conditioned on a labeled set of student ability levels. By learning to reproduce the systematic error patterns of students across a discrete range of abilities, the LLM implicitly captures the underlying response probabilities encoded in the 3PL and MCM curves. This allows us to accurately approximate item difficulty on a held-out test set directly from the model’s predicted option probabilities.

The study uses a dataset of 970 item models and 4,848 items, split into training, development, and test sets by item model to avoid conflation, with 727 train items, 97 development items, and 146 test items. There are 13.88 million responses total, with an average of approximately 2860 responses per item and a minimum of 860 responses. The average response ability level was 0.31 with a standard deviation of approximately 1. Items consist of a stimulus and four options, with both stimulus and options potentially having associated image components, which are combined into a single image for input.

The response modeling uses the 3PL model defined by Pij(X=1) = ci + (1 - ci)σ(ai(θj - bi)), where ai is the discrimination parameter, bi is the difficulty parameter, and ci is the guessing parameter. The MCM models the probability of each option k as Pijk = (exp(ηijk) + dik) / (1 + Σk exp(ηijk)), where ηijk = aikθj + bik. Parameters are estimated by minimizing negative log likelihood with L2 regularization using L-BFGS.

Ability levels are discretized into 20 intervals with descriptive labels ranging from Foundational to Elite mastery, with cutoffs from-2.7 to 2.7 in increments of 0.3. Each interval is associated with probability values for each option, creating an L × K matrix of probability values. The conditional probability that θ is in interval Ij is defined as wj = F(γj) - F(γj-1), and the expected value of the response function within each interval is computed via integration.

The language modeling uses Qwen3.5 models with 4-billion and 9-billion parameters, employing parameter-efficient fine-tuning with Low-Rank Adaptation (LoRA). Due to training instabilities, LoRA was applied only to the Gated Attention components, affecting roughly a quarter of the hidden layers, leaving the Gated DeltaNet layers untouched. The models are prompted with a system text indicating the model should act as a student, a user text containing the question and options, and an assistant text left blank to obtain next-token probabilities. The loss function is the Mean Squared Error between the token probabilities for each option and the target probability values.

Baseline models include MathBERT (110M parameters) and MetaMath (70B parameters), which use regression approaches treating IRT parameters as targets. Metrics include Pearson correlation, root mean squared error (RMSE), and quadratic weighted kappa (QWK) for discretized difficulty labels.

Results show that the fine-tuned Qwen3.5-9B model attains a Pearson correlation of 0.85 on the difficulty parameter b, exceeding MathBERT (0.68) and MetaMath (0.75), a relative improvement of roughly 15% over the stronger baseline, with an RMSE of 0.55 after linear development-set correction. When difficulty is discretized into five bands, the model reaches a QWK of 0.835, compared to MetaMath's 0.692 and MathBERT's 0.625. For the discrimination parameter a, the Qwen3.5-9B achieves a Pearson correlation of 0.31, and for the guessing parameter c, it achieves 0.48, notably higher than the baselines (0.25 and 0.28 for c).

The discussion notes that recovering the guessing parameter c at a correlation of 0.48 is the clearest evidence that the approach captures genuine response behavior rather than surface features of the stimulus, since a model that merely reads the question cannot easily infer how often low-ability students will be drawn to a particular distractor. The weak recovery of the discrimination parameter a indicates that slope information is partially lost, most plausibly through the discretization of ability into intervals and through the regression-based correction. The paper also notes that every arrow in the pipeline introduces approximation error, so the reported correlations are best read as lower bounds on what the framing can achieve under tighter modeling assumptions.

Improvements to AI Systems:

1. Ability-Aware Response Prediction: Train LLMs to output full response probability distributions over all answer options, conditioned on discretized student ability levels (e.g., 20 bins from-2.7 to 2.7), rather than single correct answers. This enables the model to simulate heterogeneous student behavior, including systematic errors and guessing patterns.

2. Multimodal Item Difficulty Estimation: Fine-tune a vision-language model (e.g., Qwen3.5-9B with LoRA on Gated Attention) on combined image-text stimuli to directly predict item difficulty (b), discrimination (a), and guessing (c) parameters via token-level probability matching. The improved system can estimate these IRT parameters for new, unseen items without needing human response data, achieving a Pearson correlation of 0.85 on difficulty and 0.48 on guessing.

3. Distractor Quality Assessment: Use the model’s predicted option probabilities to identify which distractors are most attractive to low-ability students (via the guessing parameter c and option-specific probabilities). This allows automated evaluation of multiple-choice item quality, flagging poorly performing distractors or items with ambiguous cues.

4. Adaptive Test Item Generation: Leverage the fine-tuned model to generate new multiple-choice items with calibrated difficulty levels by conditioning on desired ability intervals. The system can produce items whose predicted response curves match target 3PL or MCM parameters, enabling rapid creation of psychometrically valid test banks.

5. Student Ability Estimation from Response Patterns: Invert the model’s learned mapping—given a student’s actual response pattern on a set of items, use the model’s predicted probabilities to infer the student’s ability level (θ) via maximum likelihood or Bayesian updating. This yields a lightweight, LLM-based adaptive testing engine that updates ability estimates in real time.

6. Noise-Robust Parameter Recovery: Apply the development-set linear correction (as done for RMSE reduction) to any downstream IRT parameter predictions, improving calibration. The system can also output confidence intervals for difficulty estimates by leveraging the model’s token-level probability variance, aiding in test construction decisions.

7. Cross-Modal Transfer for Low-Resource Domains: Use the fine-tuned Qwen3.5-9B as a teacher to distill IRT prediction capabilities into smaller models (e.g., MathBERT) via knowledge distillation on probability outputs, enabling deployment in resource-constrained settings while retaining 80% of the correlation gains.

We present results from reconstructing multiple-choice model (MCM) and three-parameter logistic (3PL) model curves using a fine-tuned multimodal large language model (LLM) based on Qwen3.5. The model is prompted and fine-tuned to replicate choice probabilities across a large training corpus of multiple-choice items containing both image and text stimuli, conditioned on a labeled set of student ability levels. By learning to reproduce the systematic error patterns of students across a discrete range of abilities, the LLM implicitly captures the underlying response probabilities encoded in the 3PL and MCM curves. This allows us to accurately approximate item difficulty on a held-out test set directly from the model's predicted option probabilities.