Autograder+: A Multi-Faceted AI Framework for Rich Pedagogical Feedback in Programming Education

The gist: Autograder+ is a comprehensive and intelligent framework designed to evolve autograding from a summative evaluation tool into a formative learning platform by combining feedback generation via fine-tuned LLMs, visualization of student code submissions, and dynamic prompt steering.

The Problem Addressed

The rapid growth of programming education has outpaced traditional assessment tools, leaving faculty with limited means to provide meaningful, scalable feedback The rapid growth of programming education has outpaced traditional assessment tools, leaving faculty with limited means to provide meaningful, scalable feedback >

Conventional autograders act as “black-box” systems that merely indicate pass/fail status, offering little instructional value or insight into the student’s approach Most automated grading tools rely on dynamic test suites or static analysis and typically provide binary pass/fail outcomes or output comparisons, giving little insight into the student’s approach or conceptual errors Building comprehensive test suites is time-consuming and incomplete test coverage can produce misleading feedback, failing to diagnose root causes or connect errors to underlying concepts Consequently, students often engage in trial-and-error loops—making surface-level changes to pass tests without resolving conceptual misunderstandings These tools, in essence, act as opaque arbiters of correctness, offering minimal educational scaffolding Beyond these limitations, automatic feedback systems also face challenges of trust: students frequently doubt the correctness of the system’s evaluations or the reliability of the feedback provided, which can diminish their confidence in the learning process

Framework Architecture

Autograder+ is architected as a modular, multi-stage pipeline designed to systematically process student submissions, enriching them at each step with layers of analysis The journey of a student’s code through the Autograder+ framework is a structured progression, orchestrated by a series of specialized engines

The core components and data flow are as follows:

1. Code Ingestion: This process begins with the Ingestor module, which reads the source code and links it with the corresponding assignment configuration file

2. Static Analysis Engine: The submission then passes through the Static Analyzer, which performs a fast, low-cost analysis of the code without executing it By parsing the code into an Abstract Syntax Tree (AST), it can efficiently validate syntax, count key structural elements (e.g., number of loops, function definitions), verify adherence to assignment constraints (e.g., presence of a required function name/absence of forbidden libraries), and flag basic anti-patterns This stage provides an immediate structural and syntactic health check

3. Dynamic Execution Engine: Code that is structurally sound proceeds to the Dynamic Analyzer, the crucible to test functional correctness To ensure safety, security, and reproducibility, this engine leverages Docker containers Each test case for a submission is executed in a fresh, isolated container This one-shot container strategy guarantees that each test run is independent and clean The engine meticulously captures the program’s standard output (stdout), standard error (stderr), and exit code, comparing them against the expected outcomes defined in the assignment configuration This stage delivers the definitive ground truth about the code’s runtime behavior

4. The Semantic Core: Following dynamic analysis, the complete submission package is passed to the semantic core, where an LLM is employed to generate feedback This core contains two key sub-modules:

** Embedding Engine:**

This module uses an embedding model to convert the student’s source code into a high-dimensional vector embedding This embedding captures the code’s semantic meaning, abstracting away from surface-level syntax to represent its deeper algorithmic intent

Feedback Engine:

This module orchestrates the generation of pedagogical feedback, taking the code and dynamic analysis results Crucially, it houses the Prompt Pooling mechanism, which enhances the final output The engine then calls the generative model (Base or Fine Tuned) to produce the final textual feedback

5. Reporting and Analytics: This final stage includes a Feedback Generator and an Analytics Engine

Feedback Generator:

This module aggregates all the structured information gathered throughout the pipeline and compiles comprehensive reports, generating an individual Markdown report for each student, presenting the static analysis, a test-by-test breakdown of dynamic results, and the rich, qualitative AI-generated feedback It also creates aggregated summaries

Analytics Engine:

This final engine serves the instructor. It collects the semantic code embeddings from every submission in the class and uses the Uniform Manifold Approximation and Projection (UMAP) algorithm to project them into an interactive 2D scatter plot This visualization provides an intuitive map of the class’s collective problem-solving approaches, transforming raw submission data into actionable pedagogical insights

AI Model Variants and Enhancement

Autograder+ employs two distinct AI modeling strategies to generate feedback, representing a direct approach focused on fine-tuning a generative model and a more sophisticated method of structuring the semantic space through contrastive learning

1. Fine-Tuned LLM: This variant involves taking a pre-trained Large Language Model (LLM), and specializing it through supervised finetuning The model is trained to generate the debugging insight when conditioned on the student code, learning to identify common errors and articulate explanations in a manner that is both conceptually precise and accessible to students While highly effective, this approach treats the code and its performance as input text, without explicitly structuring the underlying semantic relationships between different student solutions

2. Contrastively Fine-Tuned Embedding Model: This variant enhances instructor-facing analytics by fine-tuning the embedding model itself to create a performance-aware semantic space where the geometric arrangement of code embeddings reflects their functional correctness These structured embeddings are then used to drive UMAP visualizations, enabling instructors to identify clusters of correct, partially correct, and incorrect solutions This is achieved by fine-tuning a base code embedding model using a combination of powerful contrastive loss functions The total Multi-Label Supervised Contrastive Loss for the entire batch is calculated as LMulSupCon = 1/I+ sum over i in I+ L i,k, where L i,k is the supervised contrastive loss for anchor i with respect to class k

Feedback Enhancement Mechanism

A key innovation within the Autograder+ framework is the Prompt Pooling mechanism, which enhances the pedagogical quality of feedback from any underlying generative model This technique provides a lightweight and dynamic method for steering the LLM’s focus at inference time, ensuring that its output is not only technically accurate but also aligned with a specific, contextually relevant instructional goal

The mechanism operates as follows:

1. Curate a Prompt Pool: A repository of expert-written instructional prompts is created by instructors or curriculum designers

2. Pre-computation of Prompt Embeddings: At initialization, the framework uses an external embedding generator to compute a high-dimensional vector embedding for every prompt in the pool These prompt embeddings are then cached for efficient retrieval

3. Runtime Code Embedding: When a student submission is processed, the same embedding generator is used to create a vector embedding for the student’s code snippet

4. Semantic Similarity Search: The system calculates the cosine similarity between the student’s code embedding and every cached prompt embedding in the pool This identifies which instructional prompt is most semantically relevant to the student’s specific solution

5. Dynamic Prompt Injection: The prompt corresponding to the highest cosine similarity score is selected as the “bestmatching” instructional focus, and this selected prompt is then appended to the context provided to the generative LLM

Empirical Results

In empirical evaluation across 600 student submissions, Autograder+ achieved an average BERTScore F1 of 0.7658 against TA- written goldstandard feedback, demonstrating that its generated explanations closely align with human instructors To make visualizations meaningful, Autograder+ employs contrastively learned embeddings trained on 1,000 annotated submissions, which organize solutions into a performance-aware semantic space resulting in clusters of functionally similar appraoches

When comparing models without fine-tuning or prompt pooling against enhanced configurations, the most consistent improvements come from prompt pooling For instance, falcon3-10B (Base) achieved the best lexical match (F1 = 0.7658, Precision = 0.

1. textbf Prompt Pooling Mechanism Enhancement: Dynamic Instructional Steering + Specificity Mapping

The system can be improved by expanding prompt curation beyond general concepts to include error type-specific or pedagogical strategy prompts, allowing instructors to dynamically inject the prompt corresponding to the highest cosine similarity score with greater granularity, as described in Section 4.2.

2. textbf Contrastive Fine-Tuning Refinement: Multi-Label Supervised Contrastive Loss Integration

By fully leveraging the Multi-Label Supervised Contrastive Loss (LMulSupCon), the system can create embeddings that are aware of both the problem type (e.g., Fibonacci, Palindrome) and the correctness of the solution, leading to a more performance-aware semantic space.

3. textbf Semantic Space Visualization: Performance-Aware UMAP Clustering

The Analytics Engine can be enhanced by utilizing the contrastively trained embeddings to generate UMAP visualizations that reveal common strategies, misconceptions, and outliers for targeted pedagogical correction, transforming raw data into actionable pedagogical insight.

4. textbf Feedback Grounding: Dynamic Prompt Injection for Conceptual Diagnosis

The Feedback Engine can be more precise by dynamically selecting the prompt most semantically relevant to the student's code embedding, ensuring the final request to the LLM is conceptually targeted and moves beyond binary correctness.

5. textbf Model Selection Optimization: Latency vs. Semantic Quality Trade-off

The system can implement a rigorous selection strategy based on inference time, as noted in Table 4, balancing semantic quality and computational feasibility by favoring models like llama3.2:3b over high-latency reasoning models for practical classroom deployment.

The rapid growth of programming education has outpaced traditional assessment tools, leaving faculty with limited means to provide meaningful, scalable feedback. Conventional autograders, while efficient, act as black-box systems that simply return pass/fail results, offering little insight into student thinking or learning needs. Autograder+ is designed to shift autograding from a purely summative process to a formative learning experience. It introduces two key capabilities: automated feedback generation using a fine-tuned Large Language Model, and visualization of student code submissions to uncover learning patterns. The model is fine-tuned on curated student code and expert feedback to ensure pedagogically aligned, context-aware guidance. In evaluation across 600 student submissions from multiple programming tasks, the system produced feedback with strong semantic alignment to instructor comments. For visualization, contrastively learned code embeddings trained on 1,000 annotated submissions enable grouping solutions into meaningful clusters based on functionality and approach. The system also supports prompt-pooling, allowing instructors to guide feedback style through selected prompt templates. By integrating AI-driven feedback, semantic clustering, and interactive visualization, Autograder+ reduces instructor workload while supporting targeted instruction and promoting stronger learning outcomes.