Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering

This paper, published at ASE '26 (41st IEEE/ACM International Conference on Automated Software Engineering, October 12–16, 2026, Munich, Germany), presents the first systematic investigation of the LLM-as-a-Judge paradigm for Human-Oriented Binary Reverse Engineering (HOBRE). HOBRE aims to transform decompiled pseudocode into a more human-friendly representation, thereby reducing the cognitive burden of reverse analysis and improving efficiency. The authors are affiliated with the University of Science and Technology of China, Singapore Management University, and the University of Alberta.

The paper addresses a fundamental challenge: reliably evaluating HOBRE outputs remains a fundamental challenge. Human evaluation is costly, time-consuming, as well as difficult to scale. Existing automated metrics either "rely on executable test cases and runtime environments, which are often unavailable for real-world binaries, or depend on high-quality source code references that are typically inaccessible and fail to capture semantically equivalent but lexically diverse outputs."

The authors note that binary reverse engineering is "the cornerstone of security-critical scenarios such as malware analysis, software vulnerability detection, and patch analysis, where reverse analysts often lack access to source code and must confront obscure machine code directly. They explain that due to aggressive compiler optimizations and symbol table stripping, the resulting pseudocode often lacks essential semantic information, such as meaningful function names and developer comments, and suffers from distorted code structures, making reverse engineering still a cognitively demanding and labor-intensive task."

The paper's major contributions are:

• BinJudgeBench Benchmark: "the first high-quality, expert-annotated dataset specifically designed for reference-free HOBRE evaluation. It contains 1,233 meticulously audited samples, providing a reliable ground-truth based on multi-dimensional human judgment."

• Systematic Empirical Study: the first comprehensive study to explore the effectiveness of LLM-as-a-judge for HOBRE evaluation, analyzing its correlation with human experts across three core tasks.

• BinJudge Router: a lightweight adaptive routing mechanism that can dynamically select the optimal judge configuration for each individual sample, thereby improving correlation while reducing costs.

The paper focuses on three complementary tasks spanning identifier-level, natural-language-level, and program-level semantic dimensions:

1. Function Name Recovery (FNR): rename anonymous functions in stripped binaries with descriptive names (e.g., sub 401B20 → handle options), providing direct clues to the program's intent. The evaluation challenge lies in "the inherently highly diverse and style dependence of function naming, as well as the prevalence of domain-specific abbreviations and acronyms, both of which make hard string-matching metrics inadequate for fair assessment."

2. Binary Code Summarization (BCS): generate concise natural-language descriptions that explain the functionality of a given function. Evaluations are challenging because "summarization is a typical open-ended generation problem in which a single function can be described in many semantically equivalent ways, causing traditional n-gram–based matching metrics to over-penalize reasonable rewriting."

3. Decompilation Optimization (DO): post-editing decompiler outputs through transformations including redundancy elimination, variable and type enhancement, and statement restructuring, making the result closer to source code conventions and style. Although Re-executability Rate is commonly used, it only reflects functional correctness and cannot capture non-functional properties such as readability.

The paper identifies three categories of evaluation approaches and their limitations:

• Test-based metrics (e.g., re-executability rate): "require comprehensive unit tests and execution environments, limiting their application to single-function or single-file 'toy programs'. In practical reverse engineering scenarios, complex binary projects often lack corresponding unit test cases, and their environmental dependencies are extremely difficult to simulate, making dynamic verification infeasible."

• Match-based metrics (BLEU, CodeBLEU, etc.): rely on high-quality and accessible reference texts, but "in the most pressing real-world scenarios beyond experimental environments, such as malware analysis and vulnerability discovery in closed-source firmware, source code is often unavailable, rendering evaluation methods based on source-derived references ineffective. Moreover, after compilation optimization and stripping, the source code exhibits a fundamental semantic mismatch with its corresponding binary code."

• Embedding-based metrics (BERTScore, CodeBERTScore, etc.): focus more on lexical similarity than semantic similarity, whereas correct results often exist in various forms that are semantically equivalent but phrased differently.

The paper argues that several attributes make LLMs particularly suitable for HOBRE evaluation: existing research indicates that LLMs exhibit a certain level of binary program comprehension and human-like reasoning abilities; LLMs are typically trained through reinforcement learning from human feedback (RLHF), aligning closely with human judgment preferences; and most importantly, "LLMs excel at handling 'semantically equivalent but diversely expressed' scenarios, while leveraging internal reasoning to directly verify generated artifacts against the code's inherent logic, obviating the dependency on explicit reference text."

Dataset collection: The authors selected 51 representative projects from the GNU repository and compiled them "with GCC 8.2.0 compiler while preserving DWARF debugging information, producing 24 distinct binary variants that span 6 target architectures (ARM 32, ARM 64, X86, X64, MIPS 32, and MIPS 64) and 4 optimization levels (-O0, -O1, -O2, and-O3). IDA Pro was used to decompile both original and stripped binaries; srcML was used to extract developer comments from source code. This produced 346,596 function-level samples."

Response generation: 8 representative models for each task generated candidate outputs, including DL-based binary-specific models (SymLM and HexT5 for FNR; BinT5 and CP-BCS for BCS; LLM4Decompile and SK2 Decompile for DO), general-purpose LLMs (GPT-4o, DeepSeek-V3.2, DeepSeek-Coder-V2-Lite-Instruct, Qwen3-Coder-30B-A3B-Instruct, CodeLlama-34B-Instruct), and the binary-domain LLM ReCopilot. This yielded a total of 2,772,768 (346,596 × 8) candidate responses for each task.

Sampling: Targeting a 95% confidence level with a 5% confidence interval, the authors randomly sampled 385 samples plus 26 supplementary samples to cover diverse combinations of 'generative model—target architecture—optimization level' (totaling 8 × 6 × 4 = 192 triplets), forming a set of 411 samples per task for annotation (1,233 total).

Multi-dimensional evaluation framework: A five-point multi-dimensional Likert scale incorporating shared dimensions applicable to all three HOBRE tasks (Semantic Faithfulness and Analyst-centered Utility) and task-specific dimensions (Naming Distinctiveness & Naturalness for FNR, Information Coverage & Density for BCS, Idiomization & Faithfulness for DO).

Annotation quality: Three authors with over three years of experience in binary reverse engineering served as annotation experts. Inter-annotator agreement measured by ordinal Krippendorff's Alpha reached 0.7996, 0.7077, and 0.6619 for the three tasks, indicating a satisfactory level of agreement. Review meetings were held for samples with disagreement range ≥ 2 (73, 105, and 76 samples across the three tasks), and the mode was used for minor disagreements.

Score distribution: BinJudgeBench covers the full score spectrum from low quality (score 1) to high quality (score 5). FNR received the lowest average score (2.43), BCS the highest (2.81), and DO averaged 2.66 with a clear scarcity of high-scoring samples.

RQ1: LLM-as-a-Judge vs. Traditional Metrics. The study compared 9 backbone LLMs against 11 traditional metrics using Kendall's τ, Spearman's r s, and Pearson's r p correlation coefficients. The LLM-as-a-Judge paradigm achieves an average correlation of 63.20% across all three HOBRE tasks, substantially outperforming the average of traditional metrics (35.04%). On FNR, the best-performing LLM, Gemini-2.5-Flash, reaches a Kendall's τ of 69.02%, nearly 1.4 times that of the best traditional metric, METEOR (49.97%). LLMs performed best on BCS (average Kendall's τ of 62.68%), where surface-level lexical overlap often decouples from actual semantic similarity, and the DO task showed LLMs at 49.74% average τ versus 23.55% for traditional metrics, with the key LLM advantage being their ability to detect 'superficially plausible but logically flawed' refactorings.

On self-evaluation bias, results showed "τself ≈ τothers, with the difference not reaching statistical significance, indicating that in the HOBRE evaluation context, LLMs maintain an objective evaluation scale and do not exhibit any preference toward their own generation style."

Hard samples (HumanScore − avg. LLMScore ≥ 2) comprised 149 out of 1,233 samples (12.08%). Three difficulty patterns emerged: "(1) Lack of semantic anchors (34%): pseudocode lacking residual semantic clues (e.g., hard-coded strings or external API calls) forces LLM judges into speculative scoring. (2) Lack of contextual information (14%): isolated single-function snippets miss caller/callee relations, key global variables, or struct fields required for reliable judgment. (3) Rewarding fluent-but-unsupported outputs (12%): LLMs tend to over-score syntactically fluent or conventional outputs even if semantically unsupported. Regarding the Fluency Trap, in the fluent sub-pool of semantically unsupported samples, the proportion of LLM judge scores >3 rises to 59.0% (compared to 46.7% in the overall unsupported pool)," mainly from the FNR task.

RQ2: Impact of Judge Configurations. The study examined 81 configurations (9 LLMs × 3 prompting strategies × 3 temperatures). Key findings include:

• FSL generally achieves the highest correlation for ultra-large-scale LLMs, but for small- to medium-scale LLMs (e.g., Codestral-2508 and Phi-4), FSL leads to a notable decline in correlation due to anchoring effects; instead, CoT provides a substantial boost for these smaller LLMs, while CoT often leads to a reduction in correlation for ultra-large LLMs in most cases.

• Lower temperatures (0.1 or 0.5) generally yield higher correlation, particularly for ultra-large LLMs, but moderate randomness may help weaker LLMs escape local scoring traps.

• Correlation is positively associated with the LLM scale, accompanied by a marked increase in cost, but cost and performance are not strictly linearly coupled: Gemini-2.5-Flash achieves a 61.36% average correlation at a cost of only 0.131, whereas Claude-3.5-Sonnet requires 2.774 (approximately 21 times higher) to reach a similar correlation of 61.65%.

RQ3: Optimal Configuration Variability. Cross-task rank correlations for performance metrics were only at a moderate level (e.g., the Kendall's τ between BCS and DO is only 52.56%), demonstrating rank flipping across tasks, while the rank correlation for Cost across tasks is extremely high. Sample-level analysis revealed that Static Configuration is far below Oracle Configuration, with a significant 'Oracle Gap' existing across all three tasks — the static best configuration achieved max 70.31% τ versus the oracle upper bound of 92.95% on FNR. This quantitatively proves that using a single static configuration for all samples cannot reach the theoretical upper bound of evaluation performance.

Motivated by the finding that the optimal judge configuration dynamically varies with task types and sample characteristics, the authors propose BinJudge, a lightweight routing mechanism designed for task- and sample-aware adaptive configuration selection.

Architecture: BinJudge takes the stripped pseudocode and the task type as input, and dynamically routes the optimal judge configuration for each sample. It uses a pretrained UniXcoder-base encoder (frozen as a fixed feature extractor), a learnable task embedding layer for cross-task awareness, and a stacked multi-layer perceptron to predict the preference distribution over all 81 judge configurations.

Training objective: Maximize a sample-level utility function:

U i,c = −(MSE i,c + λ · Cost i,c)

with λ = 0.1 controlling the trade-off between correlation and cost, and the cost term logarithmically scaled for normalization. Distribution alignment is used with Softmax temperature T = 0.5, and the training loss is the KL divergence between the target and the predicted distribution, enabling BinJudge not only to learn the top-ranked configuration, but also to capture the relative utility ordering of all 81 configurations.

RQ4: Effectiveness: Using five-fold cross-validation on BinJudgeBench, BinJudge consistently outperforms all Static Configurations across the three HOBRE tasks, improving Kendall's τ by ranges of 4.5%–12.0%, 6.4%–20.4%, and 5.8%–24.7%. Economically, BinJudge's API expenditure is only 0.14×, 0.15×, and 0.25× of Random Configuration costs, and 0.07–0.58×, 0.06–0.81×, and 0.10–0.84× of Static Configuration costs. Even relative to the Oracle Configuration, its cost is only about 2–3 times higher. BinJudge also outperformed fine-tuned UniXcoder and Qwen2.5-Coder-7B judge models, indicating that under the same annotation budget, a fine-tuned judge generalizes poorly, whereas BinJudge leverages complementary prior knowledge from different LLMs and adapts to sample variations via lightweight routing.

The paper analyzes evaluator disagreements across three dimensions:

• LLM vs. Traditional Metrics: discrepancies stem from (1) low-quality or incomplete references reduce the reliability of reference-based metrics; (2) traditional metrics underestimate outputs that are semantically equivalent but lexically different from the reference.

• LLM vs. Human: the three patterns of lack of semantic anchors, lack of contextual information, and rewarding fluent-but-unsupported outputs.

• Human vs. Human: 253 out of 1233 samples had score ranges ≥ 2, mainly occurring on medium-quality outputs due to intrinsic subjectivity. The overlap between LLM-Human and Human-Human disagreement sets was only 23 samples, indicating that the two are largely distinct phenomena.

• Internal: inherent subjectivity of human evaluation; mitigated via a rigorous three-stage protocol: initial calibration, independent scoring, and a consensus meeting for high-variance samples, plus validation by an independent industrial expert.

• External: generalizability; mitigated by covering 51 real-world GNU projects compiled across 6 target architectures and 4 optimization levels and responses from 8 distinct models.

• Construct: whether metrics accurately capture the phenomena; Kendall's τ and Spearman's r s are particularly well-suited for evaluating our ordinal Likert-scale data, with Pearson's r p as supplementary.

The paper concludes that LLM evaluators significantly outperform traditional metrics in approximating human judgment, effectively capturing deep semantic equivalence rather than relying on superficial lexical overlaps, and that BinJudge, "by dynamically selecting the most suitable LLM configuration for each individual sample, substantially improves evaluati…

Improved AI system for reference-free, human-oriented binary reverse engineering (HOBRE) evaluation:

• Semantic, reference-free judging instead of lexical matching. The AI system can evaluate decompiled pseudocode for function name recovery, binary code summarization, and decompilation optimization without access to source code, test cases, or execution environments. It compares generated outputs against the code’s inherent logic and internal reasoning, so semantically equivalent but lexically diverse outputs are not unfairly penalized. This raises average correlation with human experts to 63%, versus 35% for traditional metrics.

• Adaptive per-sample judge routing. The AI system includes a lightweight router that takes stripped pseudocode and task type as input and selects the best judge configuration (LLM, prompting strategy, temperature) for each individual sample, rather than using one static configuration. It improves Kendall’s τ by 4.5–24.7% over static configurations while reducing API costs to 6–84% of static-configuration costs and only 2–3× the theoretical oracle cost.

• Multi-dimensional, human-aligned scoring rubric. The system scores outputs on a 1–5 Likert scale using shared dimensions (semantic faithfulness, analyst-centered utility) and task-specific dimensions (naming distinctiveness/naturalness for function name recovery; information coverage/density for summarization; idiomization/faithfulness for decompilation optimization). This provides fine-grained, interpretable feedback that aligns with expert judgments, with ordinal Krippendorff’s alpha of 0.80, 0.71, and 0.66 across the three tasks.

• Fluency-trap mitigation. The AI system can detect outputs that are syntactically fluent but semantically unsupported by the binary code. It cross-checks generated claims against residual semantic anchors such as hard-coded strings, external API calls, and control-flow structure, reducing over-scoring in fluent-but-unsupported samples. This is critical for catching superficially plausible but logically flawed decompilation refactorings.

• Scale-aware judge configuration. The system can automatically choose different prompting strategies based on judge model size: few-shot learning and low temperature for ultra-large LLMs, and chain-of-thought with moderate temperature for small- to medium-scale LLMs. This avoids anchoring effects and improves correlation while balancing randomness and scoring stability.

• Hard-sample and uncertainty detection. The AI system can flag samples where the pseudocode lacks semantic anchors or contextual information, indicating that the judge is forced into speculative scoring. These samples can be routed to human experts for review, improving reliability in real-world malware analysis, closed-source firmware, and stripped binary reverse engineering tasks.

• Economical high-fidelity evaluation at scale. The improved system can evaluate large volumes of decompiled artifacts (e.g., 346k+ function-level samples) with near-human judgment fidelity, but at a fraction of the cost of human annotation—using the BinJudgeBench ground truth for training and validation, and the BinJudge router to minimize API spend while maximizing correlation.

Human-Oriented Binary Reverse Engineering (HOBRE) aims to transform decompiled pseudocode into a more human-friendly representation, thereby reducing the cognitive burden of reverse analysis and improving efficiency. However, reliably evaluating HOBRE outputs remains a fundamental challenge: human evaluation is costly, time-consuming, and difficult to scale, while existing automated metrics either require executable test cases and runtime environments that are often unavailable for real-world binaries, or rely on high-quality source code references that are typically inaccessible and fail to capture semantically equivalent but lexically diverse outputs. Although LLM-as-a-Judge paradigm is naturally well-suited to HOBRE evaluation, its effectiveness remains underexplored. This paper presents the first systematic investigation of the LLM-as-a-Judge paradigm for HOBRE across three representative tasks: function name recovery, binary code summarization, and decompilation optimization. We introduce BinJudgeBench, the first expert-annotated, reference-free evaluation benchmark based on multi-dimensional human judgment, where LLM-as-a-Judge achieves an average correlation of 63.20% with human judgment, outperforming traditional automated metrics at 35.04%. By analyzing judge configurations across backbone LLMs, prompting strategies, and decoding temperatures, we find that no ``one-size-fits-all'' configuration exists, as the optimal setup varies across tasks and individual samples. To address this, we propose BinJudge, which employs a lightweight routing mechanism to adaptively select the optimal judge configuration for each task and sample. BinJudge improves correlation with human experts by 4.5%-24.7% and reduces API cost to 0.06 times-0.84 times of that of static best configurations, providing a scalable, cost-effective, and high-fidelity automated evaluation scheme for HOBRE.