Measuring Duplication, Leakage, and Rendering Dependence in Malware-as-Image Benchmarks

arXiv:2610.39351 · cs.CR · Submitted 2026-10-29 · Read on arXiv

cs.CR

Submitted: 2026-10-29

Updated: 2026-10-29

License: http://creativecommons.org/licenses/by/4.0/

The gist: Classifying malware binaries rendered as images is a popular approach to malware family identification, and papers on the standard benchmarks for this task report accuracies of 97-99%.

Terminology

Abstract

Classifying malware binaries rendered as images is a popular approach to malware family identification, and papers on the standard benchmarks for this task report accuracies of 97-99%. In this paper we audit ten of these benchmarks for near-duplicate images and test-train leakage, and retrain standard classifiers under naive and content-aware splits. Deduplicating by file hash, which is the standard practice, does not prevent leakage, because distinct binaries render to identical images. Deduplicating on image content reduces top-1 accuracy by up to 8 points and macro-F1 by up to 15 points on the released benchmarks we find leaky. The rendering step also adds duplication, since byteplots of the same binaries contain 19.7% near-duplicates where static feature vectors contain 2.2%, and changing the width at which bytes are wrapped into rows reduces accuracy by 20 to 33 points, although no byte changes. We recommend that content-level duplication and a non-image baseline be reported alongside accuracy.

Related papers