Real Faults in Deep Learning Fault Benchmarks: How Real Are They?
Researchers manually analyzed 490 faults across five deep learning benchmarks to check their realism. They found that only 18.5% of faults were eligible for study, and successful reproduction was achieved in just 52% of cases. The study highlights significant inconsistencies between reported bugs and their source code or training data, suggesting a need for stricter benchmark curation.