On Reliability of Membership Inference Vulnerability Evaluation
Evaluating membership inference attack success rates is hard because combining scores from many samples doesn't properly adjust for individual false positive rates. The study shows that naive score concatenation overestimates risk. A post-processing method calibrates these scores, providing a reliable, conservative estimate of the average privacy risk across different models and samples.