From Formulas to Protocols: A Compositional View of Binary Target Segmentation Metrics
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "From Formulas to Protocols".
Title and authors: Tom: So we’re looking at this paper today called "From Formulas to Protocols: A Compositional View of Binary Target Segmentation Metrics." Jane would say that sounds a bit heavy for something that explains how to build metrics.
Jane: It is, Tom, but what it means is they aren't just giving us new formulas; they’re showing us the whole system under the hood. They take all these different segmentation metrics and break them down into five distinct stages so we can see exactly where the choices are made.
Lu: Exactly. The core idea is that instead of looking at a metric as one big equation, you look at it as a path through five option sets: prediction representation, target extraction, target matching, score computation, and metric reporting.
Meng: So it’s like they’re saying every single segmentation metric is just one specific route through these five choices? That sounds like a way to organize the whole mess of metrics.
Lalam: Right. It decouples the shared operations—like how you represent the prediction—from the specific design choices for each stage, which makes it much clearer what’s actually being changed when you switch from one metric to another >
The paper's summary: Tom: So, what does this decomposition actually look like in practice? Basically, they show how any metric is just a combination of these five decisions. For instance, the way you decide to represent your prediction—whether it’s a soft grayscale map or a fixed threshold—is one stage that you have to choose first >
Jane: That’s right. Then there's target extraction, where you decide if you want to look at the whole map, separated components, or maybe just edges and skeletons of the target area >
Lu: And then target matching comes in. You decide how prediction and ground truth actually correspond—is it pixel-by-pixel matching, or is there some kind of threshold-based deterministic matching involved >
Meng: I mean, that makes sense for understanding why two metrics might look similar on the surface but behave totally differently if they make different choices at any of those points >
Lalam: It really helps map out the design space. They show that current metrics only cover a small slice of what’s possible, and by seeing these five stages, we can see where there are unexplored combinations like soft predictions paired with connected targets >
The paper's improvements: Tom: So what’s the actual improvement here? It’s that it exposes the protocol choices. They take metrics like Mean Absolute Error or Confusion Matrix metrics and show exactly which stage dictates their final formula, so we know where to look for variation >
Jane: That lets us understand *why* a metric is different. For example, if two metrics only differ in how they report the score—that’s Stage five—they might actually have the same raw data sequence but be reported as something completely different like an AUC versus a mean threshold >
Lu: They also point out that by making these choices explicit, we can organize new mechanisms. We can design entirely new things by combining existing options in ways that haven't been explored before, like uncertainty-aware representations or topology units >
Meng: So they’re saying the future of metric design isn't just finding a better formula for MAE; it’s designing a whole new path through these five stages >
Lalam: It suggests that if you want to make comparisons truly meaningful, you have to be upfront about your entire protocol—from the prediction representation all the way down to how you report the final score >
Conclusion: Tom: So, looking at it all together, this paper is really about making sure we aren't just comparing formulas blindly. It’s a framework for seeing segmentation metrics as compositions of modular design choices >
Jane: It gives us a structure to think about what makes one metric different from another. We see that diversity comes from three main kinds of variation: changing how you represent the prediction, changing the entities you are looking for, or changing how you score or report the final result >
Lu: I think this is powerful because it shows that we can systematically explore all those combinations of existing mechanisms rather than just hoping to stumble upon a better isolated metric >
Meng: For us in production, it means if we want a metric that actually matters on real traffic data, we need to be very specific about our prediction representation and target entity choices because the final score depends on those early decisions >
Lalam: Yeah, I think the paper "From Formulas to Protocols: A Compositional View of Binary Target Segmentation Metrics" gives us a much clearer map for where evaluation protocols are incomplete and where we should focus our design efforts next >
Nanyang Technological University
cs.CV
Submitted: 2026-07-01
Updated: 2026-11-17
Code: https://github.com/lartpang/PySODMetrics
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 100/100
The gist: Binary target segmentation metrics are decomposed into five stages covering prediction representation, target extraction, target matching, score computation, and metric reporting <ref:2607.00886#pg6>.
Key concepts
- Framework Decomposition
- This is a five-stage structure used to break down any binary target segmentation metric. It separates shared operations from metric-specific design choices, making it modular. Instead of viewing a metric as one equation, it's seen as one path through five distinct option sets.
- Protocol Choices
- These are the specific decisions a researcher makes at each of the five stages, such as choosing between different ways to represent a prediction (like soft grayscale maps) or how to match predictions to ground truth. Explicitly documenting these choices is key for comparing metrics fairly.
- Metric Interpretation
- The paper argues that apparent differences between metrics often reflect variations in these fundamental design choices across the five stages—representation, entity extraction, matching, scoring, or reporting. This helps researchers understand *why* one metric might perform differently than another.
- Design Space Exploration
- The framework reveals that current metrics only cover a small part of all possible ways to evaluate segmentation. Unused combinations within the five stages suggest many unexplored possibilities, such as using uncertainty-aware representations or reporting on target size distributions.
Terminology
Summary
Binary target segmentation metrics are decomposed into five stages covering prediction representation, target extraction, target matching, score computation, and metric reporting <ref:2607. p. 000886#pg6>. This framework makes metric protocol choices explicit by decoupling shared operations from metric-specific design choices <ref:2607. p. 100886#pg4>.
Framework Decomposition
The proposed five-stage framework decomposes each metric into five stages covering prediction representation, target extraction, target matching, score computation, and metric reporting. This decomposition separates shared evaluation operations from metric-specific design choices. p. 1 The stage view makes metric design modular by showing that a metric is one path through five option sets rather than a single indivisible equation. p. 2
Stage Analysis
The framework analyzes representative metrics as modular compositions where every metric must decide how the prediction is represented, what target is extracted, how prediction and GT are put into correspondence, how the score is computed, and how the final result is reported.
-
Prediction Representation involves choices like Soft Grayscale Map (S1-1), Fixed-Threshold Binary (S1-2), Adaptive-Threshold Binary (S1-3), or Dynamic Threshold Sweep (S1-4). p. 2
-
Target Extraction involves decisions on Target Granularity and Target Type, such as Whole Map/Separated, Region/Edge/Skeleton. p. 2
-
Target Matching specifies correspondence rules, ranging from Pixelwise Matching (S3-1) to Threshold-Based Deterministic Matching (S3-2) or Globally Optimized Matching (S3-3). p. 2
-
Score Computation involves modes like Pixel Error (S4-1), Confusion Matrix (S4-2), Importance Weighting (S4-3), Structural Similarity, Enhanced Alignment, Multiscale AUC, or Correction Effort. p. 2
-
Metric Reporting specifies how the score is presented, such as Raw Data (S5-1) or Threshold-wise Mean/Maximum/AUC for dynamic protocols. p. 2
Metric Interpretation
The analysis shows that apparent metric diversity often reflects different compositions of the same basic design choices. The five-stage mapping separates three kinds of metric variation: representation change, entity change, and scoring or reporting change. p. 6 For instance, metrics differing only at Stage 5 may have the same raw score sequence even if the reported scalar differs. p. 6
Design Space Exploration
The framework reveals that current metrics cover only part of the possible design space. Unused combinations within existing stages imply many rarely explored possibilities, such as keeping a soft prediction while extracting connected targets or reporting distributions over target size. p. 1 Genuinely new mechanisms can be organized by the same five stages, including uncertainty-aware representations or topology units. p. 1
Protocol Requirements
For interpretable comparison, a practical benchmark should document the prediction representation, threshold policy, target entity, matching rule, score formula, empty-case convention, and reporting rule. Protocol-dependent scores require specifying normalization and interpolation rules to ensure reproducibility. p. 9
Conclusion
The five-stage framework organizes binary target segmentation metrics into a structure that makes their protocol choices explicit, allowing formulas, procedural choices, and reported scores to be discussed in the same language. This structured understanding helps researchers identify where current evaluation remains incomplete and outlines directions for future task-aware metric design. p. 1 The framework demonstrates that different metrics respond to different evaluation needs, highlighting that target segmentation quality is not exhausted by overlap alone. p. 9 The diversity observed is interpretable only when the evaluation protocol is clear. p. 0
B Metric Formula Details
The framework details specific metric paths, such as MAE following a simple path of soft prediction, whole-map region target, and pixelwise correspondence. Size-invariant extension SI-MAE [36] changes Stage 2 to separated target-aware regions while keeping the same absolute error principle. p. 6 Confusion-matrix metrics share a path involving binary prediction, whole-map region target, and pixelwise correspondence before deriving scores from TP, FP, TN, and FN. p. 6
B.1 Mean Absolute Error (MAE)
MAE is the most basic expression of pixelwise error by keeping the soft prediction map P and comparing it with the whole-map region target G using spatially aligned pixels as correspondence units. Lower MAE is better, and the value is reported as raw scalar data. p. 5 Size-Invariant Extension SI-MAE [36] balances absolute errors over encoded target regions and the outside-background region instead of averaging all pixels as one whole image. p. 6
B.2 Confusion-Matrix Metrics
Confusion-statistics metrics share a path where Stage 1 supplies a binary prediction, Stage 2 usually extracts one whole-map region target, and Stage 3 keeps pixelwise correspondence. These scores differ mainly in their Stage 4 formula and Stage 5 reporting mechanism. p. 6 Primitive rates like True Positive Rate (TPR) and False Negative Rate (FNR) normalize the four counts by actual target and background populations. p. 6
B.3 Importance-Weighted Metrics
Weighted F-measure [31] keeps the soft whole-map path but changes Stage 4 from uniform counting to importance-weighted error accounting by smoothing the error map with a Gaussian kernel. This preserves the Fbeta interpretation while reducing the assumption that every pixel error has the same perceptual effect. p. 7
B.4 Structural-Similarity Metrics
Structural Similarity Measure (SM) [33] keeps the soft prediction map and whole-map region target, using a structural scoring rule that combines an object-aware term and a region-aware term. The resulting SM rewards coherent target structure rather than only matching the number of target pixels. p. 7 p. 7
B.5 Enhanced-Alignment Metrics
Enhanced Alignment Measure (EM) [34] starts from a binary prediction and measures local-global alignment after centering the prediction and GT by their image-level means. EM enhances this response with a quadratic mapping to strengthen positive alignment before spatial averaging. p. 7
B.8 Multi-Scale IoU (MSIoU)
MSIoU [35] changes Stage 2 from region masks to whole-map edge maps, extracting these using Sobel filters and performing grid-based shrinking at multiple cell sizes. The metric integrates scale-wise target-edge coverage by reusing the primitive TPR from confusion-matrix metrics. p. 8 p. 8
B.9 Human Correction Effort (HCE)
HCE [22] estimates the amount of manual work needed to correct a predicted mask by relaxing false-positive and false-negative regions morphologically. It counts two forms of correction work, such as polygon control points and independent connected error regions, to estimate the effort cost. p. 8
B.10 Context Measure (Cbeta)
Context Measure (Cbeta) [32] explicitly makes contextual dependency explicit by combining forward inference and reverse deduction using a Gaussian context kernel. The weighted extension further incorporates the original RGB image to estimate a camouflage degree map, making the final score depend on both mask agreement and scene context. p. 4
C Additional Reporting and Design-Space Notes
Complementary metrics and explicit protocol descriptions are necessary for making comparisons interpretable when different metrics expose different failure modes. The open design space suggests that new mechanisms can be organized by the same stages, such as topology units or context-conditioned difficulty terms. p. 1 A practical benchmark should document the prediction representation, threshold policy, target entity, matching rule, score formula, empty-case convention, and reporting rule. p. 9 p. 9 The five-stage view is useful only when the changed stage is made explicit. p. 1 p. 1
A Additional Protocol Conventions
Stage 1 assumes that the prediction used for target-only binary evaluation is a normalized target-channel map satisfying 0 ≤ Pij ≤ 1. Fixed-threshold protocols should specify both the cutoff and the boundary convention, for example P > 1/2 or P ≥ 1/2. p. 2 Dynamic-threshold protocols should specify the threshold set T, the ordering of threshold-generated points, and whether Stage 5 reports raw data, a maximum, a mean, or an area summary. p. 9 Area summaries such as AUC and AP depend on the curve envelope and interpolation rule. p. 2
B.2.1 Basic Confusion Quantities
Confusion-matrix metrics are organized around target-background agreement and disagreement quantities, most commonly TP, FP, TN, and FN. For hard partitions, TP counts target pixels predicted as target, FP counts background pixels predicted as target, TN counts background pixels predicted as background, and FN counts target pixels predicted as background. p. 7
B.2.2 Basic Primitive Rates
Four primitive rates normalize the four counts by these two actual populations: True Positive Rate (TPR), False Negative Rate (FNR), True Negative Rate (TNR), and False Positive Rate (FPR).
B.2.3 Receiver Operating Characteristic (ROC) Curve
ROC curve reporting uses dynamic thresholds to trace paired FPR and TPR values, resulting in a raw-curve reporting mode. ROC-AUC changes only the Stage 5 report by replacing the raw curve with the area under it. p. 2
B.3 Precision, Recall, F-measure (Fbeta), and Average Precision (AP)
Precision and recall describe complementary sides of target detection, while Fbeta combines them as a score. Dynamic-Threshold Derivatives evaluate precision, recall, and Fbeta over a threshold-indexed sequence. p. 2 AP summarizes the same PR relation by integrating precision over recall to yield a scalar summary of a threshold-generated curve. p. 5
B.3.1 Intersection-over-Union (IoU) and Dice
IoU measures target overlap relative to target union, while Dice is another widely used overlap score closely tied to IoU. When computed from the same TP, FP, and FN, Dice can be written as an increasing function of IoU. p. 6
B.3.2 Overall Accuracy (OA)
OA measures the proportion of correctly predicted pixels among all pixels, making it a direct whole-image average score. For binary predictions and binary GT, OA equals 1− MAE. p. 6
B.3.3 Balanced Error Rate (BER)
BER [17, 20] is a balance-oriented confusion-matrix metric used in TOS, SD and related binary target segmentation settings where target and background regions can be highly imbalanced. BER gives the target class and the background class equal class-level weight by measuring the error rate of each side separately and then averaging the two errors. p. 6
B.4 Hierarchical IoU (hIoU)
hIoU [37] modifies the whole-map IoU protocol by first parsing connected target components from the thresholded prediction and the GT. Its target matching rule assigns prediction components to GT components by combining overlap qualification, Hungarian assignment with centroid distance as cost, and distance-based compensation. p. 4
B.5 Weighted Fbeta
Weighted F-measure [31] starts from the observation that pixel errors do not always have the same perceptual importance by smoothing the error map with a Gaussian kernel. It computes weighted TP, TN, FP, and FN terms based on spatial dependency and distance from the target. p. 7
B.6 Structure Measure (SM)
SM [33] evaluates target maps through object-aware and region-aware structural similarity instead of reducing the evaluation to pixelwise accuracy. The resulting SM rewards coherent target structure rather than only matching the number of target pixels.
B.7 Enhanced Alignment Measure (EM)
EM [34] measures local-global alignment after centering the prediction and GT by their whole-map means, using a quadratic mapping to enhance positive alignment. EM keeps pixelwise comparison simple while adding global centering to reduce the preference for random noise. p. 7
MSIoU [35] compares detected and GT regions at multiple resolution levels by extracting whole-map edge maps using Sobel filters and performing grid-based shrinking at multiple cell sizes. The metric integrates scale-wise target-edge coverage by reusing the primitive TPR from confusion-matrix metrics.
Complementary metrics and explicit protocol descriptions are necessary for making comparisons interpretable when different metrics expose different failure modes. The open design space suggests that new mechanisms can be organized by the same stages, such as uncertainty-aware representations or context-conditioned difficulty terms. p. 1 A practical benchmark should document the prediction representation, threshold policy, target entity, matching rule, score formula, empty-case convention, and reporting rule. The five-stage view is useful only when the changed stage is made explicit.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
1
The system can be redesigned to explicitly model different evaluation protocols by separating them into five distinct stages: prediction representation, target extraction, target matching, score computation, and metric reporting. This allows researchers to understand exactly how choices like thresholding or target granularity affect the final reported number.
2
The system can utilize a modular approach where new metrics are designed as specific paths through these five stages rather than isolated formulas. This provides a structured way to understand what each metric measures
by making the protocol choices explicit.
3
The system can incorporate task-aware scoring mechanisms at the stage four level, such as using importance weighting or context relevance. For instance, it can compute scores where the quality of a COD prediction depends on the mask and on how the object is visually embedded in its scene.
Sources
- Context-measure: Contextualizing Metric for Camouflage
- Rethinking Evaluation of Infrared Small Target Detection
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs