A joint error metric for scoring CFD against wind tunnel data, scaled by the robust spread of the participants (AutoCFD5 Case 2)
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "A joint error metric for scoring CFD against wind tunnel data, scaled by the robust spread of the participants (AutoCFD5 Case 2)".
Jane: Here's the gist of the paper. AutoCFD5 Case 2 compares participants’ CFD results with wind tunnel measurements of surface pressure, velocity profiles and force coefficients The joint error is a fixed weighted sum of five component errors This metric is designed to place each run on a diagram of error against computational cost.
Tom: Defining the Joint Error Metric The joint error is defined as a fixed weighted sum of five component errors.
Jane: The components are the pressure coefficient Cp, the velocity V/V∞, and the force and moment coefficients CD, CL and CM Each component needs a scale to decide how much each component counts in the final metric.
Tom: Establishing Robust Scales The scales are derived by one standard method based on the spread of participants’ results.
Paper summary: Tom: So, wrapping up the "A joint error metric for scoring CFD against wind tunnel data, scaled by the robust spread of the participants (AutoCFD5 Case two)", what we’ve covered is that this paper introduces a single number that combines errors from pressure, velocity, and force coefficients into one weighted sum <ref:2610.11522#pg1,A joint error metric for scoring CFD against wind tunnel data, scaled>
Jane: That number is designed to place simulation runs on an error against computational cost diagram, and it uses robust spread calculations from ISO thirteen thousand five hundred twenty-eight to set the scale for each component <ref:2610.11522#pg1>
Lu: The implication for the research community is that this provides a standardized way to compare CFD results against wind tunnel measurements using a single metric, allowing researchers to see how their methods perform relative to others across different computational costs
Meng: For production deployment, it means we have a more rigorous way of assessing the risk associated with choosing one AI model over another when we have real-world physical validation data to test against
Tom: Exactly. And what this paper does is provide that fixed weighted sum so that everyone is using the same language when they are talking about simulation quality across different tests
Jane: It’s really about moving from checking individual errors to assessing the overall performance profile of a simulation run in a way that respects both accuracy and resource consumption
Lalam: The limitation they mentioned is that the scales are frozen once computed from all submissions received by five October two thousand twenty-six which means we can't easily update those scales if new data comes in later <ref:2610.11522#pg1,submissions received by 5 October 2026>
Lu: And they also state plainly that because there isn't a single agreed-upon accuracy requirement for every geometry, this metric is intended for ranking and plotting, not for making pass or fail verdicts on whether a simulation is correct
Tom: So the final word is that this metric offers a structured way to compare CFD against wind tunnel data by combining component errors with computational cost in a way that’s designed to be reproducible and comparable across different testing efforts
Conclusion: Tom: So we've looked at how they build this joint error metric, and now we're looking at what it actually means for people using these simulations and wind tunnel tests.
Jane: The paper is called "A joint error metric for scoring CFD against wind tunnel data, scaled by the robust spread of the participants," and it’s about taking all those individual simulation errors—pressure, velocity, force—and boiling them down into one single score that shows how good the whole run is.
Lu: It's essentially a fixed weighted sum of five different kinds of errors because they needed a way to combine things that are measured in totally different units.
Meng: What this means for us at the startup level is that we have a standardized way to judge which of our AI models are actually performing well against real physical tests, not just against some idealized computer test.
Lalam: It’s about moving away from looking at just one thing, like maybe just the pressure coefficient, and instead seeing how all those different parts add up to the overall performance.
Tom: And they used these robust spread calculations from ISO thirteen thousand five hundred twenty-eight to set the scale for each component error, which is important because it makes sure that if some runs are way off in one area, that huge deviation gets weighted properly against others.
Jane: So, for a person just listening who isn't reading the math: this metric gives you a single number to compare two different computer simulations against a real wind tunnel test and see which one is more accurate overall.
Lu: It’s designed specifically to put each simulation run on an error versus computational cost diagram, showing you if you get good results for a certain amount of computing power.
Meng: But there are some caveats they mentioned, like those scales being frozen after a certain date, which means the comparison might not be perfectly fresh with the very latest submissions.
Tom: And they also said this metric is for ranking and plotting, not for giving a final pass or fail on whether a design is actually good enough for a specific job.
Jane: So it’s a tool to help you decide which simulation runs are worth looking at more closely, rather than being the final judge of quality.
Lu: It sets up this framework for how we can systematically compare these complex AI models against real-world physics data in a reproducible way.
Upstream CFD GmbH
cs.CE
Submitted: 2026-10-08
Updated: 2026-11-12
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 87/100
The gist: AutoCFD5 Case 2 compares participants’ CFD results with wind tunnel measurements of surface pressure…
Key concepts
- Joint Error Metric
- A single score combining errors from five different CFD output parameters (Cp, V/V∞, CD, CL, CM). This metric is designed to place each simulation run on a graph showing how much error it has relative to the computational cost of the simulation.
- Robust Scales
- The scales used for comparison are not simple averages but are derived from the spread of all participant results using Algorithm A (ISO 13528:2022). This method ensures that large variations in results across participants are proportionally represented, making the metric more reliable.
- Component Error
- The error for each individual parameter (like Cp or CD) is calculated by dividing that run's deviation from the wind tunnel result by a specific scale. This tells you how far off that single measurement is compared to the typical scatter observed in all runs for that specific parameter.
Terminology
Summary
AutoCFD5 Case 2 compares participants’ CFD results with wind tunnel measurements of surface pressure, velocity profiles and force coefficients The joint error is a fixed weighted sum of five component errors This metric is designed to place each run on a diagram of error against computational cost. p. 0 Mockett
Upstream CFD GmbH, Berlin
October 2026
Abstract
AutoCFD5 Case 2 compares the participants’ CFD results with wind tunnel measurements
of surface pressure, velocity profiles and force coefficients. To place each run on a diagram of
error against computational cost, these quantities must be combined into one number.Where the paper says it
Defining the Joint Error Metric
The joint error is defined as a fixed weighted sum of five component errors. The components are the pressure coefficient Cp, the velocity V/V∞, and the force and moment coefficients CD, CL and CM Each component needs a scale to decide how much each component counts in the final metric. p. 0 We call each of these five physical quantities a component: the pressure coefficient Cp, the velocity
V/V∞, and the force and moment coefficients CD, CL and CM. The components differ in kind, in
number of points and in magnitude.Where the paper says it
Establishing Robust Scales
The scales are derived by one standard method based on the spread of participants’ results. This involves calculating point-wise spreads using Algorithm A of ISO 13528:2022, which is a robust standard deviation calculation. p. 2 Since the experimental value is a constant at each point, this equals the spread of the
deviations di
. The robust standard deviation is computed with Algorithm A of ISO 13528:2022, Annex C, which
originates in ISO 5725-5 [3].Where the paper says it
Calculating Component and Joint Errors
The error of a run in component k is its root mean square deviation divided by the pooled scale. The joint error E is calculated as a weighted sum E = ∑k wk ek, where ek = s1/Nk ∑i∈Ik d2iσpt,k. p. 0 The point-wise spreads are pooled into one scale per component, so that every measurement
point carries equal weight within its component. The joint error is a fixed weighted sum
of five component errors. (4)
For a force coefficient this reduces to ek = |d|/σpt, the magnitude of the ISO z score (1). The joint
error is the weighted sum
E = ∑
k
wk ek
, (5)
with the default weights of Table 1.Where the paper says it
Where the paper says it
Design Choices in Metric Construction
The choice to pool the spread instead of using a z score at every point ensures that regions of large scatter count proportionally to the deviations they produce. Pooling the variances rather than averaging spreads is done because it aligns with how variances add and follows the structure where the error is related to the square root of total squared deviation divided by field variance The joint error is preferred over a score derived from R2 because it is linear in deviations within every component, allowing weights to act over the whole range without compressing differences between good runs. p. 5 We use the joint error because it is zero for
perfect agreement and grows in proportion to the deviations. Unlike a score derived from R
2
, it
is linear in the deviations within every component, so the weights act as stated over the whole
range and do not compress the differences between good runs.Where the paper says it
Interpretation and Limitations
A component error ek = 1 means that the run’s RMS deviation from the experiment equals the RMS participant spread for that component. The joint error is intended for ranking and plotting on an error–cost diagram, not for pass or fail verdicts. p. 5 Since E is a
weighted combination of RMS values, these limits give only a rough sense of magnitude. The
joint error is intended for ranking and for the error–cost diagram, not for pass or fail verdicts. 2) notes that criteria set independently of the participants, in particular criteria based
on fitness for an end use, generally have advantages. No such criterion exists for Case 2: there is
no agreed accuracy requirement for surface pressure, velocity and forces on this geometry, and
a hand-picked tolerance would reintroduce hidden weights (Section 1.Where the paper says it
Where the paper says it
Summary of Definitions
The summary defines key parameters such as the assigned value (Wind tunnel result), point-wise spread σpt,i calculated by Algorithm A over all runs at the point, and component scale σpt,k which is the RMS of the point-wise spreads. The component error ek is defined as RMS deviation divided by pooled scale. p. 3 Each variant has its own five scales. 4 Component errors and joint error
The error of a run in component k is its root mean square deviation divided by the pooled scale:
ek =
s
1
Nk
∑
i∈Ik
d
2
i
σpt,k
=
vuuuuut
∑
i∈Ik
d
2
i
∑
i∈Ik
σ
2
pt,i
. Each variant has its own five scales. 4 Component errors and joint error
The error of a run in component k is its root mean square deviation divided by the pooled scale:
ek =
s
1
Nk
∑
i∈Ik
d
2
i
σpt,k
=
vuuuuut
∑
i∈Ik
d
2
i
∑
i∈Ik
σ
2
pt,i
. The point-wise spreads are pooled into one scale per component, so that every measurement
point carries equal weight within its component. The joint error is a fixed weighted sum
of five component errors. The point-wise spreads are pooled into one scale per component, so that every measurement
point carries equal weight within its component. The joint error is a fixed weighted sum
of five component errors. Profiles carry 75 % of the weight and forces 25 %. The
default weights follow the scoring methodology of the AutoCFD5 AI/ML technical focus group
(TFG). The joint error is a fixed weighted sum
of five component errors. The scales are computed once from all AutoCFD5 submissions
received by 5 October 2026 and then frozen, so that scores are reproducible and comparable
across workshops. The joint error is a fixed weighted sum
of five component errors. The scales are computed once from all AutoCFD5 submissions
received by 5 October 2026 and then frozen, so that scores are reproducible and comparable
across workshops. The uncertainty in the scales does not affect rankings within a component, because all runs are
divided by the same value. Between components it acts like an uncertainty of similar size in
the effective weights, which is modest compared with the judgement inherent in the weights
themselves. The uncertainty in the scales does not affect rankings within a component, because all runs are
divided by the same value. Between components it acts like an uncertainty of similar size in
the effective weights, which is modest compared with the judgement inherent in the weights
themselves. The method treats the wind tunnel values as exact. Where the measurement uncertainty u is not small compared with the participants’ spread
(ISO 13528 uses 0. The method treats the wind tunnel values as exact. Where the measurement uncertainty u is not small compared with the participants’ spread
(ISO 13528 uses 0.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Summary of Definitions and Parameters
The summary defines key parameters such as the assigned value (Wind tunnel result), point-wise spread σpt,i calculated by Algorithm A over all runs at the point, and component scale σpt,k which is the RMS of the point-wise spreads. The component error ek is defined as RMS deviation divided by pooled scale. The joint error E is the weighted sum of these component errors. Default weights are specified for Cp (0. p. 3 (4)
For a force coefficient this reduces to ek = |d|/σpt, the magnitude of the ISO z score (1). The joint
error is the weighted sum
E = ∑
k
wk ek
, (5)
with the default weights of Table 1. Profiles carry 75 % of the weight and forces 25 %. The
default weights follow the scoring methodology of the AutoCFD5 AI/ML technical focus group
(TFG).Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved scoring metric for CFD runs by defining a joint error as
a fixed weighted sum of five component errors
to combine pressure coefficient Cp, velocity V/Vinf, and force coefficients CD, CL, and CM into one number for comparison against computational cost. -
The improved system can rank CFD runs on a diagram of error against computational cost by calculating the joint error for each run and variant so that
Runs that no other run beats on both axes form the Pareto front.
-
Error calculation is based on
the robust spread of the participants’ results at that point, estimated robustly with the standard’s Algorithm A,
ensuring deviations are expressed in units of this spread to provide a standardized comparison. -
The system implements a method for calculating participant spread using
Algorithm A of ISO 13528:2022, Annex C,
which involves median calculation, winsorising, and recomputation to handle outliers robustly. -
Scales are computed once from all submissions received by a cutoff date and then frozen so that
scores are reproducible and comparable across workshops,
allowing progress between workshops to be visible. -
Component errors ek are calculated as
its root mean square deviation divided by the pooled scale,
where the pooled scale is theroot mean square of the point-wise spreads
for that component, ensuring equal weight per measurement point within its component. -
The joint error E is a
weighted sum of five component errors
with default weights (e.g., Profiles 75%, Forces 25%), allowing users to adjust these weights in an interactive dashboard to see the effect on rankings.
Abstract
Case 2 of the 5th Automotive CFD Prediction Workshop (AutoCFD5) compares the participants' CFD results with wind tunnel measurements of surface pressure, velocity profiles and force coefficients. To place each run on a diagram of error against computational cost, these quantities must be combined into one number. This note defines that number, the joint error. Following the practice of interlaboratory proficiency testing (ISO 13528), every deviation from the experiment is expressed in units of the spread of the participants' results at that point, estimated robustly with the standard's Algorithm A. The point-wise spreads are pooled into one scale per component, so that every measurement point carries equal weight within its component. Configurations 2a and 2b share these scales and are compared only on the measurements both have, so that their errors can be shown on one diagram. The joint error is a fixed weighted sum of five component errors. The scales are computed once from all AutoCFD5 submissions received by 28 October 2026 and then frozen, so that scores are reproducible and comparable across workshops.
Related papers
- Constrained Sensing and Reliable State Estimation with Shallow Recurrent Decoders on a TRIGA Mark II Reactor
- DREChanNet: Patient-Relative iEEG Model for Channel-Wise Epileptogenic Zone Localization
- A matrix-free, differentiable PyTorch solver for phase-field fracture: Formulation, benchmarks, and inverse analysis
- BizAgentBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business
- Colour me shocked: Exact Molecular Hessians from local MLIPs in O(N) time using sparse differentiation!
- In-context learning as a retraining-free paradigm for constitutive modeling