A joint error metric for scoring CFD against wind tunnel data, scaled by the robust spread of the participants (AutoCFD5 Case 2)

arXiv:2610.11522 · cs.CE · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "A joint error metric for scoring CFD against wind tunnel data, scaled by the robust spread of the participants (AutoCFD5 Case 2)".

Jane: Here's the gist of the paper. AutoCFD5 Case 2 compares participants’ CFD results with wind tunnel measurements of surface pressure, velocity profiles and force coefficients The joint error is a fixed weighted sum of five component errors This metric is designed to place each run on a diagram of error against computational cost.

Tom: Defining the Joint Error Metric The joint error is defined as a fixed weighted sum of five component errors.

Jane: The components are the pressure coefficient Cp, the velocity V/V∞, and the force and moment coefficients CD, CL and CM Each component needs a scale to decide how much each component counts in the final metric.

Tom: Establishing Robust Scales The scales are derived by one standard method based on the spread of participants’ results.

Paper summary: Tom: So, wrapping up the "A joint error metric for scoring CFD against wind tunnel data, scaled by the robust spread of the participants (AutoCFD5 Case two)", what we’ve covered is that this paper introduces a single number that combines errors from pressure, velocity, and force coefficients into one weighted sum <ref:2610.11522#pg1,A joint error metric for scoring CFD against wind tunnel data, scaled>

Jane: That number is designed to place simulation runs on an error against computational cost diagram, and it uses robust spread calculations from ISO thirteen thousand five hundred twenty-eight to set the scale for each component <ref:2610.11522#pg1>

Lu: The implication for the research community is that this provides a standardized way to compare CFD results against wind tunnel measurements using a single metric, allowing researchers to see how their methods perform relative to others across different computational costs

Meng: For production deployment, it means we have a more rigorous way of assessing the risk associated with choosing one AI model over another when we have real-world physical validation data to test against

Tom: Exactly. And what this paper does is provide that fixed weighted sum so that everyone is using the same language when they are talking about simulation quality across different tests

Jane: It’s really about moving from checking individual errors to assessing the overall performance profile of a simulation run in a way that respects both accuracy and resource consumption

Lalam: The limitation they mentioned is that the scales are frozen once computed from all submissions received by five October two thousand twenty-six which means we can't easily update those scales if new data comes in later <ref:2610.11522#pg1,submissions received by 5 October 2026>

Lu: And they also state plainly that because there isn't a single agreed-upon accuracy requirement for every geometry, this metric is intended for ranking and plotting, not for making pass or fail verdicts on whether a simulation is correct

Tom: So the final word is that this metric offers a structured way to compare CFD against wind tunnel data by combining component errors with computational cost in a way that’s designed to be reproducible and comparable across different testing efforts

Conclusion: Tom: So we've looked at how they build this joint error metric, and now we're looking at what it actually means for people using these simulations and wind tunnel tests.

Jane: The paper is called "A joint error metric for scoring CFD against wind tunnel data, scaled by the robust spread of the participants," and it’s about taking all those individual simulation errors—pressure, velocity, force—and boiling them down into one single score that shows how good the whole run is.

Lu: It's essentially a fixed weighted sum of five different kinds of errors because they needed a way to combine things that are measured in totally different units.

Meng: What this means for us at the startup level is that we have a standardized way to judge which of our AI models are actually performing well against real physical tests, not just against some idealized computer test.

Lalam: It’s about moving away from looking at just one thing, like maybe just the pressure coefficient, and instead seeing how all those different parts add up to the overall performance.

Tom: And they used these robust spread calculations from ISO thirteen thousand five hundred twenty-eight to set the scale for each component error, which is important because it makes sure that if some runs are way off in one area, that huge deviation gets weighted properly against others.

Jane: So, for a person just listening who isn't reading the math: this metric gives you a single number to compare two different computer simulations against a real wind tunnel test and see which one is more accurate overall.

Lu: It’s designed specifically to put each simulation run on an error versus computational cost diagram, showing you if you get good results for a certain amount of computing power.

Meng: But there are some caveats they mentioned, like those scales being frozen after a certain date, which means the comparison might not be perfectly fresh with the very latest submissions.

Tom: And they also said this metric is for ranking and plotting, not for giving a final pass or fail on whether a design is actually good enough for a specific job.

Jane: So it’s a tool to help you decide which simulation runs are worth looking at more closely, rather than being the final judge of quality.

Lu: It sets up this framework for how we can systematically compare these complex AI models against real-world physics data in a reproducible way.

Upstream CFD GmbH

cs.CE

Submitted: 2026-10-08

Updated: 2026-11-12

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 87/100

The gist: AutoCFD5 Case 2 compares participants’ CFD results with wind tunnel measurements of surface pressure…

Key concepts

Joint Error Metric
A single score combining errors from five different CFD output parameters (Cp, V/V∞, CD, CL, CM). This metric is designed to place each simulation run on a graph showing how much error it has relative to the computational cost of the simulation.
Robust Scales
The scales used for comparison are not simple averages but are derived from the spread of all participant results using Algorithm A (ISO 13528:2022). This method ensures that large variations in results across participants are proportionally represented, making the metric more reliable.
Component Error
The error for each individual parameter (like Cp or CD) is calculated by dividing that run's deviation from the wind tunnel result by a specific scale. This tells you how far off that single measurement is compared to the typical scatter observed in all runs for that specific parameter.

Terminology

Summary

AutoCFD5 Case 2 compares participants’ CFD results with wind tunnel measurements of surface pressure, velocity profiles and force coefficients The joint error is a fixed weighted sum of five component errors This metric is designed to place each run on a diagram of error against computational cost. p. 0

Where the paper says it

Mockett Upstream CFD GmbH, Berlin October 2026 Abstract AutoCFD5 Case 2 compares the participants’ CFD results with wind tunnel measurements of surface pressure, velocity profiles and force coefficients. To place each run on a diagram of error against computational cost, these quantities must be combined into one number.

Defining the Joint Error Metric

The joint error is defined as a fixed weighted sum of five component errors. The components are the pressure coefficient Cp, the velocity V/V∞, and the force and moment coefficients CD, CL and CM Each component needs a scale to decide how much each component counts in the final metric. p. 0

Where the paper says it

We call each of these five physical quantities a component: the pressure coefficient Cp, the velocity V/V∞, and the force and moment coefficients CD, CL and CM. The components differ in kind, in number of points and in magnitude.

Establishing Robust Scales

The scales are derived by one standard method based on the spread of participants’ results. This involves calculating point-wise spreads using Algorithm A of ISO 13528:2022, which is a robust standard deviation calculation. p. 2

Where the paper says it

Since the experimental value is a constant at each point, this equals the spread of the deviations di . The robust standard deviation is computed with Algorithm A of ISO 13528:2022, Annex C, which originates in ISO 5725-5 [3].

The pooled scale for each component is the root mean square of the point-wise spreads across all measurement points

Calculating Component and Joint Errors

The error of a run in component k is its root mean square deviation divided by the pooled scale. The joint error E is calculated as a weighted sum E = ∑k wk ek, where ek = s1/Nk ∑i∈Ik d2iσpt,k. p. 0

Where the paper says it

The point-wise spreads are pooled into one scale per component, so that every measurement point carries equal weight within its component. The joint error is a fixed weighted sum of five component errors.

Default weights are given for Cp (0. p. 3
Where the paper says it

(4) For a force coefficient this reduces to ek = |d|/σpt, the magnitude of the ISO z score (1). The joint error is the weighted sum E = ∑ k wk ek , (5) with the default weights of Table 1.

30), V/V∞ (0.45), CD (0.15), CL (0.05), and CM (0.05).

Design Choices in Metric Construction

The choice to pool the spread instead of using a z score at every point ensures that regions of large scatter count proportionally to the deviations they produce. Pooling the variances rather than averaging spreads is done because it aligns with how variances add and follows the structure where the error is related to the square root of total squared deviation divided by field variance The joint error is preferred over a score derived from R2 because it is linear in deviations within every component, allowing weights to act over the whole range without compressing differences between good runs. p. 5

Where the paper says it

We use the joint error because it is zero for perfect agreement and grows in proportion to the deviations. Unlike a score derived from R 2 , it is linear in the deviations within every component, so the weights act as stated over the whole range and do not compress the differences between good runs.

Interpretation and Limitations

A component error ek = 1 means that the run’s RMS deviation from the experiment equals the RMS participant spread for that component. The joint error is intended for ranking and plotting on an error–cost diagram, not for pass or fail verdicts. p. 5

Where the paper says it

Since E is a weighted combination of RMS values, these limits give only a rough sense of magnitude. The joint error is intended for ranking and for the error–cost diagram, not for pass or fail verdicts.

Limitations include that scales are frozen after a cutoff date to ensure reproducibility across workshops Furthermore, criteria based on fitness for an end use are not included as there is no agreed accuracy requirement for this geometry. p. 5
Where the paper says it

2) notes that criteria set independently of the participants, in particular criteria based on fitness for an end use, generally have advantages. No such criterion exists for Case 2: there is no agreed accuracy requirement for surface pressure, velocity and forces on this geometry, and a hand-picked tolerance would reintroduce hidden weights (Section 1.

Summary of Definitions

The summary defines key parameters such as the assigned value (Wind tunnel result), point-wise spread σpt,i calculated by Algorithm A over all runs at the point, and component scale σpt,k which is the RMS of the point-wise spreads. The component error ek is defined as RMS deviation divided by pooled scale. p. 3

Where the paper says it

Each variant has its own five scales. 4 Component errors and joint error The error of a run in component k is its root mean square deviation divided by the pooled scale: ek = s 1 Nk ∑ i∈Ik d 2 i σpt,k = vuuuuut ∑ i∈Ik d 2 i ∑ i∈Ik σ 2 pt,i .

p. 3
Where the paper says it

Each variant has its own five scales. 4 Component errors and joint error The error of a run in component k is its root mean square deviation divided by the pooled scale: ek = s 1 Nk ∑ i∈Ik d 2 i σpt,k = vuuuuut ∑ i∈Ik d 2 i ∑ i∈Ik σ 2 pt,i .

The joint error E is the weighted sum of these component errors. p. 0
Where the paper says it

The point-wise spreads are pooled into one scale per component, so that every measurement point carries equal weight within its component. The joint error is a fixed weighted sum of five component errors.

p. 0
Where the paper says it

The point-wise spreads are pooled into one scale per component, so that every measurement point carries equal weight within its component. The joint error is a fixed weighted sum of five component errors.

Default weights are specified for Cp, V/V∞, CD, CL and CM based on the AutoCFD5 AI/ML technical focus group scoring methodology. p. 3
Where the paper says it

Profiles carry 75 % of the weight and forces 25 %. The default weights follow the scoring methodology of the AutoCFD5 AI/ML technical focus group (TFG).

The method uses frozen scales computed once from all submissions received by a specific date to ensure scores are reproducible. p. 0
Where the paper says it

The joint error is a fixed weighted sum of five component errors. The scales are computed once from all AutoCFD5 submissions received by 5 October 2026 and then frozen, so that scores are reproducible and comparable across workshops.

p. 0
Where the paper says it

The joint error is a fixed weighted sum of five component errors. The scales are computed once from all AutoCFD5 submissions received by 5 October 2026 and then frozen, so that scores are reproducible and comparable across workshops.

The uncertainty in the scales does not affect rankings within a component because all runs are divided by the same value. p. 6
Where the paper says it

The uncertainty in the scales does not affect rankings within a component, because all runs are divided by the same value. Between components it acts like an uncertainty of similar size in the effective weights, which is modest compared with the judgement inherent in the weights themselves.

p. 6
Where the paper says it

The uncertainty in the scales does not affect rankings within a component, because all runs are divided by the same value. Between components it acts like an uncertainty of similar size in the effective weights, which is modest compared with the judgement inherent in the weights themselves.

The method treats wind tunnel values as exact, penalizing runs for differences the measurement cannot resolve if measurement uncertainty is large. p. 6
Where the paper says it

The method treats the wind tunnel values as exact. Where the measurement uncertainty u is not small compared with the participants’ spread (ISO 13528 uses 0.

p. 6
Where the paper says it

The method treats the wind tunnel values as exact. Where the measurement uncertainty u is not small compared with the participants’ spread (ISO 13528 uses 0.

Summary of Definitions and Parameters

The summary defines key parameters such as the assigned value (Wind tunnel result), point-wise spread σpt,i calculated by Algorithm A over all runs at the point, and component scale σpt,k which is the RMS of the point-wise spreads. The component error ek is defined as RMS deviation divided by pooled scale. The joint error E is the weighted sum of these component errors. Default weights are specified for Cp (0. p. 3

Where the paper says it

(4) For a force coefficient this reduces to ek = |d|/σpt, the magnitude of the ISO z score (1). The joint error is the weighted sum E = ∑ k wk ek , (5) with the default weights of Table 1.

30), V/V∞ (0.45), CD (0.15), CL (0.05), and CM (0.05) based on the AutoCFD5 AI/ML technical focus group scoring methodology. p. 3
Where the paper says it

Profiles carry 75 % of the weight and forces 25 %. The default weights follow the scoring methodology of the AutoCFD5 AI/ML technical focus group (TFG).

The method uses frozen scales computed once from all submissions received by a specific date to ensure scores are reproducible. The uncertainty in the scales does not affect rankings within a component because all runs are divided by the same value. The method treats wind tunnel values as exact, penalizing runs for differences the measurement cannot resolve if measurement uncertainty is large.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved scoring metric for CFD runs by defining a joint error as a fixed weighted sum of five component errors to combine pressure coefficient Cp, velocity V/Vinf, and force coefficients CD, CL, and CM into one number for comparison against computational cost.

  2. The improved system can rank CFD runs on a diagram of error against computational cost by calculating the joint error for each run and variant so that Runs that no other run beats on both axes form the Pareto front.

  3. Error calculation is based on the robust spread of the participants’ results at that point, estimated robustly with the standard’s Algorithm A, ensuring deviations are expressed in units of this spread to provide a standardized comparison.

  4. The system implements a method for calculating participant spread using Algorithm A of ISO 13528:2022, Annex C, which involves median calculation, winsorising, and recomputation to handle outliers robustly.

  5. Scales are computed once from all submissions received by a cutoff date and then frozen so that scores are reproducible and comparable across workshops, allowing progress between workshops to be visible.

  6. Component errors ek are calculated as its root mean square deviation divided by the pooled scale, where the pooled scale is the root mean square of the point-wise spreads for that component, ensuring equal weight per measurement point within its component.

  7. The joint error E is a weighted sum of five component errors with default weights (e.g., Profiles 75%, Forces 25%), allowing users to adjust these weights in an interactive dashboard to see the effect on rankings.

Abstract

Case 2 of the 5th Automotive CFD Prediction Workshop (AutoCFD5) compares the participants' CFD results with wind tunnel measurements of surface pressure, velocity profiles and force coefficients. To place each run on a diagram of error against computational cost, these quantities must be combined into one number. This note defines that number, the joint error. Following the practice of interlaboratory proficiency testing (ISO 13528), every deviation from the experiment is expressed in units of the spread of the participants' results at that point, estimated robustly with the standard's Algorithm A. The point-wise spreads are pooled into one scale per component, so that every measurement point carries equal weight within its component. Configurations 2a and 2b share these scales and are compared only on the measurements both have, so that their errors can be shown on one diagram. The joint error is a fixed weighted sum of five component errors. The scales are computed once from all AutoCFD5 submissions received by 28 October 2026 and then frozen, so that scores are reproducible and comparable across workshops.

Related papers