SeFaR: Semantic Feature-aware Robustness Testing of Deep Neural Networks

SeFaR: Semantic Feature-aware Robustness Testing of Deep Neural Networks

Abstract

Deep neural networks are increasingly deployed in safety-critical domains as perception modules, where failures are often caused due to rare and under-represented scenarios. This necessitates the need to evaluate the semantic robustness of perception models; conformance of behavior to high-level requirements over real-world perceptual variability. To address this, we propose SeFaR, a framework for systematic semantic-feature-centric testing of vision models. Given a natural-language requirement and a set of satisfying inputs, SeFaR evaluates robustness with respect to diverse realistic semantic variations that preserve requirement satisfaction. The approach employs a novel hierarchical concept model enabling structured exploration of the feature space and incorporation of domain knowledge via user-defined concepts. State-of-the-art diffusion and vision-language models are leveraged to generate photorealistic semantics-preserving perturbations and identification of previously unknown features impacting behavior. A feedback-driven adaptive process is adopted to generate interpretable failure-inducing semantic concepts along with corresponding test inputs. Evaluation on case studies demonstrates that the proposed framework effectively satisfies requirement preconditions while identifying requirement-independent features that influence model decisions, enabling it to both uncover faults and relate them to such features.

Introduction

Deep neural networks (DNNs) are widely deployed in safety-critical systems such as autonomous vehicles, small unmanned aerial systems, and planetary exploration. They aid in several safety-critical tasks such as perception-based collision avoidance, object detection and tracking, trajectory and motion control, terrain assessment and anomaly detection in uncertain environments. Despite strong empirical accuracy, DNNs have been shown to fail in rare, underrepresented, or semantically complex scenarios, leading to unsafe behavior. It is indeed very challenging to validate performance across all relevant scenarios, necessary for safety-critical systems, and this is even more acute in space applications where real-world data collection is extremely expensive and impractical. This motivates the need for systematic testing approaches that support coverage of safety-relevant semantic conditions to help uncover failure-inducing scenarios.

Semantic features are human-interpretable attributes of a scene that influence how it is perceived. They offer a natural abstraction for expressing system requirements and for reasoning about model behavior in human-interpretable terms. In safety-critical domains, it is essential to ensure that the system behavior meets specified requirements. However, this is challenging for systems with perception components that consume image data.

Prior research has proposed a range of testing and analysis techniques to expose failures in DNN models for visual perception. However, many of these approaches focus on pixel-level transformations or variations that aim to exercise internal neuron activations or regions of latent representations, which may not correspond to meaningful changes. Approaches that systematically explore structured scenario-level semantics using scene graphs and simulators are limited in capturing real-world perceptual variability. Hybrid methods combining simulators with generative models improve realism and diversity, however, are constrained by simulator parameterization, limiting their ability to explore unconstrained real-world semantics.

More recently, there has been a surge of techniques leveraging generative models to explore real-world semantic variations. The focus, however, has been on image editing, representation analysis, benchmark construction and augmentation, and post-hoc explainability. They are not designed for systematic robustness evaluation of downstream vision models. The approach in [55] evaluates vision model behavior with respect to requirements expressed in natural language. It leverages text-conditional diffusion to generate photorealistic images satisfying the precondition and checks for behavior conformance to the specified postcondition. However, it does not support systematic assessment of model robustness with respect to unintended semantic variations irrelevant to the requirement. Perception models often rely on spurious correlations with unintended features. Evaluating robustness to such irrelevant semantic variations is crucial for uncovering brittle behaviors which can occur despite requirement satisfaction. The importance of this is amplified by recent evidences of failures in real-world safety-critical systems.

Our Approach

We propose SeFaR, a framework for systematic semantic-feature-centric testing of vision models. Given a natural-language requirement describing desired model behavior of the form pre implies postcondition, and a set of images that satisfy it, the framework evaluates the robustness of the model to perturbations along semantic features that are orthogonal to the preconditions. The key idea is to preserve semantics relevant to precondition satisfaction (such as scene configuration, geometry, spatial relationships, and other task-relevant entities) while systematically generating realistic variations along other semantic dimensions, potentially uncovering rare, subtle, and previously unseen failure-inducing scenarios that may not be represented in available datasets.

SeFaR employs a novel hierarchical concept model that organizes concepts in a top-down structure spanning across multiple levels of abstraction and granularity. This enables structured exploration ranging from variations in coarse factors (such as background, object type) to fine-grained attributes (such as color, pose). Domain expertise is leveraged by allowing users to populate the hierarchy with relevant concepts, thereby enabling alignment of testing with knowledge of the operational domain.

For each concept in the hierarchy, SeFaR leverages a state-of-the-art diffusion model to generate perturbed images that vary the selected feature while preserving rest of the scene semantics. The vision model is evaluated on the perturbed images and the misclassified and correctly classified inputs are analyzed using visual question answering to identify finer-grained semantic factors that correlate with misbehavior. These may correspond to user-specified concepts or to previously unknown semantic features discovered through analysis.

This feedback is used for adaptive exploration of the concept space and targeted test generation to progressively refine the characterization of failure-inducing conditions. The iterative process continues until the relevant depth of the hierarchy is covered, producing a structured set of failure-inducing semantic concepts together with corresponding test inputs.

SeFaR, thereby, presents a novel combination of hierarchical semantic modeling, diffusion-based generation, and feedback-guided exploration, to enable systematic discovery of scenarios which can cause safety-critical failures in real-world deployments.

Illustrative Examples

Consider the SGSM case-study; a collection of real-world autonomous driving datasets with requirements derived from the Virginia Driving Code. Consider the requirement, if a vehicle is within 10 meters, in front, and in the same lane, the ego will not accelerate. The user-defined root-level of the hierarchical model explored the replacement of the vehicle in front of the ego. The images causing failure were analyzed for change with respect to the user-defined attributes vehicle color and vehicle type. The next iteration perturbed original images only for color, and only for type, respectively. Figure 1(a) shows example images leading to misbehavior where only the color of the vehicle was changed. Further exploration of deeper portions of the color feature tree revealed that specific colors, like the changes to blue (Figure 1(b)), green, and teal consistently led to failure-inducing inputs. The color teal was not in the original feature model and was discovered by the analysis. Further exploration of the type feature tree revealed that vehicle change to truck emerged as the feature present in the most failure-inducing inputs (Figure 1(c)).

AI4MARS is a large-scale dataset for terrain-aware autonomous navigation on Mars, collected from NASA's rover missions. We trained a perception model that enables identification of big rocks in the scene and tested this model for the requirement, if the MARS terrain contains big rocks then it should be detected. The image panel in Figure 1(d) highlights that a Dust Storm is a failure-inducing weather condition; a novel scenario not present in the available data.

Contributions

1. A semantic-feature-centric testing framework, SeFaR, that systematically evaluates robustness of vision models with respect to semantic feature variation.

2. A hierarchical concept model that facilitates exploration of a semantic feature space, accommodates user-defined domain knowledge, and supports unspecified features.

3. Diffusion-based generation of requirement-preserving semantic variations to detect model non-robustness to variations in realistic, rare scenarios.

4. Adaptive discovery of feature-oriented failure explanations through VQA-based semantic analysis of failure-inducing images and generation of test sets that witness model feature non-robustness.

5. Evidence of the applicability of SeFaR. Our experiments demonstrate that SeFaR is effective in generating tests that satisfy requirement preconditions while identifying new features that influence model decisions, enabling it to both uncover faults and relate them to such features.

Related Work

There is considerable work in the realm of scenario-level semantics and exploration of scenes with structured entities and relationships (e.g., objects, positions, and interactions). Scenic is a probabilistic programming language for specifying scene configurations. Approaches perform systematic scene generation, testing and falsification with respect to formal requirements in simulated environments. While these enable principled exploration and formal reasoning over the scenario space, they are typically constrained by simulator fidelity and parameterization. Failures in deployment, however, are frequently induced by unexpected real-world variations such as changes in real-world factors like weather, lighting, terrain, geomorphology and other environmental variations, which in turn impact perception model behavior.

Traditional robustness testing of vision DNNs focus on low-level pixel perturbations, global corruptions (blur, brightness, noise) and affine transformations, with limited ability to reason about rich semantic variability. With the advent of generative models, recent approaches leverage them to generate photorealistic images. Diffusion specifically enables stable, high-quality generation of perturbations that are controllable via modalities such as text. Image-editing techniques leverage diffusion and vision-language models to perform fine-grained modifications of visual attributes. Approaches explore the latent space of generative models to identify and manipulate directions to enable controlled variation of visual attributes. Benchmark generation and dataset augmentation approaches produce semantically diverse datasets covering variations in environmental conditions, style, and nuisance factors. Counterfactual explanation generation focuses on generating counterfactual images that alter model predictions, providing localized insights into sensitivity to semantic features. A recent technique systematically generates diverse test images that satisfy a given requirement in natural language.

Table 1 characterizes these approaches in terms of six factors: User Concepts, Concept Model, Diffusion Perturbations, Used for Model Testing, Failure Concepts Discovery, and Adaptive Exploration. SeFaR leverages a user-defined, potentially open, hierarchical concept model that enables specification and iterative exploration of concepts at varying granularities. The concept model can be populated by the user, as well as refined automatically by the discovery of new concepts. It employs diffusion to generate photorealistic image variants that constitute model test inputs that witness non-robustness feature variation. It performs failure analysis to identify features within the user-provided model that best explain failure and discovers novel features not included in the model. Finally, it uses the results of failure analysis to iteratively and adaptively explore finer-grain concepts to produce more accurate feature-based characterizations of failure.

Background

Foundation models are large models pre-trained on massive data to learn generic representations that can be adapted to a wide range of downstream tasks with minimal supervision. Vision–Language Models (VLMs) learn joint representations of images and text, enabling semantic alignment between visual content and natural language descriptions. These models support multimodal reasoning and retrieval tasks such as Visual Question Answering (VQA), which requires them to answer natural-language questions grounded in image content. Generative models learn the underlying data distribution of natural images enabling them to produce photorealistic semantic variation. Diffusion models synthesize images by iteratively learning a sequence of conditional distributions that progressively refine noise into realistic samples. Text-conditional diffusion models leverage multimodal representation to align image synthesis with semantic descriptions specified via natural-language prompts. This enables tasks such as image editing and counterfactual generation guided by natural language descriptions.

Traditional notions of robustness evaluates the consistency of a model's output with respect to low-level perturbations in the input space (such as pixel-level perturbations in images). Robustness to more meaningful semantic features was presented most recently in [55]. This work defines semantic-feature robustness as the consistency of model output to semantic perturbations. This work also defines semantic feature functional requirement as the consistency of model behavior with respect to a pre and postconditions. It generates diverse inputs satisfying the precondition to check conformance of model output with respect to the postcondition. In SeFaR, we extend this work to check the robustness of the model in satisfying the semantic feature functional requirement in the presence of semantic perturbations. Our framework supports a more general, and structured, definition of the feature set which consists of an ontology of features that can be defined by a domain-expert and expanded on during robustness testing by foundation models.

Approach

SeFaR accepts four primary inputs: a neural network under test; a formalized correctness requirement for that network, where when the precondition holds on inputs then the postcondition must hold on corresponding outputs; a set of inputs that satisfy the requirement; and a structured set of semantic features under whose variation the requirement is expected to be invariant. SeFaR produces as output a set of pairs, each comprised of a set of test inputs that violate the requirement and that vary from an input in the dataset with respect to the associated feature. To generate such outputs, SeFaR is parameterized by a feature-perturbation operator that non-deterministically modifies an input with respect to a given semantic feature.

SeFaR consists of three phases that can be run repeatedly. The first selects features to be perturbed within the current round, and in the first round, these are the most general features, typically defined as the roots of the hierarchical model. The second perturbs features present in inputs from the dataset to generate new inputs and checks that the perturbed inputs violate the requirement. This yields a set that drives the third phase. The third phase analyzes the perturbed inputs to detect features that are common to groupings of failing inputs. This detection…

Based on the paper, here are the specific improvements I can make to AI systems and what the improved systems can do:

• Improvement: Integrate a testing layer that systematically perturbs semantic features (e.g., object color, background type, weather conditions) while preserving task-relevant preconditions.

• What the improved system can do: Automatically generate photorealistic test cases that vary human-interpretable attributes (e.g., changing a vehicle from white to teal) without violating the original task requirements (e.g., vehicle is within 10 meters in front). This uncovers spurious correlations and brittle behaviors that pixel-level perturbations miss.

• Improvement: Implement a structured, user-extensible feature tree (e.g., VehicleInFront → Type → Truck, Bus, Sedan ) with open-world placeholders (``) for unknown features.

• What the improved system can do: Enable domain experts to encode operational knowledge (e.g., Mars terrain features like DustStorm, Sunglare) and systematically explore coarse-to-fine feature variations. The system can discover novel features not in the initial ontology (e.g., Teal as a failure-inducing color) and refine explanations iteratively.

• Improvement: Add a closed-loop process that analyzes failure-inducing inputs via vision-language models (VLMs) to identify common semantic attributes, then uses those to guide deeper feature perturbations.

• What the improved system can do: Automatically narrow down root causes of model failures. For example, if changing Background causes failures, the system identifies Lighting and TrafficLightPole as specific culprits, then generates targeted tests for those features, producing precise, actionable failure explanations (e.g., model fails when lighting changes to dusk and a traffic light pole appears).

• Improvement: Use text-conditional diffusion models (e.g., Qwen-Image-Edit) with structured prompts that explicitly constrain edits to be orthogonal to the requirement's preconditions.

• What the improved system can do: Generate diverse, realistic test inputs that satisfy preconditions (93.4% average success rate across datasets) while varying only the target feature. This ensures that detected failures are due to semantic feature non-robustness, not invalid test cases.

• Improvement: Integrate a module that uses VLMs to compare original vs. perturbed images, extracts textual descriptions of differences, embeds them (e.g., via sentence transformers), and clusters them to identify recurring semantic themes.

• What the improved system can do: Automatically discover and name failure-inducing features (e.g., Blue, Green, Truck) with high precision, even when the VLM uses varied linguistic forms (e.g., blue, dark blue). This provides developers with human-readable, actionable insights into why the model fails.

• Improvement: Support perturbation of feature products (e.g., GreenTruck or Lighting+TrafficLightPole) to test combined semantic changes.

• What the improved system can do: Identify synergistic vulnerabilities that individual feature perturbations miss. For example, the system found that changing both Lighting and TrafficLightPole causes 51.3% failures in autonomous driving, whereas each alone causes <30%, revealing critical interaction effects.

1. Proactively uncover hidden failure modes in perception models (e.g., object detection, segmentation, control) before deployment, especially in safety-critical domains like autonomous driving, planetary rovers, and aerial systems.

2. Generate a structured failure report that maps each failure to specific semantic features (e.g., model fails when vehicle color is teal or model fails when dust storm occurs on Mars terrain), along with corresponding test images as evidence.

3. Iteratively refine explanations from coarse (e.g., background changed) to fine-grained (e.g., sky lighting changed to dusk and a building appeared), enabling developers to prioritize fixes.

4. Validate robustness against real-world variability (weather, lighting, object appearance) without requiring expensive real-world data collection, by synthesizing photorealistic perturbations.

5. Adapt to new domains by allowing users to define custom feature hierarchies (e.g., for Mars terrain: Weather → DustStorm, Sunglare ) and automatically discovering novel features (e.g., Teal color) that were not pre-specified.

6. Quantify robustness with metrics like precondition satisfaction rate (93.4%) and failure rate per feature (e.g., 39.7% for green vehicles), providing clear, comparable benchmarks for model improvement.

Deep neural networks are increasingly deployed in safety-critical domains as perception modules, where failures are often caused due to rare and under-represented scenarios. This necessitates the need to evaluate the semantic robustness of perception models; conformance of behavior to high-level requirements over real-world perceptual variability. To address this, we propose SeFaR, a framework for systematic semantic-feature-centric testing of vision models. Given a natural-language requirement and a set of satisfying inputs, SeFaR evaluates robustness with respect to diverse realistic semantic variations that preserve requirement satisfaction. The approach employs a novel hierarchical concept model enabling structured exploration of the feature space and incorporation of domain knowledge via user-defined concepts. State-of-the-art diffusion and vision-language models are leveraged to generate photorealistic semantics-preserving perturbations and identification of previously unknown features impacting behavior. A feedback-driven adaptive process is adopted to generate interpretable failure-inducing semantic concepts along with corresponding test inputs. Evaluation on case studies demonstrates that the proposed framework effectively satisfies requirement preconditions while identifying requirement-independent features that influence model decisions, enabling it to both uncover faults and relate them to such features.