SAGE-OR: Semi-supervised Adaptive Scene Graph Generation for Operating Rooms

The gist: SAGE-OR, a feature-centric framework that replaces the traditional detect-then-reason paradigm with a decoupled representation-reasoning paradigm in which localization is derived from frozen foundation models, encoded implicitly in pre-computed features, and used without any localization supervision, enables a lightweight 15M-parameter graph transformer that trains in 1.4 hours and runs relational inference at ∼1ms per frame with peak memory under 2GB, suitable for edge hardware used in the operating room

How it works

SAGE-OR proposes a decoupled representation-reasoning paradigm where localization is derived from frozen foundation models and encoded implicitly in pre-computed features, and this approach requires only RGB images and ground-truth relationship and class annotations during training, while preserving zero-shot localization information for downstream tasks The core insight underlying SAGE-OR is that if sufficiently rich instance-level features can be pre-computed using accessible foundation models, then explicit localization supervision becomes unnecessary This design yields several advantages, including significantly reducing dataset construction complexity and improving adaptability to new surgical procedures by providing an incentive to create simple and diverse datasets not currently available

Cache Generation

The pipeline begins with a three-stage decoupled feature caching strategy 1. 3D Scene Representation Cache, which uses Pi3 to obtain dense 3D structure from RGB alone, producing dense point maps that enable cross-view instance fusion 2. Object Segmentation Cache, which segments each camera view independently using SAM3 prompted with general category names to maximize recall and defer role disambiguation to the graph transformer 3. Visual Feature Cache, which uses a hybrid extraction strategy with DINOv3 to extract features for large entities from low-resolution input and small entities from high-resolution crops The complete per-frame cache comprises visual features vi ∈ R d (where d depends on the encoder), geometric features gi ∈ R Dg derived from the instance’s 3D point cloud, and category hints ci ∈ R Nc for each of N instances This per-frame cache is three orders of magnitude smaller than the raw multi-view images it replaces, making them suitable for edge hardware often used in the OR

Scene Graph Generation

The scene graph generation process involves two main stages: Node Classification and Edge Prediction 1. Node Classification, where a learned projection maps the concatenated representation of visual features, 3D centroid, and coarse category hint to a shared embedding space The encoder then contextualizes each node through attention over all other nodes in the scene, augmented with a learned bias ϕ(dij) derived from the pairwise Euclidean distance dij between node centroids 2. Edge Prediction, which requires reasoning over multiple complementary signals to construct a rich pairwise representation for each ordered node pair This representation fuses three sources of information: the pair of node embeddings from the graph transformer, the set of soft node class priors produced by the node head, and pairwise geometry computed from cached geometric features The edge transformer processes these pairwise features through a dedicated edge transformer that enables global consistency reasoning

Semi-supervised Loss

Since SAGE-OR eliminates localization supervision, the model learns correspondence end-to-end through Hungarian matching The total training objective combines three terms: L = λnodeLnode + λedgeLedge + λproxyLproxy The node loss (Lnode) and edge loss (Ledge) are standard cross-entropy losses computed over the matched pairs in M, with the edge loss weighted by the inverse class frequency to handle high class imbalance A proxy loss Lproxy is introduced as an auxiliary supervision term, formulated as a frame-level multi-label binary cross-entropy task to stabilize training for rare relations Unmatched predictions receive no supervision, while unmatched ground-truth entities are treated as missed detections

Results and Adaptability

Extensive experiments demonstrate that SAGE-OR achieves 76% F1 on the 4D-OR benchmark without any localization supervision and with a 4× smaller trainable model, and reaches 86% F1 through unsupervised hand augmentation The addition of unsupervised hand nodes improves average F1 from 76% to 86%, a 10-point gain, particularly for action-oriented relations where hand presence provides disambiguating context This result validates the semi-supervised mechanism and demonstrates a practical mechanism for adaptation without additional annotation effort, as new entity types can be added to the segmentation prompts without corresponding annotations Furthermore, control studies confirm that irrelevant prompts do not improve or degrade performance, confirming that hands provide spatially and semantically disambiguating information for action-oriented relations The decoupled architecture enables feature reuse across tasks and rapid research iteration by requiring only regenerating the cache when visual encoders are swapped

Encoder Analysis

The study provides a systematic evaluation of modern visual encoders for surgical SGG, finding that DINOv3 ViT-L achieves 82% F1, 7–16 points above non-DINOv3 alternatives The best encoder is not the largest, but the one that best balances feature quality with downstream trainability Visual features are essential for entity disambiguation, as removing them causes a catastrophic drop to 53% F1, while geometric features alone cannot substitute for visual identity The final model with all features achieves 80% F1 and trains in approximately 1.4 hours

Conclusion

SAGE-OR presents a scene graph generation framework for operating rooms that decouples visual feature extraction from graph reasoning through offline pre-computation using accessible frozen foundation models By replacing dense multi-modal supervision with a semi-supervised formulation based on Hungarian matching and general prompts, SAGE-OR achieves 76% F1 on the 4D-OR benchmark, rising to 86% with unsupervised hand augmentation The framework is designed to address the challenges of data scarcity and hardware burden by requiring only RGB input and sparse scene graph annotations while preserving spatial grounding This design enables rapid experimentation with architectures and entity configurations needed to develop new technologies The unsupervised context augmentation experiment demonstrates a practical mechanism for adaptation without additional annotations

Discussion and Future Work

SAGE-OR is designed to address the annotation burden and hardware burden of existing datasets by generating 3D structure from arbitrary uncalibrated camera configurations using Pi3 In future work, the team plans to explore lightweight, on-the-fly feature extraction and segmentation to enable end-to-end real-time inference The claim is that competitive accuracy is attainable without the acquisition burden of additional modalities like MM-OR The results demonstrate that high-fidelity foundation features alone provide a sufficient signal for complex relational disambiguation This framework is the first adaptable OR-SGG framework requiring only RGB input and sparse scene graph annotations while preserving spatial grounding The training completes in 1.

1. Improved system can perform scene graph generation in operating rooms using only RGB images and sparse annotations, eliminating dense multi-modal supervision requirements which are currently prohibitively expensive. This enables deployment on existing recording infrastructure without needing specialized hardware like synchronized RGB-D sensors or calibration rigs.

2. Improved system utilizes a decoupled representation-reasoning paradigm in which localization is derived from frozen foundation models, encoded implicitly in pre-computed features, which allows the model to be adapted to new surgical settings via prompt-level modification rather than requiring substantial re-annotation or retraining of detection pipelines.

3. Improved system incorporates unsupervised context augmentation by adding additional prompt-driven entities, such as hands, which are absent from annotations, demonstrating that this mechanism can raise performance to 86% F1 and provides a practical pathway for adaptation to new surgical settings without annotation other than relationship and class labels.

4. Improved system features a feature caching strategy that separates visual encoding from graph reasoning using only RGB input, resulting in an inference speed of ∼1ms per frame with peak memory under 2GB, suitable for edge hardware used in the operating room.

5. Improved system employs a robust multi-view instance fusion technique that clusters masks based on category-constrained 3D IoU and uses an adaptive voxelization scheme to handle Pi3’s scale ambiguity, ensuring that the final set of fused instances per frame, each described by a coarse category, the set of cameras in which it is visible, a 3D centroid in world coordinates, accurately represents physical entities.

6. Improved system leverages an edge-aware attention bias where attention logits are augmented with a learned spatial bias derived from pairwise Euclidean distance dij =∥pi − pj∥2, enabling the model to disambiguate entities that are visually similar but occupy different roles because of their position.