SCALE: Real-Time Latent Communication for Cyber-Physical Agents

arXiv:2611.08390 · cs.RO, cs.MA · Submitted 2026-11-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "SCALE: Real-Time Latent Communication for Cyber-Physical Agents".

Dev: Here's the gist of the paper. Latent communication enables agents to collaborate efficiently by directly exchanging intermediate model representations rather than raw sensor data or text messages.

Rosa: The authors propose SCALE, a real-time latent communication method tailored for cyber-physical agents.

Dev: Problem Formulation The core problem addressed is the realization of realtime latent communication for CPS agents, where multiple latent messages from different downstream tasks with heterogeneous deadlines share a bandwidth-constrained communication link.

Rosa: The system model involves local encoders producing latent representations and agents competing for shared communication resources under time-varying rates.

Paper summary: Rosa: So we’re looking at this paper called SCALE: Real-Time Latent Communication for Cyber-Physical Agents. Basically, the big idea is that agents should share their internal thinking—their latent representations—instead of sending all the raw data or long text messages back and forth.

Dev: That’s right. The core problem they tackle is making this happen in real time for cyber-physical systems, where you have multiple tasks running, each with a different deadline, all trying to use one shared communication link that only has so much bandwidth.

Taro: So it’s about managing those competing needs under tight communication constraints. How does this system even work on the inside?

Rosa: Well, they propose three main steps for SCALE. First up is semantic change evaluation, where they figure out how much the latent message actually changes compared to what the receiver already knows.

Dev: And that’s important because physical observations evolve continuously, but if you use fixed positions for comparison, it can overstate how much things have moved semantically. They match tokens across messages using similarity scores to find reusable information and avoid sending stuff twice.

Taro: So they are trying to be smart about what information is worth sending versus what’s just noise or redundant data. What’s the next step in this process?

Rosa: Next is utility–cost profiling. Here, they rank the latent tokens based on their semantic change score to see which ones give you the biggest semantic gain for a given communication cost, exploring different transmission sizes.

Dev: That profiling helps them understand that sending more information isn't always better; sometimes it’s not efficient because you hit a point where the cost outweighs the gain. They characterize this trade-off across different transmission sizes to guide what happens next.

Taro: So they are essentially building a map of how much benefit you get for every unit of bandwidth you spend on each potential message?

Rosa: Exactly. And then they use a real-time scheduler to decide which messages get sent and how many tokens to send for each one, all while respecting those different deadlines and the limited link rate.

Paper summary: Dev: The scheduling part is where the real engineering comes in. They use a greedy algorithm that looks at the marginal semantic gain per communication cost, always keeping an eye on feasibility under earliest-deadline-first ordering.

Taro: That sounds tricky because if you send tokens for one message to meet its deadline, you might delay another message that has an even tighter deadline later on. How does the scheduling handle that conflict?

Dev: The system prioritizes based on the gain versus cost ratio, but it’s constrained by the deadlines. The goal is maximizing total semantic gain delivered before each respective deadline while staying within those bandwidth limits.

Rosa: So, overall, SCALE is about using this combination of reusability detection and adaptive sizing to manage that complex communication situation effectively for these cyber-physical agents.

Taro: What does this mean for us in the real world? If we’re talking about robots or autonomous systems operating on limited networks, what’s the practical impact of this idea?

Dev: It means agents can collaborate much more efficiently because they are directly exchanging their intermediate model representations instead of sending massive amounts of raw sensor data or long text messages. That direct exchange saves bandwidth.

Rosa: And it allows them to meet those tight timing requirements that were previously impossible under bandwidth constraints, which is huge for real-time control loops in physical systems.

Taro: So, when you look at the results they showed—like on the OPV2V benchmark where they got best mIoU scores—it suggests this method actually delivers better performance and on-time delivery compared to other ways of communicating.

Dev: They did show significant improvements, for instance, on the OPV2V benchmark at fifteen megabits per second, SCALE achieved the best mean Intersection over Union scores and message on-time rates when compared against other baselines.

Rosa: And in the physical spider robot case study, they saw success rates of seventy-seven point six two percent and ninety-one point six seven percent at one and two megabits per second for detection tasks. That shows it works outside the lab too.

Paper summary: Taro: But we have to remember what they flagged as a limitation on page three. They say that the communication savings depend on spatiotemporal redundancy in those latent representations, and that this redundancy can get reduced by abrupt changes in physical observations, which might hurt performance.

Dev: That’s a valid point about the physical world changing fast; if things move too suddenly, those fixed position comparisons might overestimate how much things have semantically changed. Also, their scheduling algorithm only assesses deadline feasibility using the current link rate and can't handle unexpected bandwidth decreases that happen faster than their minimum scheduling interval.

Rosa: So it’s a powerful method for managing latency and bandwidth when you have multiple tasks competing for resources, but it needs those latent representations to be stable enough, and the scheduler has some limitations on sudden network drops.

Taro: It seems like they’ve put together a system that addresses the core challenge of coordinating diverse agents in real time under real-world communication pressures. It’s a practical tool for making these agents actually work together better.

Dev: The authors, Shanghao Shi, Jian Xiao, Ching-Hsiang Chan, Sanjoy Baruah, Sastry Kompella, Yevgeniy Vorobeychik, William Yeoh and Ning Zhang from Washington University in St. Louis are the ones who put this work out there with SCALE: Real-Time Latent Communication for Cyber-Physical Agents.

Rosa: So to wrap up on this paper, SCALE is a method that combines spatiotemporal latent reuse with utility–cost profiling and deadline-aware scheduling to boost both task utility and message on-time delivery in bandwidth constrained settings.

Dev: It’s an effective approach for meeting the communication and timing requirements of cyber-physical agents because it focuses on how much semantic gain you get for every unit of communication cost while respecting those heterogeneous deadlines.

Taro: The implication is that if we can implement this, we could see better performance in complex, networked physical systems where real-time coordination is critical.

Rosa: That’s what they are pointing toward. It’s about making the communication between these agents smarter and more efficient than just sending everything in bulk.

Conclusion: Rosa: So we’ve been looking at SCALE, which is this method for agents to talk to each other directly using their internal model ideas instead of sending all that raw sensor data back and forth.

Dev: Yeah, it’s about making that happen in real time for these cyber-physical systems, which is tricky when you have many tasks with different deadlines sharing one limited communication link.

Taro: It tackles the problem of scheduling those messages efficiently so they all get through before their respective deadlines, even when the network is busy.

Rosa: The authors are Shanghao Shi and his team at Washington University in St. Louis, and this paper introduces SCALE as a way to combine reusing latent information with smart scheduling to boost both task performance and on-time delivery under bandwidth constraints.

Dev: That’s what it does, using semantic change evaluation, utility cost profiling, and a greedy algorithm for real time scheduling to manage those competing demands.

Taro: It seems like they’ve put together a system that addresses the core challenge of coordinating diverse agents in real time under real-world communication pressures.

Rosa: So what does this mean for us? It suggests that we can get much better collaboration between these physical systems because they’re exchanging their thinking directly, which saves bandwidth and helps meet those tight timing requirements.

Dev: It’s an effective approach for meeting the communication and timing requirements of cyber-physical agents by focusing on how much semantic gain you get for every unit of communication cost while respecting those different deadlines.

Taro: The implication is that if we can implement this, we could see better performance in complex, networked physical systems where real time coordination is critical.

Rosa: That’s what they are pointing toward: making the communication between these agents smarter and more efficient than just sending everything in bulk.

Washington University in St. Louis

cs.RO, cs.MA

Submitted: 2026-11-05

Updated: 2026-11-05

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: Latent communication enables agents to collaborate efficiently by directly exchanging intermediate model representations rather than raw sensor data or text messages.

Key concepts

Latent Communication
Agents communicate by sharing compressed, intermediate model representations (latent messages) instead of sending large amounts of raw sensor data or text. This method allows for more efficient collaboration between different tasks.
Semantic Change Evaluation
This stage measures how much the meaning or semantic content of a latent message changes when sent to a receiver. It establishes correspondences between the current message and the receiver's state to quantify potential gains.
Utility–Cost Profiling
SCALE prioritizes which parts of a message are most important by sorting tokens based on their semantic change score. This helps balance maximizing the semantic gain against the communication cost at different transmission sizes.
Real-Time Scheduling
A greedy algorithm determines the optimal transmission size for each message based on its marginal gain versus communication cost. It ensures that messages meet their deadlines while respecting bandwidth limitations.

Terminology

Summary

Latent communication enables agents to collaborate efficiently by directly exchanging intermediate model representations rather than raw sensor data or text messages. p. 0

Where the paper says it

Louis, 2Nexcepta Inc. Abstract Latent communication enables agents to collaborate efficiently by directly exchanging intermediate model representations rather than raw sensor data or text messages.

The authors propose SCALE, a real-time latent communication method tailored for cyber-physical agents. p. 0
Where the paper says it

Existing approaches, however, are primarily designed for LLM agents and do not consider the bandwidth constraints and heterogeneous task deadlines faced by cyber-physical agents. In this work, we propose SCALE, a real-time latent communication method tailored for cyber-physical agents.

Problem Formulation

The core problem addressed is the realization of realtime latent communication for CPS agents, where multiple latent messages from different downstream tasks with heterogeneous deadlines share a bandwidth-constrained communication link. The system model involves local encoders producing latent representations and agents competing for shared communication resources under time-varying rates. p. 2

Where the paper says it

, 2023), we consider cyber-physical agents that locally encode their observations and exchange latent representations over a communication link. For an observation xi , a local encoder produces a latent representation: Hi = E(xi) = {hi,1, .

The scheduling objective is to maximize the total semantic gain delivered before respective deadlines subject to available bandwidth constraints. p. 2
Where the paper says it

We denote the semantic gain and communication cost of transmitting Ki tokens by Ui(Ki) and Bi(Ki), respectively. Our goal is to determine which pending messages to transmit and how many tokens to transmit for each message, so as to maximize the total semantic gain delivered before their respective deadlines: max {Ki} X i∈Q(t) Ui(Ki)1[Ci ≤ di ] , (2) subject to available bandwidth constraints.

SCALE Approach

SCALE employs three stages: semantic change evaluation, utility–cost profiling, and real-time scheduling. The semantic change evaluation stage first establishes token-level correspondences between the current latent message and the receiver’s acknowledged latent state before quantifying their semantic changes. p. 2

Where the paper says it

A key challenge is that directly comparing two latent messages token by token can be unreliable because their corresponding observations may be misaligned due to object or viewpoint motion. SCALE therefore first establishes token-level correspondences between the current latent message and the receiver’s acknowledged latent state before quantifying their semantic changes.

In the utility–cost profiling stage, SCALE prioritizes tokens with larger semantic change scores by sorting them in descending order according to si,j to characterize the trade-off between semantic gain and communication cost at different transmission sizes. p. 3
Where the paper says it

In this stage, we further characterize the trade-off between semantic gain and communication cost at different transmission sizes K, where K ≤ Ni denotes the number of latent tokens transmitted for message i. Specifically, we prioritize tokens with larger semantic change scores by sorting them in descending order according to si,j .

The real-time scheduling stage uses a greedy algorithm to determine the transmission size Ki for each pending message based on the marginal semantic gain per communication cost while maintaining deadline feasibility under earliest-deadline-first ordering. p. 1
Where the paper says it

Allocating more latent tokens to one message increases its semantic gain but may delay other messages beyond their deadlines. SCALE therefore employs a realtime scheduler that greedily allocates latent tokens based on marginal semantic gain per communication cost, while maintaining deadline feasibility for all selected messages under earliest-deadline-first (EDF) ordering.

Evaluation and Results

SCALE was evaluated across three cyber-physical benchmarks: CooperScene, OPV2V, and LIBERO, alongside a physical spider robot case study under diverse network conditions and heterogeneous task deadlines. The results show that SCALE significantly improves both task performance and on-time delivery in bandwidth-constrained networks. p. 0

Where the paper says it

We evaluate SCALE across three cyber-physical benchmarks and a physical spider robot case study under diverse network conditions and heterogeneous task deadlines. Our results show that SCALE significantly improves both task performance and on-time delivery in bandwidth-constrained networks.

For instance, on the OPV2V benchmark at 15 Mbps, SCALE achieves the best mIoU scores and message on-time rates compared to other baselines. p. 1
Where the paper says it

09 Figure 1: Comparison between SCALE and other latent communication baselines on the OPV2V benchmark at 15 Mbps. SCALE achieves the best mIoU scores and message on-time rates.

In the physical spider robot case study, SCALE substantially outperforms both baselines under constrained bandwidth, achieving success rates of 77. p. 7
Where the paper says it

00% Table 4: Spider robot mini-city case study results under different bandwidth constraints. that SCALE substantially outperforms both baselines under constrained bandwidth.

62% and 91.67% at 1 and 2 Mbps for detection tasks. p. 6
Where the paper says it

To better illustrate real-time performance, Fig. 3 shows the lateness distributions for CooperScene at 20 Mbps for both object detection and trajectory prediction tasks.

Conclusion

The work presents SCALE, a real-time latent communication method that combines spatiotemporal latent reuse, utility–cost profiling, and deadline-aware scheduling to improve both task utility and message on-time delivery under bandwidth constraints. The findings highlight SCALE as an effective and efficient method for meeting the communication and timing requirements of CPS agents. p. 7

Where the paper says it

The results demonstrate improved task performance and high message on-time rates under constrained and varying network conditions. These findings highlight SCALE as an effective and efficient method for meeting the communication and timing requirements of CPS agents.

Limitations

Two limitations are noted: first, the communication savings of SCALE depend on spatiotemporal redundancy in latent representations, which may be reduced by abrupt changes in physical observations. Second, the scheduling algorithm assesses deadline feasibility using the current link rate and cannot handle unexpected bandwidth decreases occurring on timescales shorter than the minimum scheduling interval. p. 8

Where the paper says it

Second, the scheduling algorithm assesses deadline feasibility using the current link rate. However, this design cannot handle unexpected bandwidth decreases that occur on timescales shorter than the minimum scheduling interval, which may affect message timeliness before the scheduler can adapt.

AI Usage Statement

The authors used AI assistants to support the implementation and debugging of the experimental code described in Section 5. They also used AI assistants for grammar checking and revising the manuscript. p. 9

Where the paper says it

, πm) ← messages in S sorted by increasing deadline 24: for q = 1, . , m do 25: Cπq ← t + Xq ℓ=1 8Bπℓ (Kπℓ ) R(t) 26: if Cπq > dπq then 27: return False 28: end if 29: end for 30: return True 31: end function We provide the detailed scheduling procedure used by SCALE in Algorithm 1.

The authors take full responsibility for the experimental results and the final manuscript. p. 3
Where the paper says it

As a result, a larger K generally preserves more new semantic information but incurs greater communication cost. 4 Real-Time Scheduling In the final stage, the scheduler takes as input the utility–cost profiles {(Ui(K), Bi(K))} and task deadlines of all messages in Q(t), together with the current link rate R(t).

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Real-time latent communication for cyber-physical agents can be improved by implementing SCALE to exploit spatiotemporal continuity, which matches tokens across consecutive latent messages using similarity scores to identify reusable information and avoid redundant transmission. This allows agents to directly exchange intermediate model representations efficiently, reducing the communication overhead associated with raw sensor data or text.

  2. A real-time scheduler can be designed that jointly determines which pending messages to transmit and how many latent tokens to send, subject to available bandwidth and task deadlines. This scheduling mechanism maximizes the total semantic gain delivered before their respective deadlines, improving both task utility and message on-time delivery under constrained bandwidth.

  3. Semantic change evaluation can be refined by using a receiver's acknowledged latent state to measure semantic change, where the system performs this by matching current latent tokens against the receiver’s acknowledged latent state and measuring change relative to their best correspondences. This ensures that only information confirmed to be available at the receiver is reused, addressing spatiotemporal variation challenges.

  4. Utility cost profiling can be enhanced by characterizing trade-offs through profiles like (Ui(K), Bi(K)), which characterize the available transmission choices for message i, allowing the scheduler to prioritize tokens based on marginal semantic gain per communication cost (rho i,k). This guides the selection of transmission sizes that balance semantic preservation against communication cost.

  5. The overall AI system can be improved by integrating these stages into a three-stage pipeline: semantic change evaluation, utility cost profiling, and real-time scheduling. This holistic approach allows the system to improve both task utility and message on-time delivery under bandwidth constraints across diverse cyber-physical tasks.

Abstract

Latent communication enables agents to collaborate efficiently by directly exchanging intermediate model representations rather than raw sensor data or text messages. Existing approaches, however, are primarily designed for LLM agents and do not consider the bandwidth constraints and heterogeneous task deadlines faced by cyber-physical agents. In this work, we propose SCALE, a real-time latent communication method tailored for cyber-physical agents. Motivated by the spatio-temporal continuity of the physical world, SCALE matches tokens across consecutive latent messages using similarity scores to identify reusable information and avoid redundant transmission. It then profiles the semantic gain and communication cost of transmitting different amounts of latent information. Based on these profiles, SCALE employs a real-time scheduler that jointly determines which pending messages to transmit and how many latent tokens to send, subject to available bandwidth and task deadlines. We evaluate SCALE across three cyber-physical benchmarks and a physical spider robot case study under diverse network conditions and heterogeneous task deadlines. Our results show that SCALE significantly improves both task performance and on-time delivery in bandwidth-constrained networks. Our code is available at: https://anonymous.4open.science/r/SCALE-4E80/.

Sources

Related papers