SCALE: Real-Time Latent Communication for Cyber-Physical Agents
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "SCALE: Real-Time Latent Communication for Cyber-Physical Agents".
Dev: Here's the gist of the paper. Latent communication enables agents to collaborate efficiently by directly exchanging intermediate model representations rather than raw sensor data or text messages.
Rosa: The authors propose SCALE, a real-time latent communication method tailored for cyber-physical agents.
Dev: Problem Formulation The core problem addressed is the realization of realtime latent communication for CPS agents, where multiple latent messages from different downstream tasks with heterogeneous deadlines share a bandwidth-constrained communication link.
Rosa: The system model involves local encoders producing latent representations and agents competing for shared communication resources under time-varying rates.
Paper summary: Rosa: So we’re looking at this paper called SCALE: Real-Time Latent Communication for Cyber-Physical Agents. Basically, the big idea is that agents should share their internal thinking—their latent representations—instead of sending all the raw data or long text messages back and forth.
Dev: That’s right. The core problem they tackle is making this happen in real time for cyber-physical systems, where you have multiple tasks running, each with a different deadline, all trying to use one shared communication link that only has so much bandwidth.
Taro: So it’s about managing those competing needs under tight communication constraints. How does this system even work on the inside?
Rosa: Well, they propose three main steps for SCALE. First up is semantic change evaluation, where they figure out how much the latent message actually changes compared to what the receiver already knows.
Dev: And that’s important because physical observations evolve continuously, but if you use fixed positions for comparison, it can overstate how much things have moved semantically. They match tokens across messages using similarity scores to find reusable information and avoid sending stuff twice.
Taro: So they are trying to be smart about what information is worth sending versus what’s just noise or redundant data. What’s the next step in this process?
Rosa: Next is utility–cost profiling. Here, they rank the latent tokens based on their semantic change score to see which ones give you the biggest semantic gain for a given communication cost, exploring different transmission sizes.
Dev: That profiling helps them understand that sending more information isn't always better; sometimes it’s not efficient because you hit a point where the cost outweighs the gain. They characterize this trade-off across different transmission sizes to guide what happens next.
Taro: So they are essentially building a map of how much benefit you get for every unit of bandwidth you spend on each potential message?
Rosa: Exactly. And then they use a real-time scheduler to decide which messages get sent and how many tokens to send for each one, all while respecting those different deadlines and the limited link rate.
Paper summary: Dev: The scheduling part is where the real engineering comes in. They use a greedy algorithm that looks at the marginal semantic gain per communication cost, always keeping an eye on feasibility under earliest-deadline-first ordering.
Taro: That sounds tricky because if you send tokens for one message to meet its deadline, you might delay another message that has an even tighter deadline later on. How does the scheduling handle that conflict?
Dev: The system prioritizes based on the gain versus cost ratio, but it’s constrained by the deadlines. The goal is maximizing total semantic gain delivered before each respective deadline while staying within those bandwidth limits.
Rosa: So, overall, SCALE is about using this combination of reusability detection and adaptive sizing to manage that complex communication situation effectively for these cyber-physical agents.
Taro: What does this mean for us in the real world? If we’re talking about robots or autonomous systems operating on limited networks, what’s the practical impact of this idea?
Dev: It means agents can collaborate much more efficiently because they are directly exchanging their intermediate model representations instead of sending massive amounts of raw sensor data or long text messages. That direct exchange saves bandwidth.
Rosa: And it allows them to meet those tight timing requirements that were previously impossible under bandwidth constraints, which is huge for real-time control loops in physical systems.
Taro: So, when you look at the results they showed—like on the OPV2V benchmark where they got best mIoU scores—it suggests this method actually delivers better performance and on-time delivery compared to other ways of communicating.
Dev: They did show significant improvements, for instance, on the OPV2V benchmark at fifteen megabits per second, SCALE achieved the best mean Intersection over Union scores and message on-time rates when compared against other baselines.
Rosa: And in the physical spider robot case study, they saw success rates of seventy-seven point six two percent and ninety-one point six seven percent at one and two megabits per second for detection tasks. That shows it works outside the lab too.
Paper summary: Taro: But we have to remember what they flagged as a limitation on page three. They say that the communication savings depend on spatiotemporal redundancy in those latent representations, and that this redundancy can get reduced by abrupt changes in physical observations, which might hurt performance.
Dev: That’s a valid point about the physical world changing fast; if things move too suddenly, those fixed position comparisons might overestimate how much things have semantically changed. Also, their scheduling algorithm only assesses deadline feasibility using the current link rate and can't handle unexpected bandwidth decreases that happen faster than their minimum scheduling interval.
Rosa: So it’s a powerful method for managing latency and bandwidth when you have multiple tasks competing for resources, but it needs those latent representations to be stable enough, and the scheduler has some limitations on sudden network drops.
Taro: It seems like they’ve put together a system that addresses the core challenge of coordinating diverse agents in real time under real-world communication pressures. It’s a practical tool for making these agents actually work together better.
Dev: The authors, Shanghao Shi, Jian Xiao, Ching-Hsiang Chan, Sanjoy Baruah, Sastry Kompella, Yevgeniy Vorobeychik, William Yeoh and Ning Zhang from Washington University in St. Louis are the ones who put this work out there with SCALE: Real-Time Latent Communication for Cyber-Physical Agents.
Rosa: So to wrap up on this paper, SCALE is a method that combines spatiotemporal latent reuse with utility–cost profiling and deadline-aware scheduling to boost both task utility and message on-time delivery in bandwidth constrained settings.
Dev: It’s an effective approach for meeting the communication and timing requirements of cyber-physical agents because it focuses on how much semantic gain you get for every unit of communication cost while respecting those heterogeneous deadlines.
Taro: The implication is that if we can implement this, we could see better performance in complex, networked physical systems where real-time coordination is critical.
Rosa: That’s what they are pointing toward. It’s about making the communication between these agents smarter and more efficient than just sending everything in bulk.
Conclusion: Rosa: So we’ve been looking at SCALE, which is this method for agents to talk to each other directly using their internal model ideas instead of sending all that raw sensor data back and forth.
Dev: Yeah, it’s about making that happen in real time for these cyber-physical systems, which is tricky when you have many tasks with different deadlines sharing one limited communication link.
Taro: It tackles the problem of scheduling those messages efficiently so they all get through before their respective deadlines, even when the network is busy.
Rosa: The authors are Shanghao Shi and his team at Washington University in St. Louis, and this paper introduces SCALE as a way to combine reusing latent information with smart scheduling to boost both task performance and on-time delivery under bandwidth constraints.
Dev: That’s what it does, using semantic change evaluation, utility cost profiling, and a greedy algorithm for real time scheduling to manage those competing demands.
Taro: It seems like they’ve put together a system that addresses the core challenge of coordinating diverse agents in real time under real-world communication pressures.
Rosa: So what does this mean for us? It suggests that we can get much better collaboration between these physical systems because they’re exchanging their thinking directly, which saves bandwidth and helps meet those tight timing requirements.
Dev: It’s an effective approach for meeting the communication and timing requirements of cyber-physical agents by focusing on how much semantic gain you get for every unit of communication cost while respecting those different deadlines.
Taro: The implication is that if we can implement this, we could see better performance in complex, networked physical systems where real time coordination is critical.
Rosa: That’s what they are pointing toward: making the communication between these agents smarter and more efficient than just sending everything in bulk.
Washington University in St. Louis
cs.RO, cs.MA
Submitted: 2026-11-05
Updated: 2026-11-05
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: Latent communication enables agents to collaborate efficiently by directly exchanging intermediate model representations rather than raw sensor data or text messages.
Key concepts
- Latent Communication
- Agents communicate by sharing compressed, intermediate model representations (latent messages) instead of sending large amounts of raw sensor data or text. This method allows for more efficient collaboration between different tasks.
- Semantic Change Evaluation
- This stage measures how much the meaning or semantic content of a latent message changes when sent to a receiver. It establishes correspondences between the current message and the receiver's state to quantify potential gains.
- Utility–Cost Profiling
- SCALE prioritizes which parts of a message are most important by sorting tokens based on their semantic change score. This helps balance maximizing the semantic gain against the communication cost at different transmission sizes.
- Real-Time Scheduling
- A greedy algorithm determines the optimal transmission size for each message based on its marginal gain versus communication cost. It ensures that messages meet their deadlines while respecting bandwidth limitations.
Terminology
Summary
Latent communication enables agents to collaborate efficiently by directly exchanging intermediate model representations rather than raw sensor data or text messages. p. 0 Louis, 2Nexcepta Inc. Abstract
Latent communication enables agents to collaborate efficiently by directly exchanging intermediate model representations rather than
raw sensor data or text messages. Existing approaches, however, are primarily designed for
LLM agents and do not consider the bandwidth
constraints and heterogeneous task deadlines
faced by cyber-physical agents. In this work,
we propose SCALE, a real-time latent communication method tailored for cyber-physical
agents.Where the paper says it
Where the paper says it
Problem Formulation
The core problem addressed is the realization of realtime latent communication for CPS agents, where multiple latent messages from different downstream tasks with heterogeneous deadlines share a bandwidth-constrained communication link. The system model involves local encoders producing latent representations and agents competing for shared communication resources under time-varying rates. p. 2 , 2023), we consider cyber-physical agents
that locally encode their observations and exchange
latent representations over a communication link. For an observation xi
, a local encoder produces a
latent representation:
Hi = E(xi) = {hi,1, . We denote the semantic gain and communication cost of transmitting Ki
tokens by Ui(Ki)
and Bi(Ki), respectively. Our goal is to determine
which pending messages to transmit and how many
tokens to transmit for each message, so as to maximize the total semantic gain delivered before their
respective deadlines:
max
{Ki}
X
i∈Q(t)
Ui(Ki)1[Ci ≤ di
] , (2)
subject to available bandwidth constraints.Where the paper says it
Where the paper says it
SCALE Approach
SCALE employs three stages: semantic change evaluation, utility–cost profiling, and real-time scheduling. The semantic change evaluation stage first establishes token-level correspondences between the current latent message and the receiver’s acknowledged latent state before quantifying their semantic changes. p. 2 A key challenge
is that directly comparing two latent messages token by token can be unreliable because their corresponding observations may be misaligned due to
object or viewpoint motion. SCALE therefore first
establishes token-level correspondences between
the current latent message and the receiver’s acknowledged latent state before quantifying their
semantic changes. In this stage, we further characterize the trade-off between semantic gain and
communication cost at different transmission sizes
K, where K ≤ Ni denotes the number of latent
tokens transmitted for message i. Specifically, we prioritize tokens with larger semantic change scores by sorting them in descending order according to si,j . Allocating more
latent tokens to one message increases its semantic gain but may delay other messages beyond
their deadlines. SCALE therefore employs a realtime scheduler that greedily allocates latent tokens
based on marginal semantic gain per communication cost, while maintaining deadline feasibility for
all selected messages under earliest-deadline-first
(EDF) ordering.Where the paper says it
Where the paper says it
Where the paper says it
Evaluation and Results
SCALE was evaluated across three cyber-physical benchmarks: CooperScene, OPV2V, and LIBERO, alongside a physical spider robot case study under diverse network conditions and heterogeneous task deadlines. The results show that SCALE significantly improves both task performance and on-time delivery in bandwidth-constrained networks. p. 0 We evaluate SCALE across three
cyber-physical benchmarks and a physical spider robot case study under diverse network
conditions and heterogeneous task deadlines. Our results show that SCALE significantly improves both task performance and on-time delivery in bandwidth-constrained networks. 09
Figure 1: Comparison between SCALE and other latent
communication baselines on the OPV2V benchmark at
15 Mbps. SCALE achieves the best mIoU scores and
message on-time rates. 00%
Table 4: Spider robot mini-city case study results under
different bandwidth constraints. that SCALE substantially outperforms both baselines under constrained bandwidth. To better illustrate real-time performance, Fig. 3
shows the lateness distributions for CooperScene
at 20 Mbps for both object detection and trajectory prediction tasks.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
The work presents SCALE, a real-time latent communication method that combines spatiotemporal latent reuse, utility–cost profiling, and deadline-aware scheduling to improve both task utility and message on-time delivery under bandwidth constraints. The findings highlight SCALE as an effective and efficient method for meeting the communication and timing requirements of CPS agents. p. 7 The
results demonstrate improved task performance and
high message on-time rates under constrained and
varying network conditions. These findings highlight SCALE as an effective and efficient method
for meeting the communication and timing requirements of CPS agents.Where the paper says it
Limitations
Two limitations are noted: first, the communication savings of SCALE depend on spatiotemporal redundancy in latent representations, which may be reduced by abrupt changes in physical observations. Second, the scheduling algorithm assesses deadline feasibility using the current link rate and cannot handle unexpected bandwidth decreases occurring on timescales shorter than the minimum scheduling interval. p. 8 Second, the scheduling algorithm assesses deadline feasibility using
the current link rate. However, this design cannot
handle unexpected bandwidth decreases that occur
on timescales shorter than the minimum scheduling interval, which may affect message timeliness
before the scheduler can adapt.Where the paper says it
AI Usage Statement
The authors used AI assistants to support the implementation and debugging of the experimental code described in Section 5. They also used AI assistants for grammar checking and revising the manuscript. p. 9 , πm) ← messages in S sorted by increasing
deadline
24: for q = 1, . , m do
25: Cπq ← t +
Xq
ℓ=1
8Bπℓ
(Kπℓ
)
R(t)
26: if Cπq > dπq
then
27: return False
28: end if
29: end for
30: return True
31: end function
We provide the detailed scheduling procedure
used by SCALE in Algorithm 1. As a result, a larger
K generally preserves more new semantic information but incurs greater communication cost. 4 Real-Time Scheduling
In the final stage, the scheduler takes as input the
utility–cost profiles {(Ui(K), Bi(K))} and task
deadlines of all messages in Q(t), together with the
current link rate R(t).Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Real-time latent communication for cyber-physical agents can be improved by implementing SCALE to exploit spatiotemporal continuity, which
matches tokens across consecutive latent messages using similarity scores to identify reusable information and avoid redundant transmission.
This allows agents to directly exchange intermediate model representations efficiently, reducing the communication overhead associated with raw sensor data or text. -
A real-time scheduler can be designed that
jointly determines which pending messages to transmit and how many latent tokens to send, subject to available bandwidth and task deadlines.
This scheduling mechanism maximizesthe total semantic gain delivered before their respective deadlines,
improving both task utility and message on-time delivery under constrained bandwidth. -
Semantic change evaluation can be refined by using a receiver's acknowledged latent state to measure semantic change, where the system performs this
by matching current latent tokens against the receiver’s acknowledged latent state and measuring change relative to their best correspondences.
This ensures that onlyinformation confirmed to be available at the receiver is reused,
addressing spatiotemporal variation challenges. -
Utility cost profiling can be enhanced by characterizing trade-offs through profiles like
(Ui(K), Bi(K))
, whichcharacterize the available transmission choices for message i,
allowing the scheduler to prioritize tokens based onmarginal semantic gain per communication cost
(rho i,k). This guides the selection of transmission sizes that balance semantic preservation against communication cost. -
The overall AI system can be improved by integrating these stages into a three-stage pipeline: semantic change evaluation, utility cost profiling, and real-time scheduling. This holistic approach allows the system to
improve both task utility and message on-time delivery under bandwidth constraints
across diverse cyber-physical tasks.
Abstract
Latent communication enables agents to collaborate efficiently by directly exchanging intermediate model representations rather than raw sensor data or text messages. Existing approaches, however, are primarily designed for LLM agents and do not consider the bandwidth constraints and heterogeneous task deadlines faced by cyber-physical agents. In this work, we propose SCALE, a real-time latent communication method tailored for cyber-physical agents. Motivated by the spatio-temporal continuity of the physical world, SCALE matches tokens across consecutive latent messages using similarity scores to identify reusable information and avoid redundant transmission. It then profiles the semantic gain and communication cost of transmitting different amounts of latent information. Based on these profiles, SCALE employs a real-time scheduler that jointly determines which pending messages to transmit and how many latent tokens to send, subject to available bandwidth and task deadlines. We evaluate SCALE across three cyber-physical benchmarks and a physical spider robot case study under diverse network conditions and heterogeneous task deadlines. Our results show that SCALE significantly improves both task performance and on-time delivery in bandwidth-constrained networks. Our code is available at: https://anonymous.4open.science/r/SCALE-4E80/.
Sources
- When Semantic Communication Meets Queueing: Cross-Layer Latency and Task Fidelity Optimization
- SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
- Deep Learning Enabled Semantic Communication Systems
- CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation