Multimodal Remote Inference

arXiv:2508.07555 · cs.LG, cs.IT, cs.NI, math.IT · Submitted 2025-08-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Multimodal Remote Inference".

Jane: The gist The proposed work addresses how to design an effective and efficient modality scheduling policy to minimize ML remote inference error when features from multiple remote sensors evolve dynamically…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's talk about the title and who wrote this, "Multimodal Remote Inference." It’s not just about sending data; it’s about making smarter decisions when you have multiple data streams feeding an AI model remotely.

Jane: The authors are Keyuan Zhang, Yin Sun, and Bo Ji from Virginia Tech and Auburn University. They are diving into how to handle the complexity of scheduling these sensor inputs for real-time inference tasks.

Lu: They start by defining the core challenge: when sensor observations evolve dynamically over time, fresh features are critical for inference tasks but getting them all on time is often impossible due to limited network resources.

Meng: So they're setting up a scenario where you have multiple modalities, say vision and radar, and you need a smart policy to decide which one to pull next to keep the AI model accurate.

Tom: The core of their approach is formulating this as an SMDP to minimize that inference error function, which they define as being dependent on the AoI vector at time t.

Jane: They explain that this AoI vector, denoted by Delta(t) with M modalities, is what really complicates things because the function L(Delta(t)) describing the error can be very general—non-monotonic and non-convex.

Lu: This complexity is important because it means standard approaches for scheduling data might not work well when you have these kinds of correlated sources.

Meng: So, what's the big deal here for us in production? If our inference error depends on this whole vector, a simple rule like sending the freshest piece of data from any one sensor might not be the best move.

Tom: Right, so they immediately tackle that by doing an equivalent reformulation with a reduced state set, which is a clever trick to avoid the curse of dimensionality.

Jane: They prove that this reformulation still maintains optimality while drastically simplifying the state space—reducing the exponent from M to M minus two in their formulation <ref:2508.07555#pg2>.

Lu: That reduction in state size is a big deal for computation, and they show that the chain structure of this problem changes completely between just two modalities and when you have more than two.

Meng: So for a system with just two sensors, it’s simpler than for three or four? That means the math might be tractable there.

Tom: It does; in the two-modality case, they find that the optimal policy has an index-based threshold structure, which is a very specific way to decide when to switch modalities.

Jane: That’s a concrete thing we can look at—a rule based on an index and a threshold—which helps guide the decision-making process directly.

The paper's summary: Tom: Moving into the summary of "Multimodal Remote Inference," they lay out their main contributions by focusing on how they handle that complicated AoI vector optimization.

Jane: They start by showing that for two modalities, the optimal policy is an index-based threshold structure, which helps determine when to switch between those two sensors.

Lu: But when you go beyond two modalities, say M greater than two the problem becomes multichain, and they develop the optimal error-aware switching-and-transmission policy called EAST <ref:2508.07555#pg2>.

Meng: EAST sounds like it’s the full solution for the general case, but how do you actually compute a policy that complex? That’s where I get skeptical about deployment.

Tom: They compute EAST using a multichain policy iteration algorithm, or MPI, which is necessary because the problem structure is fundamentally different when there are more than two modalities.

Jane: They also propose two lower-complexity options for when computational resources are tight: the error-aware transmission policy, EAT, and the fixed threshold policy, FT.

Lu: EAT and FT are presented as low-complexity policies that can be computed via MPI under specific settings, which is something they did to make the optimal solution more accessible.

Meng: So if I’m deploying this on an edge device with limited compute power, I’d probably look at EAT or FT first because EAST sounds like it needs a lot of overhead.

Tom: The paper shows that these proposed policies outperform simple heuristics like round-robin or greedy policies across three case studies involving two, three, and five modalities.

Jane: They show that EAST reduces the inference error by up to forty-four point eight percent compared to the best baseline they tested in each of those scenarios.

Lu: It’s interesting because it suggests that even when dealing with a general function of the AoI vector, you can still get significant performance gains by being smart about scheduling.

The paper's improvements: Tom: Let's look at the specific improvements they suggest for this multimodal scheduling problem. They focus on moving from the original problem to these structured policies.

Jane: For the two-modality case, they establish that an index-based threshold policy is optimal, which gives a clear rule for when to switch modalities based on some index function gamma m of theta exceeding a threshold g.

Lu: That index-based structure is really powerful because it means you don't just look at the raw AoI; you’re using this specific function to guide the decision over consecutive transmissions.

Meng: If we have two sensors, knowing that there’s an optimal threshold structure gives us a very concrete starting point for testing and tuning our system on a small scale.

Tom: Then they introduce EAST, the error-aware switching-and-transmission policy, as the optimal solution for the general multi-modality case when M is greater than two.

Jane: EAST is computed using MPI, but it’s shown to be optimal for that more complex scenario where you have many modalities interacting with a general AoI vector function.

Lu: They also developed EAT and FT as low-complexity alternatives under specific settings, which means we don't always have to run the full MPI algorithm if we can simplify the setup.

Meng: I’m interested in how much computational time these EAT and FT versions actually save compared to the full EAST computation, because that’s what matters for real-time deployment.

Tom: The numerical evaluation shows that while EAST is high-performing, low-complexity policies like FT can reduce computation time by about three thousand times relative to EAST in a five-modality case.

Jane: That massive reduction in computation time is a huge win for edge devices, even if it means the inference error increases by thirty-eight point six percent when using FT in that five-modality test.

Lu: The paper also points out that their state-space reduction helps computational efficiency too; they reduce the state set from K times M states down to two times M minus one times K minus two states as the modality count increases <ref:2508.07555#pg2>.

Conclusion: Tom: So, to wrap up this paper on "Multimodal Remote Inference," they establish an optimal index-based threshold policy for the two-modality case and the error-aware switching-and-transmission policy, EAST, for the general multi-modality case.

Jane: They also provide low-complexity alternatives like EAT and FT which are computationally efficient when you need to deploy on resource constrained systems.

Lu: The key point they drive home is that minimizing AoI alone isn't optimal; performance depends on complex combinations of AoIs across different modalities, as shown by the empirical error function results.

Meng: So for those of us building systems, it means we can choose a policy based on our resource constraints and whether we are dealing with two or many sensors.

Tom: They successfully reduced the state set while keeping optimality, which is a major achievement because it addresses how to handle that complexity efficiently in the first place.

Jane: Overall, this paper gives us concrete policies—the index-based one for two sources and EAST for more—and low-complexity options when we need speed over absolute peak performance.

Lu: It sets the stage nicely by showing how to tackle non-monotonic AoI functions and how state reduction helps simplify the policy iteration process significantly.

Meng: I’m curious about future work, but right now, the implication is that we have a solid framework for balancing accuracy and computational cost in these multimodal scheduling problems.

Tom: They suggest future work involving extending this communication model to unreliable channels and exploring reinforcement learning approaches to see if those methods can handle the dynamic nature better.

Jane: This paper on "Multimodal Remote Inference" gives us a lot of tools, whether we need the optimal policy for two sensors or a fast approximation for five sensors.

Department of Computer Science, Virginia Tech · Department of Electrical and Computer Engineering, Auburn University

cs.LG, cs.IT, cs.NI, math.IT

Submitted: 2025-08-11

Updated: 2026-10-14

Comments: Submitted to IEEE/ACM Transactions on Networking; a preliminary version appeared in the Proceedings of the 22nd IEEE International Conference on Mobile Ad-Hoc and Smart Systems (MASS 2025)

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 82/100

The gist: The gist The proposed work addresses how to design an effective and efficient modality scheduling policy to minimize ML remote inference error when features from multiple remote sensors evolve

Key concepts

Age of Information (AoI) Vector: This vector represents the freshness of data from every sensor in a system at a specific time. It is a collection of how long it has been since the most recent piece of information was received from each modality. Minimizing this vector's values is key to keeping the ML model's inference accurate.
Semi-Markov Decision Process (SMDP): This mathematical framework is used to model the scheduling problem. It describes a system where decisions are made at discrete time steps, and the future state depends on both the current state (the AoI vector) and the action taken (which modality to transmit next).
Multi-chain Policy Iteration (MPI): This is an algorithm used to find the optimal switching policy for systems with multiple modalities. It iteratively refines a policy by considering all possible sequences of decisions across different chains or modalities, helping to determine the best way to switch between sensors.
Inference Error Function L(∆(t)): This function quantifies how wrong the machine learning model's predictions are, and it depends on the entire AoI vector. Because this function can be complex and non-linear, simply minimizing the age of information for one sensor is not enough; the scheduling must consider all modalities together.

Terminology

Age of Information — 29× in this paper
Error-Aware Transmission Policy — 10× in this paper
Fixed Threshold Policy — 7× in this paper
Reinforcement Learning — 6× in this paper · explained in 4 episodes
Curse of Dimensionality — 5× in this paper · explained in 6 episodes
Core — 4× in this paper · explained in 2 episodes
Loss Function — 3× in this paper · explained in 2 episodes
Accuracy — 2× in this paper · explained in 1 episode
Error-Aware Switchingand-Transmission Policy — 2× in this paper
IEEE International Symposium on Information Theory — 2× in this paper
Multimodal Learning — 2× in this paper · explained in 2 episodes
Actor — 1× in this paper · explained in 3 episodes
Artificial Intelligence (AI) — 1× in this paper · explained in 2 episodes
Digital Twin — 1× in this paper · explained in 5 episodes
Edge Devices — 1× in this paper · explained in 2 episodes
Feature Engineering — 1× in this paper · explained in 3 episodes
Long Short-Term Memory — 1× in this paper
Objective Function — 1× in this paper · explained in 2 episodes
Resource Constraints — 1× in this paper · explained in 4 episodes
Signal Processing Advances in Wireless Communications — 1× in this paper
Trace — 1× in this paper · explained in 2 episodes

Summary

The gist The proposed work addresses how to design an effective and efficient modality scheduling policy to minimize ML remote inference error when features from multiple remote sensors evolve dynamically over time <ref:2508.07555#pg2>.

Problem Formulation and Reformulation

The core problem is formulated as a semi-Markov decision process (SMDP) where the objective is to minimize the ML model’s inference error, which is expressed as a general function of the Age of Information (AoI) vector <ref:2508.07555#pg4>. To mitigate computational expense due to the curse of dimensionality, an equivalent reformulation with a reduced state set is derived, reducing the exponent from M to M − 2 without sacrificing optimality <ref:2508.07555#pg5>. The state at each decision epoch is defined as the AoI vector ∆(Si) and the decision is the modality choice di.

Chain Structure Analysis

The chain structure of the reformulated problem fundamentally differs based on the number of modalities. Specifically, for the two-modality case (M = 2), the SMDP is unichain, which allows for a closed-form solution with an index-based threshold structure. Conversely, when M > 2, the problem is multichain, necessitating the use of multichain policy iteration (MPI) to compute the optimal policy.

Proposed Scheduling Policies

The paper develops several policies tailored to different complexity levels.

Index-based threshold policy: This is the optimal solution for the two-modality case, characterized by selecting a modality for θ consecutive transmissions until an index function γm(θ) exceeds a threshold g.
Error-aware switching-and-transmission policy (EAST): This is proposed as the optimal policy for the general multi-modality case, computed using MPI.
Error-aware transmission policy (EAT): This is a low-complexity policy under a fixed order and can be computed via MPI.
Fixed threshold policy (FT): This is another low-complexity option where each modality has a single threshold, computed via coordinate descent.

Numerical Evaluation and Trade-offs

Numerical results across three case studies (2, 3, and 5 modalities) demonstrate the superiority of the proposed policies over simple heuristics like round-robin or greedy policies. For instance, EAST reduces inference error by up to 44.8% compared to the best baseline in each case. Furthermore, for the five-modality case, FT reduces computation time by about 3000× relative to EAST while increasing the inference error by 38.6%. The complexity analysis shows that policy representation decreases from KM → 2M(M − 1)KM−2 → MKM−2 → M as modality count increases.

Conclusion

The work establishes an optimal index-based threshold policy for two modalities and the error-aware switching-and-transmission policy (EAST) for the general multi-modality case. The research successfully reduces the state set while preserving optimality and provides low-complexity alternatives like EAT and FT that offer significant computational efficiency gains. The analysis confirms that minimizing AoI alone is not optimal for minimizing the inference error function, as performance depends on complex combinations of AoIs. The paper concludes by suggesting future work in extending the communication model to unreliable channels and exploring reinforcement learning approaches. >

--- Page 1 ---

Multimodal Remote Inference Keyuan Zhang∗, Yin Sun†, Bo Ji∗ Department of Computer Science, Virginia Tech †Department of Electrical and Computer Engineering, Auburn University Email: keyuanz, boji@vt.edu, yzs0078@auburn.edu Abstract—We consider a remote inference system with multiple modalities where a multimodal machine learning (ML) model performs real-time inference using features collected from remote sensors. When sensor observations evolve dynamically over time, fresh features are critical for inference tasks. However, timely delivery of features from all modalities is often infeasible under limited network resources. To address this challenge, we formulate a multimodal scheduling problem to minimize the ML model’s inference error. We model this error as a general function of the Age of Information (AoI) vector, where AoI quantifies data freshness. We cast the problem as a semi-Markov decision process (SMDP) and derive an equivalent reformulation with a reduced state set. We then show that the problem has fundamentally different chain structures in the two-modality and multi-modality cases. For the two-modality case, we prove that the optimal policy has an index-based threshold structure. For the general multi-modality case (i.e., with more than two modalities), we develop the optimal error-aware switching-and-transmission policy (EAST), which is computed using a multichain policy iteration algorithm (MPI). To further reduce complexity, we also develop two low-complexity policies under special settings: the error-aware transmission policy (EAT) and the fixed threshold policy (FT). Numerical results from three case studies show that the proposed policies outperform several simple heuristics, including round-robin, greedy, and uniform random policies. In particular, EAST reduces the inference error by up to 44.8% compared with the best baseline in each case. In the five-modality case, EAT and FT reduce computation time by 6.6× and 3000×, respectively, relative to EAST, while increasing the inference error by 20.2% and 38.6%, respectively. Index Terms—Scheduling; Age of Information; Remote Inference; Multimodal Learning.

--- Page 2 ---

Age of Information (AoI) [2], [7]–[12]. Age of Information, introduced in [13], is an important metric for information freshness and is defined as the time elapsed since the freshest received information was generated. Most existing studies focus on either the single-source setting or multi-source settings with independent sources. In multimodal remote inference, however, the inference error is generally a function of the AoI vector, denoted by L(∆(t)), where ∆(t) = (∆1(t),..., ∆M(t)) is the AoI vector of the M modalities at time t. This function can be quite general: it may be non-monotonic, non-convex, and non-separable, which further complicates the scheduling problem. This type of AoIvector function optimization has been studied in correlatedsource scheduling [12], where an information-theoretic lower bound and a corresponding approximation are derived under the assumption that the ML model is optimal for the inference task. By contrast, our model removes this assumption. In addition, prior work [12] assumes unit transmission times across all sources, whereas in our setting, different modalities may have different transmission times due to their varying feature sizes. To this end, in this paper, we study the following key research question: How can we design an effective and efficient modality scheduling policy to minimize ML remote inference error? We answer this question by making the following main contributions: • We formulate a multi-modality scheduling problem that minimizes the inference error, which is a function of the AoIs across modalities.

Improvements for AI systems

A multimodal scheduling policy that minimizes ML inference error, specifically the Error-aware Switching-and-Transmission Policy (EAST), can be implemented to reduce inference error by up to 44.8% compared to baselines in multi-modality settings. This enables AI systems in fields like autonomous transportation or industrial automation to make more accurate real-time predictions by intelligently selecting which sensor modality to transmit next based on the general function of the Age of Information (AoI) vector.
An index-based threshold policy, derived for the two-modality case where the optimal policy has an index-based threshold structure, can be used to determine when to switch modalities. This allows systems with two sensor streams to dynamically decide between them based on an optimized threshold, improving inference accuracy by achieving a 40.6% reduction relative to the best baseline (uniform random) in the two-modality task.
Low-complexity policies like the Error-aware Transmission Policy (EAT) and Fixed Threshold Policy (FT) can be deployed for scenarios where computational resources are severely constrained. EAT and FT achieve performance comparable to EAST with much lower computation time, such as reducing computation time by 3000× relative to EAST in a five-modality case, making complex remote inference feasible on edge devices.
The system can leverage the derived state-space reduction for computational efficiency; specifically, the reformulation reduces the state set from KM states to M(M − 1)KM−2 states, simplifying policy iteration and other algorithms by a factor of approximately 208 when M=4 and K=50.
The policy design can be adapted for non-monotonic AoI functions, as shown by the empirical error function results, allowing the scheduler to exploit complex correlations between modalities. This enables AI systems to achieve better performance when certain combinations of AoIs across modalities yield better inference performance, potentially leading to lower overall inference error than policies based only on maximizing freshness.

Sources

Gymnasium: A Standard Interface for Reinforcement Learning Environments