Apple: Toward General Active Perception via Reinforcement Learning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Apple: Toward General Active Perception via Reinforcement Learning".
Dev: Here's the gist of the paper. Active perception is framed as a supervised learning problem where an agent must actively gather data from an interactive environment to infer properties like object classes.
Rosa: APPLE is a novel framework that jointly trains a transformer-based perception module and decision-making policy with a unified optimization objective to learn how to actively gather information.
Dev: This approach is designed to be general and not limited to specific tasks, allowing it to be applied across a wide range of active perception problems.
Rosa: Problem Formulation Active perception is formally defined as a special case of a Partially Observable Markov Decision Process (POMDP) where the agent must act under uncertainty to actively reduce ambiguity about a target property.
Paper summary: Rosa: So what they're actually doing here is defining this objective function J of pi as something that maximizes the expected discounted return based on this combined reward structure >
Dev: The gradient of that objective function splits into two parts: a policy gradient and a negative supervised prediction loss gradient >
Taro: That’s interesting because it means they can optimize the agent's behavior using policy gradient methods while also using the prediction loss to guide learning about what properties are actually there >
Rosa: The total reward is defined as r of ht, asterisk yt, at and yt as the RL reward minus a differentiable prediction loss >
Dev: This structure allows them to optimize things like the agent's behavior based on both how well it’s making decisions and how accurately it predicts what’s around it >
Taro: It moves away from just maximizing information gain in a task-specific way, aiming for something more general across different problems >
Rosa: The main idea is that this joint training helps the agent learn a way to actively gather data across many different active perception scenarios rather than being limited to one specific task >
Dev: It’s designed to be general and not limited to specific tasks, allowing it to be applied across a wide range of active perception problems >
Taro: But they're still dealing with the fact that active perception is usually about sensing something local, which is different from vision where you can get wider information coverage from one observation >
Rosa: So while tactile sensing poses distinct challenges because each contact only gives a small glimpse of the environment, this framework tries to solve that locality issue using reinforcement learning >
Dev: They look at related work like vision and touch sensing, but they point out that most methods are tailored to a specific objective or use greedy information-gain heuristics >
Taro: The paper suggests there isn't really an active perception method shown to work on a wide range of problems yet because the existing ones are too task-specific >
Rosa: So APPLE is trying to bridge that gap by using this joint training signal to learn a more versatile way to perceive and interact with the world >
Dev: The structure of the reward function, r of ht, asterisk yt, at and yt, is what allows it to handle both the uncertainty about what’s hidden and the actions taken >
Taro: It's like they are teaching the agent how to play a game where winning means finding information you didn't know was there before >
Rosa: So for someone just listening, it means we might see agents that can actively search for things in physical spaces without being explicitly told what the goal is for every single step >
Conclusion: Rosa: So to wrap up on Apple: Toward General Active Perception via Reinforcement Learning, the paper is about this novel framework that uses reinforcement learning to address a wide range of different active perception problems >
Dev: The authors are aiming for generality by jointly training a transformer-based perception module and the decision-making policy with a unified objective function >
Taro: It’s trying to move beyond methods that are tailored to specific tasks, which is what they see as a gap in the existing research area >
Rosa: The main implication is that this approach could be applied across many different active perception problems, not just one or two specific ones like shape estimation or grasping >
Dev: It suggests a way to learn how to actively gather information in an environment without having to manually design a completely new algorithm for every single problem >
Taro: It’s about learning the skill of active perception itself rather than just optimizing the final outcome for one fixed goal >
Rosa: The authors also show that their approach can be compared against other methods like APPLE-SAC and APPLE-CrossQ, which are variants based on actor critic methods >
Dev: They found that APPLE-CrossQ can be more robust to new environments without needing extensive hyperparameter tuning compared to the APPLE-SAC variant >
Taro: So while they've shown promise across five benchmarks including classification, volume estimation, and localization tasks, there's still a limitation mentioned about the high amount of training data required for tactile perception tasks >
Rosa: That’s true; they state it needs up to five million steps for those tactile perception tasks because of the combination of the transformer architecture and RL optimization >
Dev: And their future work will focus on improving sample efficiency, perhaps by leveraging pre-trained transformer models or looking at multi-modal integration with vision and touch >
Taro: The paper shows that while the core concept is strong for generalization, making it practical for real-world deployment across all environments is still a question >
Tim Schneider, Cristiana de Farias, Roberto Calandra, Liming Chen, Jan Peters
Department of Computer Science, TU Darmstadt, Germany · LIRIS, CNRS UMR5205, Ecole Centrale de Lyon, France · LASR Lab & CeTI, TU Dresden, Germany · DFKI · Hessian.AI · RIG · Centre for Cognitive Science
cs.RO, cs.LG
Submitted: 2025-05-09
Updated: 2026-05-11
Comments: 27 pages; 21 figures; accepted at the Fourteenth International Conference on Learning Representations (ICLR 2026)
Journal ref: International Conference on Learning Representations (ICLR 2026)
Code: https://github.com/ax-ml/jax
Project page: https://timschneider42.github.io/apple
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 85/100
The gist: Active perception is framed as a supervised learning problem where an agent must actively gather data from an interactive environment to infer properties like object classes.
Key concepts
- Active Perception
- This is when an agent must actively gather information from its environment to reduce uncertainty about a hidden property, like an object's class. It's treated as a decision-making problem where the agent chooses actions not just to move, but specifically to learn what it needs to know.
- POMDP
- Partially Observable Markov Decision Process is the formal mathematical framework used here. It describes a situation where the agent cannot see the true state of its environment perfectly. The hidden state includes both what is directly observable and unknown properties, requiring strategic actions under uncertainty.
- Unified Optimization Objective
- APPLE combines two main goals: maximizing expected discounted return from an RL policy and minimizing a prediction loss for inferred properties. This joint training allows the agent to learn how to gather information (prediction loss) while simultaneously learning the best way to act based on that gathered knowledge (RL reward).
- Prediction Loss ($\ell$)
- This is a differentiable loss function used during training that measures how well the agent's prediction of an unknown property matches the actual environment state. It guides the agent to actively seek information, as minimizing this loss encourages the agent to explore and gather data that helps refine its understanding.
Terminology
Summary
Active perception is framed as a supervised learning problem where an agent must actively gather data from an interactive environment to infer properties like object classes. p. 3 However,
unlike in classical supervised learning, we assume that the agent is not simply presented with a
static data point as input, but rather with an interactive environment that it can actively gather data
from. , the agent could be presented with an object and has to decide actively how to examine
it to extract the information it needs. To address this gap, this
work introduces APPLE (Active Perception Policy Learning) – a novel framework
that leverages reinforcement learning (RL) to address a range of different active
perception problems. APPLE jointly trains a transformer-based perception module
and decision-making policy with a unified optimization objective, learning how to
actively gather information. By design, APPLE is not limited to a specific task and
can, in principle, be applied to a wide range of active perception problems. We
evaluate two variants of APPLE across different tasks, including tactile exploration
problems from the Tactile MNIST benchmark.Where the paper says it
Where the paper says it
Where the paper says it
Problem Formulation
Active perception is formally defined as a special case of a Partially Observable Markov Decision Process (POMDP) where the agent must act under uncertainty to actively reduce ambiguity about a target property. The problem is framed such that the hidden state decomposes into h˜t = (ht, ∗yt), where ht is the remainder of the hidden state without the ground truth property value. p. 4 Preprint
decomposes into h˜
t = (ht,
∗
yt
), where ht is the remainder of the hidden state without the ground
truth property value. Additionally, the agent’s action space contains not only control actions at but
also a current estimate yt of the desired environment property. Additionally, the agent’s action space contains not only control actions at but
also a current estimate yt of the desired environment property. In other words, the action space
decomposes into a˜t = (at, yt), meaning that the agent predicts the desired environment property at
every step. , a desired finger movement,
which is communicated to the agent’s motor controllers. The overall reward function r˜ consists of two parts: a differentiable prediction loss ℓ and an RL
reward r.Where the paper says it
Where the paper says it
Where the paper says it
Training Signal Construction
The objective function J(π) is formulated to maximize the expected discounted return based on the combined reward structure. The gradient of the objective function J(πθ) decomposes into a policy gradient and a negative supervised prediction loss gradient. p. 4 As can be seen in Eq. (3), the gradient of the objective
function J(πθ) decomposes into a policy gradient and a negative supervised prediction loss gradient. Preprint
to dynamically recompute the total reward when evaluating the Bellman residual, yielding
Lcritic = ED [
1
2
(Qθ(o0∶t, at) − (rt − ℓπθ
(
∗
yt
, o0∶t) + γEπθ
[Qθ¯(o0∶t+1, at+1)]))2
] . During the policy update, the policy gradient is augmented by the prediction loss gradient. Preprint
to dynamically recompute the total reward when evaluating the Bellman residual, yielding
Lcritic = ED [
1
2
(Qθ(o0∶t, at) − (rt − ℓπθ
(
∗
yt
, o0∶t) + γEπθ
[Qθ¯(o0∶t+1, at+1)]))2
] . During the policy update, the policy gradient is augmented by the prediction loss gradient.Where the paper says it
Where the paper says it
Where the paper says it
Method Variants and Comparison
Two variants of APPLE are proposed based on actor-critic methods: APPLE-SAC and APPLE-CrossQ. These methods utilize off-policy RL to estimate the policy gradient in Eq. p. 4 3 DERIVING APPLE-SAC AND APPLE-CR O S SQ
We use RL-based techniques to estimate the policy gradient in Eq. Here we propose two variations
of APPLE based on two actor-critic methods: SAC Haarnoja et al. To stabilize the training of
the Q-networks, SAC deploys target networks, which slowly track the actual Q-networks. CrossQ is
similar to SAC but drops the target networks and instead uses BatchRenorm Ioffe (2017) layers in the
Q-network to stabilize their training. In contrast, we employ off-policy methods (APPLE-SAC and
APPLE-CrossQ), enabling sample reuse – a critical factor in active perception, where supervised
learning benefits from multiple passes over the same data. Both APPLE-SAC and APPLE-CrossQ
perform comparably on environments they have been tuned on, but APPLE-CrossQ has proven
more robust to new environments without hyperparameter tuning and offers a clear computational
advantage.Where the paper says it
Where the paper says it
Where the paper says it
Empirical Evaluation Across Tasks
APPLE was evaluated across five benchmarks including classification, volume estimation, and localization tasks. The experiments on the tactile domain showed that APPLE consistently outperforms baselines like HAM and APPLE-RND across CircleSquare, TactileMNIST classification, TactileMNISTVolume regression, and Toolbox pose estimation. p. 8 5 DISCUSSION
The results in Section 4 show that APPLE successfully learns exploration strategies across diverse
active perception tasks. We evaluated APPLE on the CircleSquare classification task and three
simulated tactile benchmarks: TactileMNIST (digit classification), TactileMNISTVolume (volume
estimation), and Toolbox (pose estimation). APPLE-RND, achieves a 68% accuracy, highlighting the need
for active perception in the CircleSquare task. Despite extensive tuning of the learning rate and β
parameter with HEBO and training for 10M steps, we could not find a configuration for which HAM
reached a prediction accuracy beyond random guessing. While HAM is generally able to solve this task (see
Section C. 5 for a longer run), it requires a large amount of samples to learn an effective policy.Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Future Directions
A primary limitation of APPLE is its reliance on large amounts of training data, requiring up to 5M steps for the tactile perception tasks. This high data requirement stems from the combination of a transformer-based architecture and RL-based policy optimization. p. 5 2, the agent receives this exact combination of tactile images and sensor positions. To efficiently process this
data in the policy and Q-networks, we use an architecture
similar to the Video-Vision-Transformer (ViViT) Arnab et al. These results demonstrate APPLE’s versatility across
diverse tasks, including both classification and regression. Future work will focus on improving the
sample efficiency of APPLE, extending to more realistic applications such as in-hand pose estimation
and texture recognition, and exploring multi-modal integration with vision and touch. Future work will focus on improving the
sample efficiency of APPLE, extending to more realistic applications such as in-hand pose estimation
and texture recognition, and exploring multi-modal integration with vision and touch. Another potential future research avenue will be deploying APPLE on a real robotic system.Where the paper says it
Where the paper says it
Where the paper says it
Hyperparameter Optimization
Extensive hyperparameter searches using the HEBO Cowen-Rivers et al. (2022) Bayesian optimizer were performed to tune parameters for APPLE-SAC and APPLE-CrossQ. p. 8 It is important to note that we again did not tune any of these methods on this task and instead
relied on hyperparameters optimized for TactileMNIST. While tuning APPLE-SAC on this task can
lead to stronger performance, these results indicate that APPLE-CrossQ might be more robust
w. This formulation enables adaptability
across different tasks by making minimal assumptions about the nature of the underlying POMDP. • Comprehensive empirical evaluation of two method variants, based on SAC and CrossQ, on
classification, volume estimation, and localization tasks, demonstrating that APPLE can discover
active exploration policies without task-specific heuristics. It is important to note that we again did not tune any of these methods on this task and instead
relied on hyperparameters optimized for TactileMNIST. While tuning APPLE-SAC on this task can
lead to stronger performance, these results indicate that APPLE-CrossQ might be more robust
w. For regression tasks, we report average error and final error. The hyperparameters of all methods are
tuned with HEBO Cowen-Rivers et al.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Connection to Intrinsic Reward Methods
APPLE is concerned with learning hidden properties of the environment within an episode rather than using intrinsic reward bonuses to enhance policy learning across episodes. The exploration procedure of APPLE takes place within each episode, while intrinsic reward methods occur across episodes. These two concepts are fully orthogonal and could potentially be combined to speed up learning progress. p. 3 In the remainder of this section, we formally define our problem and derive the Active Perception
Policy Learning (APPLE) framework. We propose two variants of APPLE, based on SAC Haarnoja
et al.Where the paper says it
Role of Prediction Loss in Downstream Tasks
The prediction loss can help in the presence of another downstream task by using a tracking-aware loss function l((∗y, wt), yt) = wtMSE(∗yt, yt). This allows only successful tracking steps to be considered in the prediction loss. p. 4 Computing the gradient of J(πθ) now yields
∂
∂θ J(πθ) = E
pθ
(h,
∗
y,o,a)
[
∂
∂θ
lnπθ(a ∣ o)
∞
∑
t=0
γ
t
r˜(ht,
∗
yt
, at, yt)]
´ ¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¸¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¶
policy gradient
− E
pθ
(
∗
y,o)
[
∞
∑
t=0
γ
t ∂
∂θ
ℓπθ
(
∗
yt
, o0∶t)]
´ ¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¸¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¶
prediction loss gradient
. (3)
Intermediate steps can be found in Section G. Ideally, the formulation of such methods should be
agnostic to the underlying task, enabling transfer across modalities and problem settings. Thus, in this work, we ask the following: can we design a principled RL-based algorithm for
discovering active perception policies by relying only on a ground-truth label and a differentiable
loss during training?Where the paper says it
Where the paper says it
Model Architecture Influence
The choice of model can have a significant influence on APPLE’s overall performance, as shown by the ablation study comparing transformer and LSTM models. p. 9 Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. Overall, these
emergent behaviors highlight that APPLE can learn different active perception policies, thus adapting
its exploration behavior to the demands of each environment with minimal changes to the setup,
suggesting a promising step toward more general active perception frameworks. 6 CONCLUSION
We introduced APPLE, a framework combining reinforcement learning and transformer-based models for active tactile perception. 2, the agent receives this exact combination of tactile images and sensor positions. To efficiently process this
data in the policy and Q-networks, we use an architecture
similar to the Video-Vision-Transformer (ViViT) Arnab et al. (2020), which failed to learn an effective strategy even
on the Circle-Square toy task, defaulting to predicting the mean class despite extensive tuning using
HEBO. This points to a fundamental limitation of HAM in this setting, namely that HAM relies on
on-policy RL, which discards samples after a single update, limiting sample efficiency. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies. 6 show that APPLE-CrossQ reaches a final accuracy of 1. 9cm and 13○
on average, while APPLE-SAC and the random baseline APPLE-RND stagnate at much lower accuracies.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved AI systems can learn active perception policies for tactile sensing by jointly training a transformer-based perception module and a decision-making policy with a unified optimization objective, learning how to actively gather information. This allows the system to address various active perception problems without being limited to specific tasks or strong assumptions, as stated in the abstract.
-
The system can perform classification, regression, and pose estimation tasks on tactile data by formulating active perception within the setting of partially observable Markov decision processes (POMDPs), where it must act under uncertainty to actively reduce ambiguity about a target property. This approach allows the framework to be general enough to extend across diverse problems without requiring task-specific exploration heuristics.
-
The system can discover active exploration policies by relying only on a ground-truth label and a differentiable loss during training, as the objective is framed as minimizing a loss l(yt, y∗t) between its current prediction yt and the ground-truth label y∗t. This enables the learning of an agent's ability to
actively gather data from
an interactive environment. -
The system can utilize off-policy reinforcement learning methods like APPLE-SAC and APPLE-CrossQ, which are noted to be more robust to new environments without hyperparameter tuning compared to on-policy methods like HAM. This suggests a more practical framework for deploying active perception in real robotic systems.
-
The system can learn complex exploration strategies, such as finding the handle of a wrench and then sliding along it for orientation disambiguation in pose estimation tasks, demonstrating that learned behaviors can be intuitive from a human perspective. This capability is shown to be effective across different environments like the Toolbox task.
Abstract
Active perception is a fundamental skill that enables us humans to deal with uncertainty in our inherently partially observable environment. For senses such as touch, where the information is sparse and local, active perception becomes crucial. In recent years, active perception has emerged as an important research domain in robotics. However, current methods are often bound to specific tasks or make strong assumptions, which limit their generality. To address this gap, this work introduces APPLE (Active Perception Policy Learning) - a novel framework that leverages reinforcement learning (RL) to address a range of different active perception problems. APPLE jointly trains a transformer-based perception module and decision-making policy with a unified optimization objective, learning how to actively gather information. By design, APPLE is not limited to a specific task and can, in principle, be applied to a wide range of active perception problems. We evaluate two variants of APPLE across different tasks, including tactile exploration problems from the Tactile MNIST benchmark. Experiments demonstrate the efficacy of APPLE, achieving high accuracies on both regression and classification tasks. These findings underscore the potential of APPLE as a versatile and general framework for advancing active perception in robotics. Project page: https://timschneider42.github.io/apple
Sources
- Solving Rubik's Cube with a Robot Hand
- UAV Active Perception and Motion Control for Improving Navigation Using Low-Cost Sensors
- Tactile MNIST: Benchmarking Active Tactile Perception
- Towards Embodied Scene Description
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation