Fixed-Reference Pose Residuals for Measuring Cross-Dataset Cue Transfer in Human-Robot Interaction Anticipation

arXiv:2610.12245 · cs.RO · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Fixed-Reference Pose Residuals for Measuring Cross-Dataset Cue Transfer in Human-Robot Interaction Anticipation".

Dev: The gist The fixed-reference pose residual (FRPR) model provides an exact split between geometry and pose terms,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So, this paper is called "Fixed-Reference Pose Residuals for Measuring Cross-Dataset Cue Transfer in Human–Robot Interaction Anticipation." It’s trying to figure out if what you learn about a person on one robot actually helps you predict when they're going to touch it on a different robot at a different location.

Dev: Right, so the title is pretty descriptive, focusing on that transfer problem between datasets. The authors are Yang, Xiao, Zhang, Yangli945, and Zhou1*, and it sets up this specific measurement tool called the fixed-reference pose residual model to test that idea.

Taro: It sounds like they are trying to figure out which cues—like where someone is in the picture versus how their body is moving—are reliable when you move from one environment to another.

Rosa: Exactly, and what’s interesting about this approach is that they build a geometry predictor first, train it up, and then add a correction based on body pose data to see exactly where the split happens between geometry and pose information.

Dev: That exact split is the key part of the model; it lets them treat every prediction as a precise combination of how big or where someone is versus what they are doing in terms of their body movement.

The paper's summary: Rosa: So, what’s the core idea here? They take a geometry predictor, which they keep frozen while training the rest of the system on pose data to learn an additive correction. This gives them this fixed-reference pose residual model where every prediction splits exactly into a geometry term and a pose term.

Dev: Essentially, they are creating a way to measure what the robot's prediction changes when you introduce body pose information, using that frozen geometry as the baseline for comparison across different datasets.

Taro: It seems like they are testing whether the cues—the approach geometry, the body pose sequence, and head orientation—actually transfer well between two different public egocentric datasets called HUI360 and SSUP-A.

Rosa: That’s right; they look at how geometry, body pose, and head orientation move between those two environments including a check against a stronger reference that they select specifically for the study.

Dev: The main finding is that this model lets them measure exactly what pose adds under that reference, which gives them a paired measurement of how different cues transfer between these cross-dataset environments.

The paper's improvements: Rosa: They propose several ways to use this model better. One big improvement is the ability to deploy the frozen geometry reference model independently. That lets researchers measure exactly what pose adds under that reference without having to re-train or fine-tune every component for every new robot or site.

Dev: So, they suggest this setup allows you to get an exact, paired quantity for each cue contribution, which is useful because you can deploy the geometry part separately.

Taro: And they also look at how cues behave in opposite directions between the datasets. They found that geometry learned on HUI360 actually kept its ranking quality on SSUP-A, but geometry learned on SSUP-A lost much of it when moved to HUI360.

Rosa: That’s a crucial distinction; it shows that cues aren't just good or bad across the board; they transfer in opposite ways depending on what you started with.

Dev: And they highlight head orientation as a particularly interpretable result, showing that whether a person faces the camera kept its discriminative direction across both robots and sites.

Conclusion: Rosa: So to wrap up this paper on Fixed-Reference Pose Residuals for Measuring Cross-Dataset Cue Transfer in Human–Robot Interaction Anticipation, they show how you can use this model as a measurement instrument to get an exact reading of each cue’s contribution.

Dev: The main implication is that for social robots needing to anticipate approach on a new platform, attention-related cues are candidates for transfer, but geometry and pose need their value re-established per platform.

Taro: It seems like the results suggest that cues describing attention towards the robot are what can carry over, whereas cues tied to camera geometry like pitch or the side of approach really need to be reexamined for each specific robot location.

Rosa: Exactly; they conclude that a cue selected on source data can carry a site-specific component with the wrong sign, meaning you have to be careful about what you assume transfers.

Dev: They also point out some limitations: they mentioned that the labels are physical contacts, and the models predict an event rather than necessarily what a person intends to do.

School of Automation, Nanjing University of Science and Technology

cs.RO

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 20 pages, 5 figures, 1 table. Supplementary material available as an ancillary file

Code: https://github.com/WeiZhou96/FRPR-interaction-anticipation

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 81/100

The gist: The gist The fixed-reference pose residual (FRPR) model provides an exact split between geometry and pose terms, which serves as a measurement instrument to quantify how different cues transfer

Key concepts

Fixed-Reference Pose Residual (FRPR) Model: This model splits every prediction exactly into a geometry term and a pose term. It uses a reference network for geometry and adds an additive correction learned from body pose data to create the final score. This exact separation allows researchers to precisely measure what pose information is added under a specific geometric context.
Cross-Dataset Cue Transfer: This refers to studying which cues (like body shape or head orientation) learned on one dataset can be successfully used on a new, different dataset. The study found that geometry learned on one site might not transfer well to another, while pose corrections show different patterns of transfer depending on the specific cue.
Geometry vs. Pose Terms: The model is designed so that the prediction score is composed of two distinct parts: a geometry term (learned by a reference network) and a pose term (learned by an encoder). This structure acts as a measurement tool, showing how much influence the body's physical shape versus its orientation has on the final interaction prediction.
Attention-Related Cues: The study suggests that cues related to how attention is directed toward the robot are strong candidates for transferring between environments. In contrast, cues tied strictly to camera geometry, such as pitch or the side of approach, need careful re-evaluation when deploying robots in new locations.

Terminology

Calibration — 5× in this paper · explained in 7 episodes
Accuracy — 3× in this paper · explained in 1 episode
Benchmark — 3× in this paper · explained in 2 episodes
Computer Vision and Pattern Recognition — 3× in this paper
Core — 3× in this paper · explained in 2 episodes
Generalization — 2× in this paper · explained in 13 episodes
Ablation — 1× in this paper · explained in 1 episode
Adaptation — 1× in this paper · explained in 1 episode
Alternation — 1× in this paper
Fine-Tuning — 1× in this paper · explained in 5 episodes
IEEE International Conference on Human-Robot Interaction — 1× in this paper
Multimodal Learning — 1× in this paper · explained in 2 episodes
Regularization — 1× in this paper · explained in 3 episodes
Representation Learning — 1× in this paper · explained in 3 episodes
Tenth International Conference on Learning Representations — 1× in this paper
Transformer — 1× in this paper · explained in 2 episodes
Two-stage training — 1× in this paper · explained in 2 episodes

Summary

The gist The fixed-reference pose residual (FRPR) model provides an exact split between geometry and pose terms, which serves as a measurement instrument to quantify how different cues transfer between cross-dataset environments.

How it works

The FRPR model is constructed by training a geometry predictor, denoted as the reference network, and then adding an additive correction learned from body pose data to its logit, resulting in the formulation where every prediction splits exactly into a geometry term and a pose term 3.

The model involves three stages of training:

Stage 1 trains the geometry multilayer perceptron (MLP), whose logit g = gθ∗(a) gives the stand-alone reference score pG = σ(g); it remains frozen in Stages 2 and 3 5.
Stage 2 trains a causal temporal encoder with a zero-initialised linear head, producing the pose residual r = rϕ(q) 35,393 parameters>.
Stage 3 (used by FRPR-H only) trains a small MLP with a zero-initialised output on h while g and r stay frozen 6.

The final prediction score pF is computed as pF = σ(g + r), where the difference between FRPR and its own reference is a paired measurement of what pose adds under that reference 35,393 parameters>. This exact split allows for the study of cue transfer 5.

Cross-Dataset Evaluation

The study evaluates cue transfer between two public egocentric datasets, HUI360 and SSUP-A 8. The pose correction raised average precision (AP) from 0.277 to 0.321 from SSUP-A to HUI360 and gave no measurable gain in the opposite direction 9. The same asymmetry held over a stronger, source-selected geometry reference 9. Freezing gave no AP advantage over joint training, and simple geometric baselines and tree ensembles remained competitive or better 9.

Cue Transfer Analysis

The analysis of which cues can be expected to carry over to a new platform revealed that geometry learned on HUI360 kept its ranking quality on SSUP-A, whereas geometry learned on SSUP-A lost much of it on HUI360 6. Pose behaved the other way round 9. Head orientation gave the most interpretable result, as whether a person faces the camera kept its discriminative direction across both robots and sites 7. Pitch reversed, which may reflect different camera heights relative to the faces of approaching people 7.

Model Comparison and Findings

The study compared FRPR against various models, finding no advantage for freezing the geometry branch over joint training or fusion models 9. Simple models like the box-area rule were strong, ranking by last-frame box area giving the highest AP in HUI360→SSUP-A (0.178) 9. The pose correction alone in SSUP-A→HUI360 was a modest ranker (AP 0.436 against 0.321) 9. Head orientation as a further residual showed gains, with FRPR-H exceeding FRPR by +0.011 AP in HUI360→SSUP-A and by +0.007 in SSUP-A→HUI360 9.

Conclusion

For social robots that must anticipate approach on a new platform, these results point to attention-related cues as candidates for transfer, to geometry and pose as cues whose value has to be re-established per platform, and to threshold calibration on the target site as a requirement before deployment 13. The fixed reference serves as a measurement instrument that allows for the exact reading of each cue's contribution 9. The results indicate that cues describing attention towards the robot are candidates for transfer, whereas cues tied to camera geometry, such as pitch or the side of approach, need to be reexamined for each platform 6. The study concludes that a cue selected on source data can carry a site-specific component with the wrong sign 13.

Limitations

The labels are physical contacts with the platform, and the models predict an event, not what a person intends 8. Windows are anchored on events, so the results describe ranking quality on a benchmark sample of tracks and do not give the false-alarm rate of a detector running continuously on a robot 9. Source-selected thresholds transferred poorly for all learned models 9. The orientation extension was locked before target scoring, whereas the stronger-reference analysis and the analysis of individual orientation cues are post hoc 9. The five and nine target recording dates bound the precision of every interval 9.

Data Availability

The core processed data, including the per-sample predictions and derived statistics behind every table and quantitative figure, are publicly available on GitHub at https://github.com/WeiZhou96/FRPR-interaction-anticipation 17. The HUI360 and SSUP-A annotations are distributed by their authors [25] 17. The core code is publicly available under the MIT licence on GitHub at https://github.com/WeiZhou96/FRPR-interaction-anticipation 17.

References

[1] Abbate G, Giusti A, Schmuck V, et al (2024) Self-supervised prediction of the intention to interact with a service robot 38.

[3] Arreghini S, Abbate G, Giusti A, et al (2024) A long-range mutual gaze detector for HRI 38.

[7] Bai S, Kolter JZ, Koltun V (2018) An empirical evaluation of generic convolutional and recurrent networks for sequence modeling 39.

[9] Bian T, Ma Y, Chollet M, et al (2024) Interact with me: Joint egocentric forecasting of intent to interact, attitude and social actions 39.

[17] Friedman JH (2001) Greedy function approximation: A gradient boosting machine 38.

[48] Xu Y, Zhang J, Zhang Q, et al (2022) ViTPose: Simple vision transformer baselines for human pose estimation 39.

[51] Zhou K, Liu Z, Qiao Y, et al (2023) Domain generalization: A survey 39.

[7] Bai S, Kolter JZ, Koltun V (2018) An empirical evaluation of generic convolutional and recurrent networks for sequence modeling 39.

[8] Belardinelli A (2024) Gaze-based intention estimation: Principles, methodologies, and applications in HRI 39.

[9] Bian T, Ma Y, Chollet M, et al (2024) Interact with me: Joint egocentric forecasting of intent to interact, attitude and social actions 39.

[10] Breiman L (2001) Random forests 39.

Improvements for AI systems

A frozen geometry reference model can be deployed independently, allowing researchers to measure what pose adds under that reference without needing to re-train or fine-tune every component for every new robot or site. This enables a deployment strategy where the contribution of each cue is an exact, paired quantity.
The system can perform bidirectional cross-dataset evaluation by leveraging source-only model selection, allowing for the assessment of how cues transfer between domains like HUI360 and SSUP-A, providing insights into which cues keep their meaning under such a change versus those that reverse.
A head-orientation residual (FRPR-H) can be added to the logit to produce a score where every prediction splits exactly into a geometry term and a pose term, offering an interpretable measurement of the orientation cue's contribution.
The system can distinguish between cues that transfer and those that do not, as it shows Geometry learned on HUI360 kept its ranking quality on SSUP-A, whereas geometry learned on SSUP-A lost much of it on HUI360; pose behaved the other way round.
The system can be used to guide deployment decisions by providing a metric for cue transfer, noting that Geometry and pose keep their ranking quality in opposite transfer directions, suggesting cues tied to camera geometry (like pitch) need re-examination per platform.