Variable Input Masked Autoencoders for Wrist-Worn Accelerometry

arXiv:2610.17383 · cs.LG · Submitted 2026-10-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Variable Input Masked Autoencoders for Wrist-Worn Accelerometry".

Jane: The gist The introduction introduces VarMAE, a variable input masked autoencoder that learns a single representation across sampling rates and temporal contexts for wrist-worn accelerometry,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper called "Variable Input Masked Autoencoders for Wrist-Worn Accelerometry". It’s about a model that tries to learn one representation that works across different sampling rates and time windows.

Jane: That’s the main thing, Tom. It tackles the problem of fixed input settings in foundation models when we deal with sensor data like wrist accelerometry, where things can be recorded at totally different frequencies or over wildly varying durations.

Lu: The core idea is that they construct two views of the same signal during pretraining: a long duration high frequency view and a short duration lower frequency view obtained through random cropping and downsampling. They then use masked reconstruction within each view, plus this cross-resolution objective to bridge the information gaps between those two views.

Meng: So, if I'm thinking about deployment, what does that actually mean for me? Does it just make the model bigger or slower?

Tom: Well, the architecture itself has flexibility in both input and output. They define patches over a fixed physical duration like zero point two seconds, which means how many samples are in each patch changes depending on the sampling rate <ref:2610.17383#pg2>. Then they use a variable-rate tokenizer to map those rate-dependent patches to a single fixed dimension token vector before the Transformer encoder.

Jane: That variable-rate tokenizer is pretty clever because it lets the model handle different sampling rates while keeping its main encoder structure consistent, which is important for making it work across various hardware platforms.

Lu: The cross-resolution masked reconstruction objective is where they really bridge those bottlenecks. They sample a short view duration, say one to ten seconds, and a sampling rate between ten hertz and fifty hertz randomly for each training iteration.

Meng: And how do they ensure the learning from that short view actually helps with the representation? Is it just masking things in both views at the same spot?

Lu: Exactly. Every patch masked in the short view is also masked at its corresponding location in the long view, but patches visible in one are visible in the other. The cross-decoder uses those visible tokens from the long view as keys and values while all tokens from the short view, both visible and masked, form the query for reconstruction.

Tom: The training loss is split into three parts: LShort to reconstruct the short view using only its own visible tokens, LLong for reconstructing the long view with its own tokens, and then LCross to reconstruct those masked tokens in the short view conditioned on what's visible in the long view through that cross-decoder.

Title and authors: Jane: It sounds like they’re forcing the model to learn how to use context from a longer time scale—the long view—to predict information when you only have a shorter, lower frequency window.

Lu: That's it. The within-view losses, LShort and LLong, are important for maintaining reconstruction quality on their respective views independently. But the cross-resolution loss, LCross, is what really forces the model to generalize across those different temporal scales and sampling rates.

Meng: So if we look at the results they report with this "Variable Input Masked Autoencoders for Wrist-Worn Accelerometry" paper, what kind of performance gains are we actually seeing? Are these gains significant enough for a production environment?

Tom: They pre-trained VarMAE on the NHANES dataset and it produced a single pretrained encoder that matches or outperforms controlled foundation model baselines across eight benchmark datasets. With only three point eight two million parameters, this shows that one foundation model can achieve broad transfer capabilities across sensing configurations.

Jane: That parameter count is really something when you think about the efficiency of what they’ve trained it on—a large-scale accelerometry dataset from around sixteen thousand participants. It suggests that a single foundation model can handle different sensing setups without needing specific training for every configuration.

Lu: The controlled evaluation results also showed that VarMAE matches or outperforms all controlled baselines across datasets, and importantly, all three variants of VarMAE outperformed the TSFMs, which is a big indicator about the advantage of pretraining specifically using accelerometer data.

Meng: That's good for transferability, but what about performance when we actually deploy this on a device where we only get low frequency data? Does it hold up when the sampling rate drops significantly?

Tom: The ablation studies show that removing LShort produces substantial degradation in performance at low frequencies, with performance drops exceeding twenty five percent F1-score at five hertz. So they found that Lshort and LCross are primarily useful for retaining performance as the sampling rate decreases.

Jane: That's a key piece of evidence. It proves that the cross-resolution loss is essential for keeping the model effective when you’re working with less temporal resolution, which is exactly what we often see in real-world wearable deployments.

Lu: The implication here is that this single accelerometer foundation model can effectively transfer across substantially different downstream sensing configurations without needing bespoke training at specific input configurations. It’s about building one general-purpose accelerometer foundation model.

Meng: From a deployment standpoint, if we have to ship just one model, this VarMAE approach makes sense because it handles the variation in how the data was collected on the fly during inference much better than specialized models would.

Title and authors: Tom: Exactly. The paper "Variable Input Masked Autoencoders for Wrist-Worn Accelerometry" introduces a variable input MAE pretrained on large-scale accelerometry from around sixteen thousand participants, enabling a shared Transformer encoder-decoder to operate across different sampling rates and window durations.

Jane: It’s a significant step towards developing more broadly usable foundation models for wearable sensing because it reduces the need for bespoke training at specific input configurations.

Lu: We see the ability of this single accelerometer representation to remain effective across substantially different sensing configurations, which is what matters for real-world applications.

Meng: I'm just curious about how we handle the complexity of deploying something that relies on both long and short views during inference; it’s a lot to manage when things break at scale.

Tom: The architecture itself handles the variable rate reconstruction head, which reconstructs the decoder outputs at the target sampling rate, so it keeps things consistent for deployment.

Jane: It really shows that representation learning on raw sensor data can be more robust to real-world variability than models trained on perfectly controlled lab settings.

Lu: The authors are pointing toward this flexibility as an important step towards a single foundation model that works across many different applications in the sensing domain.

Meng: If we look at the variables they use, like dS being one to ten seconds and rS being between ten hertz and fifty hertz, that shows a good range of flexibility for capturing different temporal contexts.

Tom: Yes, and the results with only three point eight two million parameters show that this level of generality is achievable without needing massive parameter counts.

Jane: So the main thing we see here is a shared Transformer encoder-decoder capable of handling these different input conditions without us having to retrain for every new sampling rate or window size.

Lu: The method demonstrates that a single accelerometer representation can remain effective across substantially different sensing configurations, which is what matters for real-world applications.

Meng: It means we don't need to maintain multiple specialized models just because the data collection process changed slightly.

Tom: So the VarMAE approach, as described in "Variable Input Masked Autoencoders for Wrist-Worn Accelerometry", offers a way to build one powerful, general-purpose accelerometer foundation model.

Jane: It’s an important step towards developing more broadly usable foundation models for wearable sensing because it reduces the need for bespoke training at specific input configurations.

Tom: So the VarMAE approach, as described in "Variable Input Masked Autoencoders for Wrist-Worn Accelerometry", offers a way to build one powerful, general-purpose accelerometer foundation model.

The paper's summary: Tom: So, to wrap up what we just talked about, this paper introduces something called VarMAE that lets you train one big Transformer encoder and decoder to handle wrist-worn accelerometer data at different sampling rates and with different time windows.

Jane: Right, so it’s not building a whole new model for every single sampling rate or duration you might encounter when someone records activity. It’s trying to build one general representation that just works across that variety of conditions.

Lu: The core mechanism is creating two different views of the same signal during pretraining: one long, high-frequency view and a shorter, lower-frequency view made by cropping and downsampling randomly.

Meng: And then they use a special cross-resolution loss to bridge the gap between those two views. It forces the model to learn how to use context from that longer view to predict information in the shorter one.

Tom: Exactly, and what I find really compelling about it is how flexible the architecture is designed for this variable input. They define patches over a fixed physical time, like zero point two seconds, which means the number of samples in each patch changes depending on whether you're at fifty hertz or ten hertz.

Jane: That’s smart because they use a variable-rate tokenizer to map those rate-dependent patches down into one fixed dimension token vector before it even hits the main Transformer encoder. That keeps the rest of the model structure consistent.

Lu: The training loss is split up too, with three parts: LShort for reconstructing the short view on its own, LLong for reconstructing the long view on its own, and then LCross which uses that cross-decoder to reconstruct masked tokens in the short view using what's visible from the long view.

Tom: And when I looked at the results, it’s pretty solid. They pre-trained it on a large dataset from NHANES, and that single model matches or beats several foundation model baselines across eight different benchmark datasets.

Jane: That efficiency is striking, Tom. With only three point eight two million parameters, one foundation model can achieve this broad transfer capability across sensing configurations without needing specialized training for every setup you want to deploy it on.

Meng: But we have to ask about deployment reality—what happens when we actually put this on a device and the data quality is lower than what they trained on? Does it hold up when the sampling rate drops significantly?

Lu: That’s where the ablation studies show something important. Removing LShort causes a big drop in performance at low frequencies, over twenty five percent F1-score loss at five hertz. So they showed that Lshort and LCross are crucial for maintaining performance when you’re dealing with less temporal resolution.

Tom: So the conclusion is that this single accelerometer representation can transfer effectively across substantially different sensing setups without needing bespoke training for every input configuration.

Jane: It means we don't need to maintain a bunch of specialized models just because the data collection process changed slightly, which simplifies things a lot for anyone trying to build wearable applications.

Lu: They are pointing toward this flexibility as an important step towards a single foundation model that works across many different applications in the sensing domain.

Meng: If we look at the variables they used, like dS being one to ten seconds and rS between ten hertz and fifty hertz, that shows a good range of flexibility for capturing different temporal contexts.

Tom: Yes, and it’s impressive that this level of generality is achievable without needing massive parameter counts.

Lu: The paper demonstrates that a single accelerometer representation can remain effective across substantially different sensing configurations, which is what matters for real-world applications.

Meng: It means we don't need to maintain multiple specialized models just because the data collection process changed slightly.

Tom: So the VarMAE approach, as described in "Variable Input Masked Autoencoders for Wrist-Worn Accelerometry", offers a way to build one powerful, general-purpose accelerometer foundation model.

The paper's improvements: Tom: So, we've been talking about how VarMAE works for variable input data, and now we look at what the authors are suggesting to make it even better.

Jane: What are they proposing next? Are they trying to fix some of those gaps we talked about earlier, like when the sampling rate gets really low?

Lu: They focus on improving how the cross-resolution loss interacts with those different temporal scales. They suggest that tuning the parameters controlling that cross-decoder helps ensure it bridges those bottlenecks more effectively.

Tom: I like this part where they emphasize that even though they use a fixed physical duration for patches, the variable rate tokenizer is key to maintaining consistency across hardware platforms. It’s about making sure the token vector stays useful no matter what input frequency you feed it.

Jane: That makes sense because if you can't change your model architecture when you move from one sensor setup to another, that’s a huge win for deployment. It means less bespoke training is needed for every new hardware configuration.

Lu: They also explore how the masking strategy itself can be more nuanced. Instead of just masking everything at the same spot in both views, they look into ways to make sure that visible tokens from one view really condition the prediction of masked tokens in the other view more strongly.

Meng: From an engineering standpoint, if we can make those conditioning signals stronger, it means we might get better reconstruction quality even when we're using less data or lower frequency input. That’s what I care about for production stability.

Tom: Exactly, and the results they show in their ablation studies on LShort and LCross prove that these components are actually necessary to keep performance up as the sampling rate decreases. It confirms that just having a standard MAE isn't enough for this flexible approach.

Jane: So, it’s not just about putting a variable input mechanism on top of an existing model; it’s about refining the training objective itself to leverage those different temporal contexts more intelligently.

Lu: They are looking at ways to optimize the weighting of LShort versus LLong versus LCross so that the model learns exactly which context—the long view, the short view, or that bridge between them—is most important for a given downstream task.

Tom: If they can nail that weighting, it supports their claim about achieving broad transfer capabilities across those eight benchmark datasets. It moves it from just being flexible to being highly adaptable.

Jane: What this means for us is we can start thinking about building one foundation model that you can adapt for different types of wearable sensing, rather than needing a whole new model every time the data collection method shifts slightly.

Lu: The authors are still working on refining those objective functions to ensure that when we deploy this, the performance gains hold up under real-world noise and varying signal quality.

Conclusion: Tom: So, to wrap up what we just covered on "Variable Input Masked Autoencoders for Wrist-Worn Accelerometry," this paper shows how you can build one model that works across different sampling rates and time windows without needing to retrain it for every single setup.

Jane: Right, so the core idea is building a general representation for wrist accelerometry that adapts to whatever recording conditions you happen to have when you try to use it.

Lu: It’s about using two different views of the data—a long high-frequency view and a short lower-frequency view—and then using a cross-resolution loss to make sure those two views talk to each other correctly.

Meng: From an engineering angle, this means we might be able to ship one core model and have it cover a lot more real-world sensor variations without needing a huge team of specialized engineers for every new deployment.

Tom: And the numbers back it up; they pre-trained this on NHANES data and found it outperforms several foundation models across eight different benchmarks with only three point eight two million parameters. That parameter efficiency is pretty strong for what it achieves.

Jane: That means for someone listening, you can imagine having one powerful AI that handles activity recognition whether you're measuring something at fifty hertz or just ten hertz on a smartwatch.

Lu: The implication is that this single foundation model can be a much more versatile tool across the whole wearable sensing domain than models trained only on perfectly controlled lab settings.

Meng: But we have to remember the caveat they mentioned: if you drop the sampling rate too low, like below five hertz, performance drops substantially because of how important LShort and LCross are to keeping things stable.

Tom: It’s a trade-off—you get incredible flexibility across many settings at the cost of needing those specific loss components to maintain quality at lower resolutions.

Jane: So the main thing we see here is that representation learning on raw sensor data can be more robust to real-world variability than models trained in perfect lab conditions.

Lu: They are still working on refining those objective functions to ensure that when we deploy this, the performance gains hold up under real-world noise and varying signal quality.

Meng: I just wonder how much overhead there is for running it at scale when you're trying to deploy this across millions of devices.

Tom: The architecture handles the variable rate reconstruction head, which reconstructs the decoder outputs at the target sampling rate, so it keeps things consistent for deployment regardless of input variability.

Jane: It really shows that building a flexible foundation model for wearable sensing is a very active area right now because this VarMAE approach offers a way to build one general-purpose accelerometer foundation model.

Lu: I think the work on cross-resolution objectives is what truly pushes the boundaries here, making the system learn how to use context across these different temporal scales.

Meng: We'll be looking at how they refine those loss functions in their next iterations because that’s where you figure out if this translates into a reliable production asset.

Tom: So that’s what we have for today on "Variable Input Masked Autoencoders for Wrist-Worn Accelerometry." It's a solid piece of work showing how to make foundation models more flexible for real-world sensing data.

Harish Haresamudram, Wanting Mao, Aditya Radhakrishnan, Maxwell A. Xu, James M. Rehg

University of Illinois Urbana-Champaign · Google

cs.LG

Submitted: 2026-10-12

Updated: 2026-10-12

Code: https://github.com/iantangc/ContrastiveLearningHAR

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist The introduction introduces VarMAE, a variable input masked autoencoder that learns a single representation across sampling rates and temporal contexts for wrist-worn accelerometry, which

Key concepts

Cross-Resolution Masked Reconstruction: This technique bridges information gaps between long, high-frequency data and short, low-frequency data. It uses visible tokens from the long view to guide the prediction of masked tokens in the short view, ensuring that representations learned work across different input durations and sampling rates.
Variable-Rate Tokenizer: Since patches have fixed physical durations but varying sample counts based on sampling rate, this tokenizer maps these rate-dependent patches into a single, fixed-dimension token vector. This allows the Transformer encoder to process inputs consistently regardless of how many samples are in a given patch.
Cross-Resolution Objective: The training loss combines three components: reconstructing the short view from itself (LShort), reconstructing the long view from itself (LLong), and using the cross-decoder (gcross) to reconstruct masked tokens in the short view, conditioned on visible tokens from the long view. This forces the model to learn robust features that generalize across different data scales.
Variable Input MAE: This architecture extends standard Masked Autoencoders by making both input and output flexible concerning sampling rates. It handles varying frequencies by defining patches over a fixed physical time (e.g., 0.2s) and using a variable-rate tokenizer to map these patches into a unified format for the Transformer encoder.

Summary

The gist The introduction introduces VarMAE, a variable input masked autoencoder that learns a single representation across sampling rates and temporal contexts for wrist-worn accelerometry, which addresses the fundamental challenge of fixed input configurations in existing foundation models.

How it works

VarMAE is designed to operate across different sampling rates and window durations by constructing two views of the same sensor data: (i) a long duration, high frequency view that captures the full context, and (ii) a short duration, lower frequency view obtained via random cropping and downsampling. The model introduces a cross-resolution masked reconstruction objective where visible tokens from the long view condition the prediction of masked tokens from the short view. This setup leverages cropping and downsampling as information bottlenecks, encouraging learning representations effective across input durations and sampling rate changes.

The architecture extends the standard MAE with rate flexibility in both input and output. This is achieved by defining patches over a fixed physical duration of time, such as 0.2s, causing the number of samples within each patch to vary based on the sampling rate. A variable-rate tokenizer maps these rate-dependent patches to a single fixed dimension token vector before the Transformer encoder. The variable-rate reconstruction head then reconstructs the decoder’s outputs at the target sampling rate.

Cross-Resolution Masked Reconstruction

The cross-resolution masked reconstruction objective is crucial for bridging the information bottlenecks created by cropping and downsampling. The long view, with a fixed duration and sampling rate (dL, rL) = (30s, 50Hz), provides additional temporal context. For each mini-batch, a short view duration dS ∈ 1s, 2s, 5s, 10s and sampling rate rS ∈ 10Hz, 20Hz, 30Hz, 40Hz, 50Hz are randomly sampled. The short view is obtained by cropping the dS-second crop from the long view and then resampling to rS using polyphase resampling with an anti-aliasing filter to obtain xS.

Masking and co-masking are performed such that a region masked in xS, if visible in xL would allow the cross-decoder (gcross) to directly observe the reconstruction target. Specifically, every patch masked in xS is also masked at the corresponding location in xL, while patches visible in xS remain visible in xL. The cross-decoder (gcross) uses the visible tokens from xL as key and value, while all tokens from xS (visible and masked) form the query.

Training Objective and Loss Components

The overall training loss L is defined as the sum of three masked reconstruction losses: LShort, LLong, and LCross. The within-view losses are formulated as follows: LShort reconstructs the short view using only its own visible tokens, while LLong reconstructs the long view using its own visible tokens. The cross-resolution loss LCross reconstructs masked tokens of xS, conditioning the predictions on the visible tokens from xL through the cross-decoder.

Results and Performance

VarMAE was pre-trained on the NHANES dataset, producing a single pretrained encoder matching or outperforming controlled foundation model baselines on eight benchmark datasets. With only 3.82M parameters, VarMAE shows that a single foundation model can achieve broad transfer capabilities across sensing configurations. In the controlled evaluation, VarMAE matches or outperforms all controlled baselines across datasets. Furthermore, all three variants of VarMAE outperform the TSFMs indicating the substantial advantage of pretraining specifically using accelerometer data.

The ablation studies show that removing LShort produces substantial degradation in performance at low frequencies, with performance drops exceeding 25% F1-score at 5Hz. The utility of Lshort and LCross is primarily to retain performance as sampling rate decreases. The results demonstrate that a single accelerometer FM can effectively transfer across substantially different downstream sensing configurations.

Conclusion

VarMAE introduces a variable input MAE pretrained on large-scale accelerometry from ∼16k participants, enabling a shared Transformer encoder-decoder to operate across different sampling rates and window durations. This flexibility is an important step towards the development of more broadly usable FMs for wearable sensing. The paper demonstrates that a single accelerometer representation can remain effective across substantially different sensing configurations, reducing the need for bespoke training at specific input configurations.

AI USE STATEMENT

In this work, we used generative AI tools to help write code, and to polish grammar and fix typos in the paper.

ETHICS STATEMENT

Our work aims to support personalized health applications through developing models on physiological signals.

REPRODUCIBILITY STATEMENT

To ensure complete reproducibility of our pre-training framework, model checkpoints and empirical findings, we will open-source all code, pretrained model weights, and downstream evaluation pipelines upon acceptance.

REFERENCES

Abdul Fatir Ansari et al. Chronos: Learning the language of time series.

Alexei Baevski et al. wav2vec 2.0: A framework for self-supervised learning of speech representations.

Fabio Bagala et al. Evaluation of accelerometer-based fall detection algorithms on real-world falls.

Pietro Bonazzi et al. Openwatch: A multimodal benchmark for hand gesture recognition on smartwatches.

Centers for Disease Control and Prevention (CDC). National health and nutrition examination survey data, 2010.

Shing Chan et al. Capture-24: A large dataset of wrist-worn activity tracker data collected in the wild for human activity recognition.

Charikleia Chatzaki et al. Human daily activity and fall recognition using a smartphone’s acceleration sensor.

Springer, 2016.

Ting Chen et al. A simple framework for contrastive learning of visual representations.

Mostafa Dehghani et al. Scaling vision transformers to 22 billion parameters.

Mohammad Malekzadeh et al. Dana: Dimension-adaptive neural architecture for multivariate sensor data.

Mike A Merrill et al. Homekit2020: A benchmark for time series classification on a large mobile sensing dataset with laboratory tested ground truth of influenza infections.

Girish Narayanswamy et al. Scaling wearable foundation models.

Yuqi Nie et al. A time series is worth 64 words: Long-term forecasting with transformers.

Francisco Javier Ordo´nez and Daniel Roggen. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition.

Veronika Potter et al. The physical activity assessment using wearable sensors (paaws) dataset: Labeled laboratory and free-living accelerometer data.

This paper is currently under review.

Kaiming He et al.

Improvements for AI systems

Variable Input Temporal Masked Autoencoder (VarMAE) enables a single Transformer encoder-decoder to operate across different sampling rates and window durations, rather than specializing to a fixed sensing configuration. This allows the AI system to perform activity recognition or gesture recognition using data collected under arbitrary recording conditions without requiring setting-specific pretraining.
Cross-Resolution Masked Reconstruction learns representations by using the visible regions from the long view condition the prediction of the masked tokens of the short view, which bridges these bottlenecks between different sampling rates and temporal contexts. This results in higher representation quality, particularly at low sampling rates, improving performance when deploying models on devices with lower data throughput.
The architecture features a variable-rate tokenizer that maps rate-dependent patches to a fixed dimension token vector, allowing the model to handle inputs at various sampling rates while maintaining a consistent encoder structure. This means the system can be deployed across different hardware platforms without requiring model adaptation for resampling or padding.
The method demonstrates broad and parameter-efficient transfer across eight benchmarks spanning 1–10 s window sizes, showing that a single foundation model can achieve broad transfer capabilities across sensing configurations. This efficiency allows for the creation of a single, powerful, general-purpose accelerometer FM, reducing the need to maintain multiple specialized models.
The system's performance is robust to changes in masking ratios, as shown by Table 7 results indicating that the performance is largely insensitive to the masking ratio in both views at higher sampling rates but becomes critical at lower rates. This provides a mechanism where the model learns to retain effectiveness even when less temporal resolution is available during downstream evaluation.

Abstract

Large-scale repositories of wearable sensor data have enabled the development of foundation models for human activities and behavior. However, existing models are pretrained using fixed input durations and sampling rates, despite substantial heterogeneity in recording conditions, downstream tasks, and deployment constraints. We introduce VarMAE, a variable input masked autoencoder that learns a single representation across sampling rates and temporal contexts. During pretraining we construct two views of the same signal: a long duration high frequency view, and a short duration lower frequency view, obtained via random cropping and downsampling. We perform masked reconstruction within each view and introduce a cross-resolution masked reconstruction objective that uses visible tokens from the long view to predict masked tokens from the short view. Cropping and downsampling impose complementary information bottlenecks, encouraging the model to learn representations that remain effective across input durations and sampling rate changes. We pretrain VarMAE on the large-scale NHANES dataset, producing a single pretrained encoder matching or outperforming controlled foundation model baselines on eight benchmark datasets. With only 3.82M parameters, VarMAE is parameter efficient and shows that a single foundation model can achieve broad transfer capabilities across sensing configurations.

Variable Input Masked Autoencoders for Wrist-Worn Accelerometry

Listen

Transcript

Key concepts

Summary

Improvements for AI systems

Abstract

Sources

Related papers