Is Self-Pretraining really useful to improve diagnosis in medical Time Series?

The paper is authored by Omar Coser, Antonio Orvieto, Paolo Soda, and Loredana Zollo, affiliated with Università Campus Bio-Medico di Roma, Umeå University, the Max Planck Institute for Intelligent Systems, and the ELLIS Institute Tübingen. The paper is identified as arXiv:2608.06122v1 [cs.LG], dated 6 Aug 2026.

The abstract states: "Inspired by recent evidence that transformer architectures benefit from Self-PreTraining (SPT) on long-context benchmarks, we investigate whether similar gains extend to multimodal, multivariate, and even simple univariate medical time series. Our objective is to assess the impact of SPT on the performance and scalability of transformer-based models across diverse medical applications, particularly under limited data conditions. The authors evaluate transformer architectures on three representative medical time-series tasks: rehabilitation robotics (Camargo dataset), stress detection (Non-EEG Stress), and Parkinson's disease detection (Gait Parkinson's Disease v1.0.0). Models are trained either from scratch or through SPT using four masking-based objectives designed to promote temporal and cross-modal representation learning, with model depth systematically varied to examine how capacity interacts with pre-training benefits."

The key finding is that "Across datasets and configurations, SPT consistently improves classification accuracy by 0–6 percentage points depending on masking strategy, dataset and architecture, with gains observed not only in multivariate settings but also when models are restricted to simple univariate inputs. The improvements are reported to increase for deeper models that can better exploit the enriched temporal representations learned during pre-training. The authors conclude that SPT is a simple and general strategy that enhances transformer performance on medical time-series tasks without requiring task-specific architectural changes, supporting its potential to improve robustness and accuracy in data-limited clinical settings."

The paper draws an important conceptual distinction between Self-Supervised Learning (SSL) and Self-PreTraining (SPT). The authors write: "SSL typically refers to large scale pre-training performed on an external dataset that differs from the one used during downstream fine-tuning. In contrast, SPT denotes a setting in which the model is pre-trained directly on the same dataset that will later be used for fine tuning. They elaborate that SSL can be viewed as out of domain representation learning, whereas SPT corresponds to in domain pre-training. This distinction is considered particularly important in real world time series applications, since in healthcare external datasets may be limited, heterogeneous, or inaccessible due to privacy constraints. In such cases, SPT provides a practical and effective alternative by maximizing the utility of the available in domain data."

The authors state that to the best of our knowledge, no prior work has systematically examined Self-PreTraining (SPT) in the context of medical time series, and they aim to fill this gap with five objectives: comparing transformer models trained from scratch with SPT initializations; evaluating SPT in the simplest univariate setting; extending the analysis to multivariate time series using four masking strategies; assessing whether model scale influences SPT effectiveness; and providing an optimization-based interpretation of performance differences.

Three publicly available medical time-series datasets are used, summarized in Table II of the paper:

1. CAMARGO 2021: A multimodal collection of sensor recordings from 21 participants, involving locomotion classification across five tasks (level walking, ramp ascent/descent, stair ascent/descent). Each subject wore four IMUs located on the trunk, thigh, shank, and foot, along with 11 EMG sensors yielding 35 synchronized signals. Continuous recordings were divided into non-overlapping 100 ms windows, yielding roughly 5,000 samples per subject with an average class probability of 20±2.5%. Evaluation uses Leave-One-Subject-Out (LOSO).

2. Non-EEG Dataset for Assessment of Neurological Status: A PhysioNet collection with recordings from 20 healthy adult participants, measuring electrodermal activity (EDA), skin temperature, three-axis wrist acceleration, heart rate (HR), and arterial oxygen saturation (SpO2). Recordings span seven consecutive stages designed to elicit varying physical and emotional responses, including rest, physical stress, cognitive stress (serial subtraction and Stroop test), and emotional stress (horror-movie clip). Windowed into temporal windows of 10 seconds, producing a naturally imbalanced distribution of approximately 60% control and 40% Stress. Evaluation also uses LOSO.

3. Gait in Parkinson's Disease v1.0.0: A PhysioNet dataset with multichannel recordings of foot-ground reaction forces collected from 93 patients diagnosed with idiopathic Parkinson's Disease... and 73 healthy control subjects. Under each foot, eight force sensors yield 16 individual sensor outputs in addition to two combined sum signals, digitized at 100 samples per second. Recordings are segmented into temporal windows of 500 samples (corresponding to 5 s), yielding a naturally imbalanced distribution of approximately 69% control and 31% Parkinson's. The class distribution is approximately 69% control and 31% Parkinson's in our windowed dataset—reflecting the underlying proportions of the two groups. No artificial class balancing or reweighting is applied during window generation.

The architecture (Figure 1) processes a univariate or multivariate time series X ∈ Rn×f, where n denotes the sequence length (window size) and f the number of channels (with f = 1 in the univariate case). Key components include: a learnable linear projection Rf → Rd mapping raw sensor measurements into a d-dimensional latent space; positional encoding injected elementwise to the projected embeddings to provide temporal order; layer normalization; learned Query, Key, Value projections; multi-head self-attention to capture long-range temporal dependencies and cross-channel interactions within the window; an output projection W out with residual connections; temporal pooling; position-wise feed-forward networks with GELU activation and residual connections; and a decoder head consisting of a linear transformation followed by a ReLU activation and a final output layer. During SPT this head predicts the masked input values (reconstruction objective), while during fine-tuning or training from scratch it maps the pooled latent representation to class logits for supervised classification.

The authors note they chose this architecture because is the one used in the paper [1] on which they showed of on a syntetic dataset the transformer was the architecture that was taking more benefit from SPT. By "maintaining an identical Transformer backbone across all training regimes, we isolate the effect of initialization and eliminate architectural confounders, ensuring a controlled and fair comparison between SPT and training from scratch."

Four masking strategies are proposed, all removing approximately the same proportion of input elements but differing fundamentally in their structural organization:

1. Point-wise masking: each scalar entry is independently masked with probability p, producing a fine-grained, unstructured occlusion pattern. This resembles classical denoising autoencoding and encourages distributed, locally coherent representations.

2. Column masking (also referred to as feature masking): entire feature channels are removed from the input sequence... once a feature is selected, its entire temporal trajectory is hidden for that sequence. This means the model cannot rely on that sensor at any time point and must reconstruct its values using information from the remaining channels, thereby encouraging the model to learn cross-channel relationships and capture dependencies between different sensors or modalities.

3. Block masking: "removes contiguous temporal segments across all features. Instead of selecting isolated timesteps, we iteratively sample starting indices and mask blocks of length L along the time axis until approximately p · t timesteps are covered. This concentrates supervision on longer structured gaps and requires the model to infer extended temporal dynamics rather than isolated points."

4. Mixed Masking Policy: combines multiple axes via the logical union of independently sampled masks: M mix = M point (p1) ∨ M column (p2) ∨ M feature (p3), with individual rates p1, p2, p3... chosen such that the expected overall masking proportion is approximately equal to the target ratio p. This simultaneously introduces fine-grained perturbation, structured temporal gaps, and cross-channel occlusion.

Evaluation protocols: CAMARGO uses Leave-One-Subject-Out (LOSO) cross-validation strategy for its multiclass locomotion task. Non-EEG also applies LOSO for binary stress classification. For Parkinson's, "LOSO was not feasible due to dataset organization and sample distribution constraints. Instead, we performed repeated subject-wise splits, ensuring that all samples from a given subject were assigned exclusively to either training or testing."

Metrics: Performance was evaluated using accuracy, precision, recall, and F1-score. For the multiclass CAMARGO dataset, metrics were computed using macro-averaging.

Training configuration: All experiments ran on an NVIDIA A100 GPU with 80 GB of memory. Training from scratch used 10 epochs with a learning-rate sweep over of 0.01. SPT pre-training used 100 epochs using a learning rate of 0.001, and fine-tuning used 15 epochs with a learning rate of 0.01. Early stopping monitored validation loss, terminating if no improvement was observed for 100 consecutive epochs.

Statistical analysis: We employed the Wilcoxon signed-rank test with Bonferroni correction to account for multiple comparisons, with significance denoted by asterisks (* p ≤ 0.05, ** p ≤ 0.01, *** p ≤ 0.001, **** p ≤ 0.0001).

For univariate experiments, the authors selected the most informative individual sensor channel based on prior domain knowledge and preliminary performance screening for each dataset: Gyroscope signals from the foot-mounted IMU (Y-axis and Z-axis) for CAMARGO; Accelerometer (AccY) and Heart Rate (HR) signals for Stress; and LeftFoot and RightFoot force-derived gait signals for Parkinson. Transformer models with 1, 2, and 3 layers were trained under both initialization regimes.

Dataset-specific observations:

CAMARGO: The CAMARGO dataset exhibits periodic gait structure, which naturally aligns with the temporal modeling capacity of Transformers. SPT consistently improves performance, particularly for the Y-axis gyroscope signal, where gains increase with model depth. For the Y-axis foot gyroscope, improvements range from 1.9% at depth 1 to 5.3% at depth 2 to 9.9% (p ≤ 0.0001) at depth 3. The Z-axis shows more modest gains from 1.2% to 3.8%.

Stress: For the accelerometer signal, baseline performance is already relatively strong due to clear motion-related stress patterns, and SPT yields consistent but moderate improvements (typically 2–4%), particularly at shallow depths. For HR, improvements are smaller (1–2.5%) and become more apparent at higher depths. The limited temporal landmarks in HR signals constrain the representational benefit of SPT.

Parkinson: Unlike other datasets, models trained from scratch benefit substantially from increased depth. Accuracy improves markedly from one to three layers even without SPT initialization. However, SPT still provides significant improvements at intermediate depth (two layers), particularly for the LeftFoot dataset, where gains exceed 6% with high statistical significance (6.8%, p ≤ 0.0001). At three layers, improvements diminish, indicating that increased depth alone already captures much of the relevant temporal structure.

Depth-dependent behavior: "Figure 3 shows that the average improvement provided by SPT increases from one to two layers and stabilizes thereafter. This suggests that SPT interacts synergistically with moderate model depth: shallow models lack sufficient capacity to fully exploit structured initialization, while deeper models already capture temporal dependencies effectively, reducing marginal gains. Importantly, SPT never degrades performance relative to training from scratch."

Multivariate experiments trained transformers jointly on all available sensor channels comparing from-scratch training with four SPT variants (Point, Block, Column, Mixed) at depths 1, 2, and 3. "During pre-training, all SPT models are optimized using a Masked Mean Squared Error (Masked MSE) loss, computed exclusively over masked elements. This ensures that the model is explicitly trained to reconstruct missing values rather than copying visible inputs."

Global observations: "Across all three datasets, SPT variants consistently outperform training from scratch. However, the magnitude and scaling of improvements depend on: (i) the intrinsic temporal structure of the task, (ii) the redundancy and synchronization across channels, (iii) the structural bias imposed by the masking strategy. Camargo and Parkinson exhibit strong periodic structure (gait cycles), while Stress contains heterogeneous physiological signals (EDA, HR, accelerometry) with weaker cross-channel alignment."

SPT-Point: yields consistent but moderate improvements. The gains are particularly visible in Camargo (depth 2: 0.828 → 0.848, p ≤ 0.05). Improvements do not scale aggressively with depth because pointwise masking does not explicitly enforce structured temporal or cross-channel reasoning.

SPT-Block: performs particularly well in temporally structured datasets such as Parkinson. At depth 2, accuracy reaches 0.964 (p ≤ 0.001), substantially exceeding from-scratch training. The scaling reveals that improvements peak at intermediate depth. This suggests that block masking efficiently encodes dominant temporal structure early, but additional layers do not proportionally increase the benefit.

SPT-Column: demonstrates strong gains in Camargo and Parkinson, both of which contain synchronized multi-sensor measurements. At depth 3 in Camargo, accuracy reaches 0.869 (p ≤ 0.001). It shows the strongest depth-dependent growth among all strategies. This indicates that deeper Transformers are particularly effective at exploiting structured cross-channel priors introduced by column masking.

SPT-Mixed: provides the most stable improvements across heterogeneous datasets, particularly in Stress. At depth 3, accuracy reaches 0.788 (p ≤ 0.01), outperforming individual masking strategies. The scaling varies by dataset. Parkinson peaks at intermediate depth (due to already strong baseline structure), while Stress continues improving with depth, reflecting its heterogeneous multimodal composition.

Interpretation: the effectiveness of SPT depends on the alignment between masking structure and task structure. In strongly periodic datasets (Parkinson, Camargo), block and column masking are particularly effective because they align with temporal and cross-channel regularities. In heterogeneous physiological datasets (Stress), mixed masking performs best because no single structural prior dominates. Furthermore, "As model depth increases, the performance gap between SPT and from-scratch models widens. This indicates that structured pre-training improves optimization stability and allows deeper Transformers to better exploit their representational capacity without overfitting."

The authors provide a theoretical interpretation: "From an optimizatio…

Improvements to AI systems:

• Add a Self-PreTraining (SPT) stage on the target medical time-series dataset before supervised fine-tuning, using masked reconstruction objectives. The system now pre-trains on the same data it will later classify, eliminating the need for external or out-of-domain datasets.

• Replace training-from-scratch initialization with SPT-based initialization to act as a structured warm start. The system avoids poor local minima, lowers gradient variance, and reduces loss landscape sharpness, improving optimization stability and generalization.

• Dynamically select masking strategy based on data structure:

• Block masking for temporally periodic signals (e.g., gait) → improves modelling of long-range temporal dynamics.

• Column masking for synchronized multi-sensor data → forces cross-channel reconstruction, learning inter-sensor dependencies.

• Point-wise masking for fine-grained robustness in locally structured signals.

• Mixed masking for heterogeneous physiological data (e.g., stress detection with EDA, HR, accelerometry).

• Scale model depth when using SPT: since SPT gains grow with depth (e.g., +9.9% accuracy on CAMARGO at 3 layers vs. +1.9% at 1 layer), the system can use deeper transformers without overfitting, exploiting enriched temporal representations.

• Apply SPT in univariate settings as well: the system can pre-train on a single informative channel (e.g., foot gyroscope Y-axis) and still improve classification, which is useful when only one sensor stream is available.

• Use Masked MSE loss only over masked tokens during pre-training, so the system explicitly learns to reconstruct missing values rather than copy visible inputs—this prevents shortcut learning and forces genuine temporal/cross-modal understanding.

• Support data-limited clinical scenarios by leveraging in-domain SPT: the system no longer requires large external corpora, making it viable under privacy constraints and when labeled samples are scarce (e.g., 21 subjects in CAMARGO, 20 in Non-EEG Stress).

What the improved AI system can do:

• Achieve higher classification accuracy on medical time-series tasks compared to training from scratch, with improvements of 0–10 percentage points depending on dataset, masking, and depth.

• CAMARGO locomotion classification: 0.869 accuracy with column masking at depth 3.

• Parkinson’s disease detection from gait force sensors: 0.964 accuracy with block masking at depth 2.

• Stress detection from physiological signals: 0.788 accuracy with mixed masking at depth 3.

• Maintain or boost performance even with simple univariate inputs (e.g., a single accelerometer or gyroscope channel), enabling deployment with minimal sensor setups.

• Generalize across diverse medical applications—rehabilitation robotics, stress monitoring, and Parkinsonian gait analysis—without task-specific architecture changes.

• Train deeper transformer models stably and effectively on small, imbalanced clinical datasets, reducing overfitting and improving robustness.

• Operate in privacy-constrained environments because all pre-training data is internal to the target dataset, avoiding external data sharing or transfer.

• Automatically adapt its pre-training objective to the underlying structure of the data (temporal continuity, cross-sensor correlation, or heterogeneous mixtures), maximizing downstream classification performance.

Inspired by recent evidence that transformer architectures benefit from Self-PreTraining (SPT) on long-context benchmarks, we investigate whether similar gains extend to multimodal, multivariate, and even simple univariate medical time series. Our objective is to assess the impact of SPT on the performance and scalability of transformer-based models across diverse medical applications, particularly under limited data conditions. We evaluate transformer architectures on three representative medical time-series tasks: rehabilitation robotics (Camargo dataset), stress detection (Non-EEG Stress), and Parkinson's disease detection (Gait Parkinson's Disease). Models are trained either from scratch or through SPT using four masking-based objectives designed to promote temporal and cross-modal representation learning, and we systematically vary model depth to examine how capacity interacts with pre-training benefits. Across datasets and configurations, SPT consistently improves classification accuracy by 0-6 percentage points depending on masking strategy, dataset and architecture, with gains observed not only in multivariate settings but also when models are restricted to simple univariate inputs. The improvements increase for deeper models that can better exploit the enriched temporal representations learned during pre-training. These findings indicate that SPT is a simple and general strategy that enhances transformer performance on medical time-series tasks without requiring task-specific architectural changes, supporting its potential to improve robustness and accuracy in data-limited clinical settings.