QuAnTS: Question Answering on Time Series

The gist The authors propose QuAnTS, a challenging novel time series QA (TSQA) dataset for Question Answering on Time Series data, specifically focusing on human motion in tracked skeleton trajectories, to bridge the gap between numerical time series and intuitive textual interaction. This work matters because it provides a large-scale dataset that encourages future research into interacting with time series models through text, enabling better decision-making and more transparent systems.

Motivation

Language is a highly intuitive medium of communication (Fedorenko et al., 2024). Time series are complex and precise pieces of data that are highly relevant in many applications, but end users often require specific training and tooling to interact with them (Eggleton, 1982; Harold et al., 2015). The authors propose QuAnTS because very little work has been performed in the direction of enriching data with further context and task descriptions (Han et al., 2023). They attribute this lack to a lack of sufficient high-quality data, which limits both training and evaluation opportunities (Divo et al., 2025). QuAnTS uses human motion in the form of tracked skeleton positions as the type of signal to reason about, which is natural to visualize while still challenging for machines (Divo et al., 2025).

Dataset Construction

The creation of QuAnTS involved a multi-step process designed for diversity and scale (Figure 2). The dataset comprises triplets of (1) a numerical time series of synthetic human motion tracking data, (2) a textual question, and (3) an answer, which can be binary, multiple-choice, or open format (Divo et al., 2025). They generated 30.000 time series contexts of four consecutive actions each, totaling approximately 133.3 hours (Divo et al., 2025). The diversity is ensured through sampling and diverse templating, with a balanced distribution of 46 question types and three answer types (Table 1) (Divo et al., 2025).

Question Types

The authors identified five main question categories to build the dataset (Figure 3) (Divo et al., 2025). These categories include:

1. Descriptive questions, which ask about the name of a performed action, how often it occurred, where it is happening, or insight based on metadata (Divo et al., 2025).

2. Temporal questions, which ask about the temporal ordering of events or if specific actions were performed in the very beginning or end (Divo et al., 2025).

3. Comparison questions, which ask whether some actions are the same/distinct or if they occur the same/different number of times (Divo et al., 2025).

4. Reasoning questions, which ask for higher-level concepts in the actions, like predicting the next one, counterfactual reflections, or providing reasons for a person’s behavior (Divo et al., 2025).

5. Overarching questions are a future type that either combines multiple aspects or asks absurd questions (Divo et al., 2025).

Evaluation and Baselines

The authors conducted extensive experiments to validate the dataset and provide stable baselines (Divo et al., 2025). They evaluated three increasing capabilities:

1. A naive LLM that sees the serialized numerical time series values and the question (Naive Baseline) (Divo et al., 2025).

2. ChatTS, a representative of current state-of-the-art TS-LLMs, which is designed to ingest time-series tokens and allows for very general chat-like interaction contextualized by multivariate numerical data (Divo et al., 2025).

3. xQA, a neuro-symbolic pipeline that augments a general LLM with structured action information extracted from the time series (Divo et al., 2025).

Human Performance

The authors provide human performances as a key reference for gauging the practical usability of such models (Divo et al., 2025). In Section C.2, they hired a diverse cohort of participants to evaluate binary and multi variants (Divo et al., 2025). Results generally confirm that humans can solve the dataset in the provided format, with F1 scores being overall good for the binary and multi splits (Divo et al., 2025). For the open variant, they measured a moderate LLMJudge score of 0.6156 (Divo et al., 2025). The study also verified that humans can correctly identify the individual actions in the sequences (Divo et al., 2025).

Conclusion and Outlook

The work concludes by highlighting the need for dedicated models, as naive baselines struggle with QuAnTS (Divo et al., 2025). The authors identify two primary avenues for future research: broadening the scope of TSQA to include other fields and more complex questions, and building truly multimodal models for time series and text (Divo et al., 2025). They anticipate that this work will have a positive impact on applications such as personalized health recommendations or financial insights (Divo et al., 2025). The dataset is freely available under the permissive CC BY 4.0 license, with a DOI of 10.57967/hf/6663 (Divo et al., 2025).

--- Page 1 ---

--- Page 2 ---

<ref:2511.05124#pg3>

--- Page 3 ---

<ref:2511.05124#pg4>

--- Page 4 ---

<ref:2511.05124#pg5>

--- Page 5 ---

How it works

The QuAnTS dataset is generated in several steps (Figure 2) (Divo et al., 2025). An action sequence is sampled first, where for each we sample five question and answer types (Divo et al., 2025). Each of these is then instantiated from a sampled template to increase diversity (Divo et al., 2025). The time series from the human motion diffusion is then combined with the QA-pair and auxiliary data (Divo et al., 2025).

The generation pipeline involves:

1. Sampling an action sequence, where for each we sample five question and answer types (Divo et al., 2025).

2. Instantiating each question and answer type from a sampled template to ensure diversity (Divo et al., 2025).

3. Generating the time series from human motion diffusion (Divo et al., 2025).

4. Combining the QA-pair with auxiliary data (Divo et al., 2025).

Question & Answer Generation Pipeline

The authors defined five main question categories to obtain a challenging dataset (Figure 3) (Divo et al., 2025). These categories are:

1. Descriptive questions, which ask about the name of a performed action, how often it occurred, where it is happening, or insight based on metadata (Divo et al., 2025).

2. Temporal questions, which ask about the temporal ordering of events or if specific actions were performed in the very beginning or end (Divo et al., 2025).

3. Comparison questions, which ask whether some actions are the same/distinct or if they occur the same/different number of times (Divo et al., 2025).

4. Reasoning questions, which ask for higher-level concepts in the actions, like predicting the next one, counterfactual reflections, or providing reasons for a person’s behavior (Divo et al., 2025).

Leveraging Motion Diffusion Models for Time Series Context

To generate well-recognizable human motions, they opted to use Spatio-Temporal Motion Collage (STMC) by Petrovich et al. (2024) (Divo et al., 2025). They used 19 particularly distinct action prompts commonly found in STMC’s training data to build action sequences (Divo et al., 2025). Each context time series was obtained by concatenating four action prompts of four seconds each from unique random seeds (Divo et al., 2025). The final trajectory has a dimension of 320 × 24 × 3 at 16 seconds at 20 frames per second (Divo et al., 2025).

Dataset Statistics and Diversity Analysis

The QuAnTS dataset is large and varied, guaranteeing diversity through the balanced distribution of question types (Table 1) (Divo et al., 2025). They generated 30.000 time series contexts, each with five question-answer pairs, resulting in a total of 150.000 instances (Divo et al., 25). Question and open answer lengths are diverse (Figure 4) (Divo et al., 25).

Metrics for Evaluating Answers

They grade answers to binary and multiple-choice questions using accuracy, macro-averaged precision, recall, and F1 scores (Divo et al., 2025). For open questions, they employ ROUGE and METEOR metrics augmented with a semantic comparison using an LLM judge (Divo et al., 2025).

1. Bold header: Multimodal Time Series Question Answering (TSQA) models can be developed to complement the density of numerical time series, thereby allowing improved interactions with time series models to enhance accessibility and decision-making. This allows for back-and-forth discussions contextualized by data, such as What is the person doing first? followed by a question about subsequent actions.

2. Bold header: LLM adaptation for motion data can be enhanced by using xQA, a neuro-symbolic pipeline that augments a general-purpose LLM with structured action information extracted from the time series. This system allows models to successfully solve QuAnTS, where the key step forward must allow for the reliable identification of these actions.

3. Bold header: Action encoders can be trained on ground-truth action labels to predict actions from time series segments, allowing systems to leverage the structure identified in QuAnTS. This enables a model to perform action extraction from the time series before feeding it into an LLM for reasoning, addressing the need for dedicated models that can receive both numerical and textual modalities.

Text offers intuitive access to information. This can, in particular, complement the density of numerical time series, thereby enabling improved interactions with time series models to enhance accessibility and decision-making. While the creation of question-answering datasets and models has recently seen remarkable growth, most research focuses on question answering (QA) on vision and text, with time series receiving minute attention. To bridge this gap, we propose a challenging novel time series QA (TSQA) dataset, QuAnTS, for Question Answering on Time Series data. Specifically, we pose a wide variety of questions and answers about human motion in the form of tracked skeleton trajectories. We verify that the large-scale QuAnTS dataset is well-formed and comprehensive through extensive experiments. Thoroughly evaluating existing and newly proposed baselines then lays the groundwork for a deeper exploration of TSQA using QuAnTS. Additionally, we provide human performances as a key reference for gauging the practical usability of such models. We hope to encourage future research on interacting with time series models through text, enabling better decision-making and more transparent systems.