MOTOR: A Multimodal Dataset for Two-Wheeler Rider Behavior Understanding

arXiv:2605.22550 · cs.CV · Submitted 2026-05-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "MOTOR: A Multimodal Dataset for Two-Wheeler Rider Behavior Understanding".

Jane: Here's the gist of the paper. Two-wheelers account for a disproportionately high share of road fatalities in the Global South MOTOR comprises 1,629 sequences (25+ hours of video data) collected from 16 riders and integrates synchronized front, rear, and helmet videos, rider eye-gaze from wearable trackers, on-road audio, and telemetry Rich annotations capture traffic context, rider state, 12 riding maneuvers spanning conventional and unconventional behaviors, and legality labels (Legal, Illegal, Unspecified).

Tom: Dataset Introduction The MOTOR dataset is the first large-scale resource dedicated to two-wheelers in dense, unstructured traffic.

Jane: It comprises 1,629 sequences collected from 16 riders and integrates synchronized multi-view videos (ego-vehicle front, rear, and helmet views), rider eye-gaze from wearable trackers (Aria

19: , Pupil

20: ), on-road audio capturing ambient sounds and distractions, and telemetry signals (GPS, accelerometer, gyroscope) for speed, location, and lean angles.

Tom: The dataset is richly annotated with traffic scene context, rider state, six conventional and six unconventional behaviors, along with their legality labels (Legal, Illegal, Unspecified).

Title and authors: Tom: So we're looking at this paper, "MOTOR: A Multimodal Dataset for Two-Wheeler Rider Behavior Understanding." It seems like they've put together something pretty big for the two-wheeler community in dense traffic areas.

Jane: That's right, Tom. The core idea here is addressing a gap because research on two-wheeler rider behavior hasn't kept up with four-wheelers as much as it should have been, especially when we talk about using multimodal datasets to build things like advanced driver assistance systems.

Lu: Essentially, they present the MOTOR dataset, which is described as the first large-scale resource dedicated specifically to two-wheelers in dense traffic scenarios. It’s a massive effort because it brings together a lot of different data types for one specific vehicle type.

Meng: So what's the actual scope of this dataset? What kind of data are we talking about, Jane? Because if it's just videos, we already have plenty of that stuff out there.

Jane: It’s much more than just videos, Meng. MOTOR integrates synchronized front and rear views from the ego-vehicle, a helmet-mounted view for the rider's perspective, rider eye-gaze captured from wearable trackers like Aria or Pupil Labs glasses, on-road audio for ambient sounds and distractions, and telemetry signals like GPS and accelerometer data which gives us speed and lean angles.

Tom: That multi-view setup is key there. And they've made sure everything is synchronized, which is crucial for linking the visual action to the rider's physical state in real time.

Lu: Exactly, Tom. The richness of the annotations is also a major point; they capture traffic scene context, rider state information, twelve different riding maneuvers spanning both conventional and unconventional behaviors, and they even have legality labels like Legal or Illegal for each action.

Jane: So when we talk about what this dataset actually lets us do, it’s not just recognizing if someone is turning; it’s looking at the whole picture of how a rider is moving through a complex environment.

Tom: Right, and that's where the benchmarking comes in. They used state-of-the-art video action recognition backbones like CNN based Sthree dee, ResNetthree dee, and Transformer based Video Swin Transformer to test their approach.

Meng: So they tested it against some pretty heavy hitters in the video recognition space. I wonder if that level of testing is realistic for real traffic deployment.

Lu: They did compare those backbones against each other, and they found that combining the RGB visual data with the eye gaze and telemetry cues consistently yielded the best performance across both recognizing rider behaviors and classifying maneuver legality.

Jane: That combination of visual information, eye tracking attention, and kinematic data is what makes this resource so powerful for safety analysis.

Tom: Their proposed baseline architecture is a three stream late fusion architecture that specifically integrates the ego-vehicle frontal view, the rider eye gaze data, and the vehicle telemetry signals.

Meng: How does that specific fusion look in practice? Is it just stacking them together or something more complex? What about computational load?

Title and authors: Lu: It’s a late fusion scheme where you take the outputs from those separate streams—visual features, gaze embeddings, telemetry features—and combine them later in the process. This approach is what they used to test their main model.

Jane: And when we look at the results from that fusion, it shows significant gains because they are leveraging different types of information simultaneously.

Tom: They found that using Swin Transformer with all these modalities together achieved an accuracy of fifty-two point nine percent in behavior classification, which is pretty solid for a complex task like this.

Meng: Fifty-three percent sounds okay on paper, but what about the real-world reliability? What are the numbers on the legality classification part? That’s where it matters most for safety.

Lu: For the legality classification specifically, full fusion using Swin Transformer reached an accuracy of sixty-nine point zero percent and an F1 score of fifty-three point six percent, which they found to be the strongest performance across all their experiments on this dataset.

Jane: Sixty-nine percent for legality classification means the system is pretty good at telling a rider if they are following traffic rules or not, based on what we see in the data.

Tom: That’s the main contribution, I think. Introducing MOTOR as this first large-scale, multi-view resource specifically for two-wheelers in dense traffic and providing these detailed annotations.

Meng: So what are they suggesting we should actually do with this dataset? What's the next step beyond just getting the numbers on accuracy?

Lu: They suggest that combining attention cues from gaze and kinematic data with visual information significantly improves rider behavior prediction, which is a big mechanism for their success here.

Jane: It means that just looking at the video isn't enough; we need to know where the rider is looking and how they are physically moving simultaneously to truly understand what they’re doing.

Tom: They also pointed out that gaze provides complementary attention cues, telemetry captures dominant kinematic patterns like lean and speed, but they also acknowledged that unconventional behaviors remain challenging to classify because of their high variability.

Meng: Variability is always the issue when you try to train a model on real-world data; it’s hard to generalize those tricky maneuvers.

Lu: So, in terms of future work, they plan to expand the dataset by including more diverse weather and lighting conditions, which is something that would definitely stress-test their current findings.

Jane: And they also mentioned planning to improve the baseline architecture by fusing even more camera streams and modalities to see if we can get even better results than what they achieved.

Tom: They’re looking at expanding the input space, which makes sense because real traffic isn't always perfectly sunny and clear.

Meng: But from a deployment standpoint, how much does that expansion impact the training time or the cost of running these models in production? That’s my concern.

Title and authors: Lu: The paper doesn't dive into the operational costs, but expanding the data diversity is clearly intended to make the underlying representation learning more robust to real-world noise.

Jane: So, when we wrap up on this paper, what’s the big picture implication for people who aren't building these AI systems?

Tom: The main point is that we have a much better foundation now for understanding two-wheeler behavior in complex traffic because of this MOTOR dataset and the multimodal fusion approach.

Meng: It means safety applications that rely on recognizing rider actions could get much more reliable, provided they can handle that level of data richness.

Lu: I think it sets a new standard for what a comprehensive dataset for a specific domain should look like when we're aiming for high accuracy in behavior recognition.

Jane: So, to sum up the findings on this paper "MOTOR: A Multimodal Dataset for Two-Wheeler Rider Behavior Understanding," it’s about creating a rich, multi-view resource that uses synchronized video, gaze, audio, and telemetry to understand rider behavior and legality in two-wheelers.

Tom: Right. It shows that fusing those different data types is the way to go when you're trying to predict maneuvers or check compliance.

Meng: And it proves that using advanced transformer architectures like Swin Transformer can handle all that complexity pretty well on this specific task.

Lu: The paper clearly shows the value of combining attention cues from eye gaze with kinematic patterns from telemetry, which helps separate similar actions more effectively.

Jane: It gives us a concrete way to think about how multimodal inputs help build a system that can actually assess whether a rider is behaving legally or illegally on the road.

Tom: So it’s less about just seeing what's happening and more about understanding the rider's state and intent through all those different lenses.

Meng: And for us engineers, it means we have a solid benchmark to start training models that actually consider those multiple inputs instead of just relying on one type of sensor data.

Lu: The paper also flags that unconventional behaviors are still tricky because they are so variable, which is an honest assessment of the current limitations.

Jane: So, as we look ahead at this work, they’re focusing on expanding the data diversity and trying to fuse even more streams to push those accuracy numbers higher.

Tom: It’s a solid piece of work for building a foundation in this area, especially given how much two-wheelers are involved in traffic fatalities globally.

Meng: Yeah, it's good data, but remember that running these models at scale and making them reliable on real roads is the next big hurdle for any AI application built on something like MOTOR.

Lu: That’s true; the dataset is just the first step to building something that actually works reliably when it matters most.

Jane: That's what we'll be watching as they work on those future expansions and architecture improvements for this paper, "MOTOR: A Multimodal Dataset for Two-Wheeler Rider Behavior Understanding."

The paper's summary: Tom: So, we’re looking at the summary of this paper, "MOTOR: A Multimodal Dataset for Two-Wheeler Rider Behavior Understanding." Basically, they’ve created this massive data set that brings together videos, eye tracking data from glasses like Aria and Pupil Labs gear, audio from the road, and telemetry signals like speed and lean angles.

Jane: That sounds incredibly rich. So what is the main goal of putting all that stuff together? What problem are they trying to solve with this whole collection?

Tom: The big problem they're tackling is getting a really good understanding of how people actually ride two-wheelers in busy, messy traffic environments. It moves beyond just seeing a video and tries to capture the rider’s physical state and what’s going on around them simultaneously.

Lu: They call it the first large scale resource for two-wheelers in dense traffic because they did this careful collection process using three different camera views, plus that eye-gaze data, and they made sure everything was synchronized.

Jane: So if I’m a driver listening right now, what does this actually help me understand about the riders around me? Like, is that person going to make a sudden lane change or are they just cruising?

Tom: Well, this dataset lets AI systems look at those twelve different riding maneuvers—both the standard stuff like turning and overtaking and the weird stuff like maybe weaving—and assign labels to them as legal or illegal.

Lu: And that legality labeling is important because it means the system isn't just classifying actions; it’s assessing compliance with traffic rules based on what’s in the scene.

Jane: That’s a big difference, so it goes from just describing *what* happened to understanding *if* what happened was safe or not according to the rules.

Tom: Right. And when they tested their models, they found that combining those different types of input—the visual video, the gaze data, and the telemetry like speed and lean angles—actually gave them better results than using just one piece of data at a time.

Lu: They used transformer architectures like Swin Transformer for testing because those ones seem to handle the complexity of dense traffic really well across all those different inputs.

Jane: So what do these sixty-nine percent accuracy numbers for legality mean in practical terms? What’s the real-world impact on someone who just listens to the show?

Tom: It means that if we build an AI system to help monitor two-wheeler traffic, using this approach gives us a much more reliable way to check if a rider is following traffic laws compared to older systems that only looked at video.

Lu: The paper points out that the gaze data provides attention cues, which helps the model focus on important parts of the scene, while the telemetry captures those physical patterns like how much lean a rider is taking or how fast they are going.

Jane: It sounds like we’re moving toward systems that don't just see a person on a bike but actually understand their body language and physics in real time.

Tom: Exactly. And they also gave us some numbers showing how much performance drops if you take one of those important pieces away, for example, removing the gaze data hurts accuracy by about one point six percent.

Lu: That’s a strong signal that the eye-gaze input isn't just noise; it’s actually adding meaningful information to the model.

Jane: It gives us a concrete way to think about how multimodal inputs build a system that can assess whether a rider is behaving legally or illegally on the road.

Tom: So, for anyone working on safety systems, this paper provides a solid foundation for training models that actually consider those multiple inputs instead of just relying on one type of sensor data.

Lu: And they acknowledged that unconventional behaviors are still tricky to classify because they change so much from one rider to the next.

Jane: So, as we look ahead, what’s the focus now? What’s next for this research?

Tom: They're planning to make the dataset more robust by adding more data under different weather and lighting conditions, which is something that would really test how well these models generalize.

Lu: And they also want to try fusing even more camera streams and modalities in their next version of the baseline architecture to see if they can push those accuracy numbers even higher.

Jane: It sounds like they’re trying to build a system that’s not only smart but also tough enough to handle the messy reality of real-world traffic, not just perfect, clear conditions.

The paper's improvements: Tom: We’re moving on to what they think is next for this research, focusing on how they can make this system even better, specifically about improving the way it classifies those rider maneuvers and legality labels.

Jane: So what are the specific tweaks they suggest to push those accuracy numbers even higher? What are they trying to fix with these improvements?

Lu: They focus on combining the cues more intelligently. They say that combining the attention cues from eye gaze with the kinematic data, like how fast you’re going or how much you’re leaning, really improves the model's ability to separate similar actions.

Tom: That makes sense. It’s not just having all the data; it’s learning how to weigh that gaze information against the physical measurements of speed and lean angle.

Lu: Right. And they specifically mention that using a transformer backbone like Swin Transformer with all these different modalities together consistently gives them the best accuracy and F1 score for classifying legality.

Jane: So, it’s not just about adding more data; it’s about refining the way the AI looks at all that data simultaneously.

Tom: They also suggested doing a detailed analysis of which input is actually doing the most work, showing how performance changes when you remove one of them.

Lu: That helps them prove that those different cues—the gaze versus the telemetry—are providing distinct, complementary pieces of information for understanding what’s happening on the road.

Jane: That gives us a much clearer picture of what each piece of sensor data contributes to the final decision, which is really useful for engineers building these kinds of safety systems.

Tom: And they pointed out that while unconventional behaviors are still very hard to classify because they vary so much, this framework is a solid starting point for tackling those tricky situations.

Lu: For future work, they want to make the dataset more diverse by including more varied weather and lighting conditions, which would test how well these models generalize outside of perfect sunny days.

Jane: So the next step is testing this system in much messier real-world scenarios, not just controlled lab settings.

Tom: Exactly. It’s about making sure that when we deploy these models on actual roads, they don't suddenly fail because it’s raining or the light is bad.

Lu: And they also mentioned planning to improve the baseline architecture by fusing even more camera streams and different types of data sources to try and find even better performance.

Jane: It sounds like they are constantly pushing the boundaries of what’s possible with multimodal learning for this specific task.

Conclusion: Tom: So we’re wrapping up on "MOTOR: A Multimodal Dataset for Two-Wheeler Rider Behavior Understanding." To recap, they built this massive collection of synchronized videos, gaze data, audio, and telemetry to understand two-wheeler riders in dense traffic.

Jane: It really shows how important it is to have this kind of comprehensive data when you’re trying to build AI that interacts with real-world driving situations.

Lu: The main takeaway is that combining visual information with attention cues from eye gaze and kinematic patterns like lean angles gives the best results for both recognizing behaviors and checking legality.

Tom: It's clear they are using a three stream late fusion architecture to put all those different data types together in a way that actually works well.

Meng: From an engineering standpoint, this is promising because it gives us a high-quality benchmark, but I still have to ask what the operational cost is for running these kinds of models in production at scale.

Lalam: Based on the literature, I’m quite sure this dataset establishes a new standard for what a comprehensive resource for specific domains like two-wheelers should look like when you're aiming for high accuracy.

Jane: It’s about moving past just looking at video and really understanding the rider's physical state and intent through all those different lenses.

Tom: Exactly. And they also flagged that unconventional behaviors are still hard to classify because they change so much, which is an honest assessment of the current limitations.

Lu: I agree, but that’s what we expect from real-world data; it’s a sign the model is learning something more complex than just simple motion patterns.

Meng: If this system can handle that variability reliably on real roads, then it means we could see safety applications become much more robust.

Jane: So, for people who just listen to the show, this paper gives us a concrete way to think about how multimodal inputs help build a system that can actually assess whether a rider is behaving legally or illegally on the road.

Tom: It’s less about just seeing what’s happening and more about understanding the rider's state and intent through all those different lenses.

Lalam: I am sure this work lays down a strong foundation for safety modeling in this area, based on how well they integrated those different data sources.

Lu: The future work they mentioned focusing on weather and lighting diversity is crucial because that’s where you really test if the representation learning is actually robust.

Jane: It sounds like the next step is making sure this understanding holds up when things aren't perfect conditions.

Tom: We’ll keep an eye on those expansions, because getting that real-world reliability from a model like this is the actual hurdle we need to clear for safety systems.

Varun A. Paturkar, Shankar Gangisetty, C.V. Jawahar

IIIT Hyderabad

cs.CV

Submitted: 2026-05-21

Updated: 2026-11-14

Journal ref: 2026 IEEE International Conference on Robotics and Automation (ICRA), Vienna, Austria, 2026, pp. 5406-5413

DOI: 10.1109/ICRA57385.2026.11696157

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: Two-wheelers account for a disproportionately high share of road fatalities in the Global South MOTOR comprises 1,629 sequences (25+ hours of video data) collected from 16 riders and integrates…

Key concepts

MOTOR Dataset
The first large-scale dataset specifically designed for two-wheelers in dense traffic. It collects 1,629 sequences integrating video (front, rear, helmet), rider gaze, audio, and telemetry. This provides rich context for studying how riders behave while riding motorcycles or scooters.
Multimodal Fusion
The technique of combining different types of data—like video pixels (RGB), eye-tracking data (gaze), and vehicle signals (telemetry)—into a single model. The paper found that combining these modalities, especially gaze and telemetry, leads to superior performance in recognizing rider maneuvers compared to using any single data source alone.
Rider Maneuver Classification
The task of identifying and categorizing the specific actions a two-wheeler rider performs during a ride. This includes both standard riding techniques (conventional) and more complex or risky actions (unconventional). The system aims to accurately label these 12 different maneuvers.
Legality Annotation
A classification layer that assesses whether a detected rider maneuver is compliant with traffic laws. Annotations are labeled as Legal, Illegal, or Unspecified. This allows the system to not just recognize what happened but also judge if it was safe or rule-breaking behavior.

Terminology

Summary

Two-wheelers account for a disproportionately high share of road fatalities in the Global South MOTOR comprises 1,629 sequences (25+ hours of video data) collected from 16 riders and integrates synchronized front, rear, and helmet videos, rider eye-gaze from wearable trackers, on-road audio, and telemetry Rich annotations capture traffic context, rider state, 12 riding maneuvers spanning conventional and unconventional behaviors, and legality labels (Legal, Illegal, Unspecified). p. 0

Dataset Introduction

The MOTOR dataset is the first large-scale resource dedicated to two-wheelers in dense, unstructured traffic. It comprises 1,629 sequences collected from 16 riders and integrates synchronized multi-view videos (ego-vehicle front, rear, and helmet views), rider eye-gaze from wearable trackers (Aria [19], Pupil [20]), on-road audio capturing ambient sounds and distractions, and telemetry signals (GPS, accelerometer, gyroscope) for speed, location, and lean angles. p. 0 The dataset is richly annotated with traffic scene context, rider state, six conventional and six unconventional behaviors, along with their legality labels (Legal, Illegal, Unspecified). p. 0

Data Collection and Annotation

The MOTOR dataset was collected using a multi-modal multi-view setup with three GoPro-10 cameras and wearable eye-tracking glasses. Video data includes front and rear views of the ego-vehicle, a helmet-mounted view, rider gaze data from Aria [19] or Pupil [20], audio capturing ambient sounds, and telemetry signals like GPS and IMU. p. 0 Annotations are categorized into four types: Traffic Scene and Rider State Annotations, Conventional Rider Behaviors, Unconventional Rider Behaviors, and Legality Annotations. p. 3

Evaluation Methodology

The work benchmarks rider behavior recognition on the MOTOR dataset using state-of-the-art action recognition backbones: CNN-based S3D [21], ResNet3D [22], and Transformer-based Video Swin Transformer [23], MViTv2 [24]. The authors find that combining RGB, gaze, and telemetry consistently yields the best performance across both behavior recognition and maneuver legality classification. p. 0 The proposed baseline architecture is a three stream late-fusion architecture integrating ego-vehicle frontal-view, rider eye-gaze, and vehicle telemetry. p. 4

Key Findings and Contributions

The main contributions include introducing MOTOR, the first large-scale dataset for two-wheelers in dense traffic. The research shows that combining attention (gaze) and kinematic (telemetry) cues with visual information significantly improves rider behavior prediction. p. 6 Specifically, using SwinT with all modalities achieves an accuracy of 52. p. 69% in behavior classification, demonstrating the effectiveness of full fusion. p. 6 Furthermore, full fusion on SwinT yields the strongest performance for legality classification, reaching 69. p. 60% accuracy and 53. p. 16% F1 score. p. 6

Future Directions

In future work, the authors aim to expand on the dataset with diverse weather and lighting conditions. They also plan to improve the baseline architecture by fusing more camera streams and modalities. p. 7 The study highlights that gaze provides complementary attention cues, telemetry captures dominant kinematic patterns such as lean and speed, and unconventional behaviors remain challenging to classify due to their variability. p. 7

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved Rider Behavior Classification through Multimodal Fusion

The improved system can classify rider maneuvers into 11 classes spanning conventional and unconventional behaviors by integrating video, gaze, and telemetry embeddings in a late-fusion scheme. This fusion leverages gaze provides complementary attention cues, telemetry captures dominant kinematic patterns such as lean and speed, leading to better separation of similar actions like lane change versus overtake.

  1. Enhanced Maneuver Legality Prediction

The system can predict whether a rider maneuver is Legal, Illegal, or Unspecified by explicitly assessing compliance with traffic rules using the MOTOR dataset annotations. This goes beyond behavior recognition to provide a legality-aware analysis of how riders behave and comply with traffic rules, which is crucial for safety-critical applications.

  1. Robust Model Selection via Transformer Architectures

The system can be optimized by employing state-of-the-art transformer backbones like Swin Transformer, as they outperforms accuracy and F1 score using all data modalities consistently. This suggests that hierarchical attention makes the model more robust to the noise and ambiguity found in dense, unstructured traffic scenarios.

  1. Modality Importance Analysis

The system can perform an exhaustive analysis of modality contribution by examining how performance changes when modalities are removed, such as Removing gaze reduces accuracy by 1.6%, while removing telemetry leads to a 2.6% drop in accuracy. This provides deep insights into the role of each cue for two-wheeler safety modeling.

Related papers