Player Disjoint Multi-Task Tennis Stroke Analysis from Monocular Pose Sequences
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Player Disjoint Multi-Task Tennis Stroke Analysis from Monocular Pose Sequences".
Jane: Here's the gist of the paper. We present a multi-task pipeline for tennis stroke biomechanics analysis from monocular RGB video <ref:2606.15992#pg2> The system performs complete biomechanical characterization of each stroke without manual segmentation <ref:2606.15992#pg4>.
Tom: Pipeline Overview The pipeline consists of four sequential stages: (1) pose extraction, (2) automated stroke segmentation, (3) sequence feature construction, and (4) multi-task transformer inference <ref:2606.15992#pg5>.
Jane: All frames are processed without temporal subsampling <ref:2606.15992#pg5>.
Tom: Pose Representation and Feature Extraction MediaPipe Pose Landmarker provides 33 landmarks in metric world coordinates, adding z-axis depth and visibility scores unavailable in prior frameworks <ref:2606.15992#pg8>.
Paper summary: Tom: So we're looking at this paper now called "Player Disjoint Multi-Task Tennis Stroke Analysis from Monocular Pose Sequences." The main point is they built a pipeline that analyzes tennis strokes from just one video camera feed without needing anyone to manually draw the stroke boundaries.
Jane: That sounds pretty ambitious, Tom. So what exactly are they trying to characterize about the stroke?
Tom: They claim this system performs complete biomechanical characterization of each individual stroke automatically. It does this by detecting the stroke boundaries themselves using a weighted joint velocity score s(t) which is defined as zero point five times wrist velocity plus zero point three times elbow velocity plus zero point two times shoulder movement.
Lu: And they use MediaPipe Pose Landmarker to get thirty-three landmarks in metric world coordinates, which adds that depth information and visibility scores that were missing from older methods.
Meng: Wait, so they're using those world coordinates instead of just image plane landmarks? That sounds important for accuracy.
Tom: It is because the paper says replacing those with image plane landmarks drops cross-player stroke type accuracy from eighty three percent down to forty seven percent and direction accuracy from sixty eight percent to twenty one percent, which shows how much that metric matters.
Jane: So they are arguing that those world coordinates give them a better picture of the action than just where the joints look on the flat video frame.
Lalam: And they say this is critical because it's what helps maintain accuracy when comparing different players or types of strokes, like forehand versus backhand.
Tom: Okay, so once they have those pose features, they feed them into something called TennisTransformerGPU. This is a transformer encoder with three parallel task heads.
Lu: That means they are trying to do three different things at the same time: classifying the stroke type, predicting the shot direction, and assessing the posture quality of that stroke.
Jane: Three tasks in one model sounds like a lot of work for an AI system to handle simultaneously. What’s the cost or what kind of data did they train this thing on?
Tom: They trained it on one thousand two hundred eighty-one annotated strokes from seven professional players and one amateur player. They use a model that is five hundred sixty four thousand one hundred thirty parameter, which has four layers, four heads, and a dimension of twelve eight.
Meng: One million two hundred eighty-one data points for seven pro players? That’s a pretty solid dataset if you can get that kind of annotation done consistently. But how does this translate to actual use on the floor?
Jane: The paper mentions they used a class-weighted loss because their direction labels were imbalanced, with seventy eight five instances of center shots and only two hundred ninety seven for cross court shots.
Lu: And they also noted that the stroke type labels were skewed too, with seventy seven percent being forehand strokes and only two hundred ninety nine for backhand strokes.
Tom: The system uses this class-weighted loss to motivate the learning process for those tasks, which helps when you have uneven data like that.
Meng: What about deployment? I mean, if I take this model and try to run it on a real production line, what does the running time look like? Is it constant time or something else?
Tom: They state that the pipeline processes every frame without any temporal subsampling at all, so they are achieving constant time processing for the sequence construction part.
Jane: That sounds efficient for real-time analysis during a match, but what about where this actually changes things for someone watching tennis who doesn't understand transformer encoders?
Lu: It means they are taking something that used to require manual segmentation and replacing it with an automatic detection method based on joint velocity scores.
Tom: They use the weighted score s(t) which is zero point five times wrist velocity plus zero point three times elbow velocity plus zero point two times shoulder movement to detect stroke boundaries.
Jane: So, for a listener, this means the computer figures out when one stroke ends and another begins just by watching how fast the player's joints are moving.
Meng: That’s cool because it removes the need for that manual segmentation step, which is usually a huge bottleneck in these kinds of projects.
Lu: And then they use this sequence X, which is a sequence of thirty times thirty-nine elements, to feed into the transformer encoder for those three tasks.
Tom: The transformer has those three parallel heads—stroke type, direction prediction, and posture quality—all trained together using a joint loss function.
Jane: And they introduce a deterministic coaching layer on top of that. That sounds like it’s not just classifying things but actually giving feedback during the game.
Meng: A rule-based decision table maps the direction and posture outputs to one of four coaching messages, which is what translates the AI's analysis into something actionable for a coach or player.
Tom: The practical implication they highlight is the cross-player experiment, where they tested professional players against amateur players.
Jane: And that test showed an eighty three point seven percent stroke type accuracy with only a zero point eight percentage point drop, which they call strong generalization.
Lu: They also found the most practically significant result was the pro to amateur transfer result, dropping from eighty three point seven percent down to eighty two point nine percent stroke type accuracy.
Meng: So, while it's good that it generalizes well, that small drop between pros and amateurs is what they are focusing on for real-world coaching scenarios where you don't have tons of professional data.
Tom: The paper also flags some limitations, though. Camera viewpoint dependency is a problem because performance degrades when the camera angle changes to side-on or front-facing views.
Jane: And they also mention that the full sequence direction modeling approach doesn't perfectly match how localized directional cues actually work in the real game.
Lu: So, for future work, they suggest implementing contact window direction modeling and adopting regression based posture assessment instead of just classification.
Tom: It sounds like the core idea here is combining better pose data with sequence modeling to achieve simultaneous analysis of what's happening—what stroke it is, where it's going, and how good the body mechanics are.
Jane: And the authors are being pretty transparent about what they couldn't measure well, like not measuring inter-rater reliability for the posture task because they did all those annotations themselves.
Meng: That’s a fair limitation to point out when you’re using self-made labels for quality assessment. What does this mean for the people who are going to use this?
Tom: It means that while the pipeline is powerful, it relies heavily on getting those initial annotated strokes right, and if your camera setup changes drastically, the accuracy will drop off significantly.
Jane: So, to wrap up on this "Player Disjoint Multi-Task Tennis Stroke Analysis from Monocular Pose Sequences," we see a system that automatically segments tennis strokes and uses a multi-task transformer to simultaneously figure out the stroke type, direction of the shot, and how good the posture is.
Lu: The authors are pushing for future work that focuses on modeling direction more locally around contact points instead of just using the whole sequence.
Meng: For deployment right now, I'd be worried about that camera viewpoint dependency and whether running a model with five hundred sixty four thousand one hundred thirty parameters is something we can realistically put on a device in the field without massive latency.
Tom: It’s definitely complex, but the fact that they achieved those accuracy numbers using world coordinates instead of image plane landmarks is a solid piece of evidence for how to approach this kind of biomechanical analysis.
Jane: So, the big picture is that we have a way to get deep biomechanical insights into tennis strokes from simple video recordings without needing tedious manual labeling for every single stroke.
Conclusion: Tom: So we've been looking at this paper called "Player Disjoint Multi-Task Tennis Stroke Analysis from Monocular Pose Sequences." Basically, they took one video and figured out how to automatically track and analyze every single tennis stroke without anyone having to draw anything.
Jane: That's a big claim, Tom. So what are the actual results showing here? What did they figure out about the players?
Tom: They used this whole pipeline—pose extraction, automatic segmentation based on joint movement speeds, and then feeding that into a transformer model—to do three things at once: figure out the stroke type, predict which way the shot was going, and judge how good the posture is.
Lu: What's interesting is how they did the segmentation; they used a score based on wrist velocity plus elbow velocity plus shoulder movement to find where one stroke ends and another begins. It’s an automated way to cut out the manual labeling step.
Meng: And what does that mean for us in production? We're talking about constant time processing because they aren't subsampling the video, which is pretty efficient for real-time analysis, but I wonder how stable that segmentation is when a player moves quickly.
Jane: Right, the speed of movement is key. And then they tested this system on professional players versus amateurs to see if it generalizes well.
Tom: They found that the stroke type accuracy was eighty three point seven percent across both groups, and the most important part is that when comparing pros to amateurs, they only saw a tiny drop of zero point eight percentage points in stroke type accuracy.
Lu: That small drop is what they call strong generalization for stroke type classification. It shows the model isn't just memorizing pro movements; it's learning the underlying mechanics.
Jane: But there are still limitations, Tom. They admit the camera viewpoint matters a lot, and they also say that modeling direction using the whole sequence isn't perfectly aligned with how quick directional cues actually work during a fast shot.
Tom: Yeah, they flagged that full sequence direction modeling as being architecturally misaligned with localized directional cues. That’s their biggest sticking point right now.
Meng: So, even though the stroke type works pretty well across skill levels, the system isn't perfectly reliable for predicting a precise shot direction if the camera angle is awkward or if the player makes a very fast swing.
Jane: Exactly. The authors are pointing toward future work like modeling direction more locally around where contact happens and using regression instead of just classification for posture quality assessment.
Tom: So, this paper gives us a working framework to analyze tennis biomechanics automatically, but it still has room to improve on the localized directional prediction and handling different camera setups.
Department of Artificial Intelligence and Machine Learning, BMS Institute of Technology and Management
cs.CV
Submitted: 2026-06-14
Updated: 2026-11-05
Comments: 17 pages, 8 figures, 7 tables
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 86/100
The gist: We present a multi-task pipeline for tennis stroke biomechanics analysis from monocular RGB video <ref:2606.15992#pg2> The system performs complete biomechanical characterization of each stroke…
Key concepts
- Pose Representation
- The system uses MediaPipe Pose Landmarker to extract 33 metric world coordinates from the video frames. These coordinates include depth information, which is crucial because using image-plane landmarks severely degrades accuracy in determining stroke type and direction.
- Automated Stroke Segmentation
- Stroke boundaries are automatically detected by calculating a weighted joint velocity score based on wrist, elbow, and shoulder movement. Local peaks in this score above a player-specific threshold mark the beginning or end of a stroke event.
- Multi-Task Transformer Inference
- A transformer encoder with three parallel heads simultaneously predicts stroke type, direction (3 classes), and posture quality. These tasks are trained together using a combined loss function, allowing the model to learn related biomechanical features at once.
Terminology
Summary
We present a multi-task pipeline for tennis stroke biomechanics analysis from monocular RGB video <ref:2606. p. 015992#pg2> The system performs complete biomechanical characterization of each stroke without manual segmentation <ref:2606. p. 015992#pg4>.
Pipeline Overview
The pipeline consists of four sequential stages: (1) pose extraction, (2) automated stroke segmentation, (3) sequence feature construction, and (4) multi-task transformer inference <ref:2606.15992#pg5>. All frames are processed without temporal subsampling <ref:2606. p. 215992#pg5>.
Pose Representation and Feature Extraction
MediaPipe Pose Landmarker provides 33 landmarks in metric world coordinates, adding z-axis depth and visibility scores unavailable in prior frameworks <ref:2606.15992#pg8>. These world coordinates are critical, as replacing them with image-plane landmarks drops cross-player stroke-type accuracy from 83% to 47% and direction accuracy from 68% to 21% <ref:2606. p. 015992#pg8>.
Automated Segmentation and Sequence Building
Stroke boundaries are detected automatically via a weighted joint velocity score s(t) = 0.5 · vwrist + 0. p. 03 · melbow + 0. p. 02 · mshoulder <ref:2606. p. 015992#pg4>. Local peaks exceeding a player-adaptive threshold τ are identified as stroke events, with thresholds set differently for professional and amateur players <ref:2606. p. 515992#pg5>. Each stroke is encoded as a fixed-length sequence X ∈ R 30×39 <ref:2606. p. 515992#pg6>.
Multi-Task Learning Architecture
TennisTransformerGPU is a transformer encoder with three parallel task heads, covering stroke type (nc = 2), direction (nc = 3), and posture quality (nc = 2) <ref:2606.15992#pg7>. The three heads are trained jointly using a loss L = Ltype + Ldir + Lposture <ref:2606. p. 615992#pg8>. A deterministic coaching layer maps direction and posture outputs to one of four coaching messages via a hand-crafted decision table <ref:2606. p. 515992#pg8>.
Generalization and Practical Implications
The cross-player experiment (professional → amateur) yields 83.7% stroke-type accuracy with a 0. p. 08 percentage-point drop, demonstrating strong generalization <ref:2606. p. 015992#pg9>. The pro-to-amateur stroke-type transfer result (83. p. 77% → 82.9%) is the paper’s most practically significant finding <ref:2606. p. 715992#pg8>. This has direct implications for low-resource coaching scenarios in which professional training data is abundant but labeled amateur data is unavailable <ref:2606. p. 615992#pg8>.
Task Performance Analysis
Table 4 reports per-task accuracy under both evaluation protocols with majority-class baselines <ref:2606.15992#pg9>. Direction accuracy of 68. p. 03% under cross-player evaluation reflects collapse to the majority class rather than genuine directional transfer <ref:2606. p. 415992#pg9>. Posture accuracy of 63. p. 04% is achieved under both random and cross-player splits <ref:2606. p. 715992#pg9>.
Limitations and Future Directions
Camera viewpoint dependency is a limitation, as performance degrades on side-on or front-facing angles <ref:2606.15992#pg8>. The full-sequence direction modeling approach is architecturally misaligned with the localized nature of directional cues <ref:2606. p. 915992#pg9>. Priority future directions include implementing contact-window direction modeling and adopting regression-based posture assessment <ref:2606. p. 015992#pg9>.
Comparison with Prior Work
The present system differs from Shimizu et al. (2019) in several key aspects, including using MediaPipe Pose Landmarker instead of OpenPose <ref:2606. p. 915992#pg9>. The pipeline design is superior as it incorporates automated velocity-based player-adaptive thresholds for segmentation <ref:2606. p. 915992#pg9>. The model utilizes a multi-task transformer encoder instead of an LSTM, allowing simultaneous classification of stroke type, direction, and posture quality <ref:2606. p. 915992#pg9>.
Data Availability
The code and dataset used in this study are available on Kaggle and can be reproduced using the pipeline described in the Materials and Methods section <ref:2606.15992#pg8>. The complete pipeline runs on Kaggle free-tier T4 GPU hardware and is openly reproducible <ref:2606. p. 115992#pg4>.
Author Contributions
JH is responsible for conceptualization, methodology, software, validation, formal analysis, investigation, data curation, writing — original draft <ref:2606.15992#pg8>. The research received no external funding <ref:2606. p. 015992#pg8>. The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest <ref:2606.15992#pg8>.
References
Faulkner, H. and Dick, A. p. 1 (2017). TenniSet: A dataset for dense fine-grained event recognition, localisation and description <ref:2606. p. 115992#pg9>.
Huang, Y.-C., Liao, I.-N., Chen, C.-H., İk, T-Ú., and Peng, W-C. (2019). TrackNet: A deep learning network for tracking high-speed and tiny objects in sports applications <ref:2606. p. 115992#pg9>.
Lugaresi, C. p. 1, Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., et al. (2019). MediaPipe: A framework for building perception pipelines <ref:2606. p. 015992#pg9>.
Parmar, P. and Morris, B. p. 1 T- (2019). Action quality assessment across multiple actions <ref:2606. p. 115992#pg9>.
Shimizu, T. p. 1, Hachiuma, R., Saito, H., Yoshikawa, T., and Lee, C. (2019). Prediction of future shot direction using pose and position of tennis player <ref:2606. p. 115992#pg9>.
Vinyes Mora, S. p. 1 and Knottenbelt, W. p. 1 J- (2017). Deep learning for domain-specific action recognition in tennis <ref:2606. p. 115992#pg9>.
Zhang, Y. and Yang, Q- (2017). p. 1 A survey on multi-task learning <ref:2606. p. 015992#pg9>.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Automated stroke segmentation eliminates manual pre-annotation by using a
velocity-based detector
defined bys(t) = 0.5 · vwrist + 0.3 · melbow + 0.2 · mshoulder.
This allows the system to performcomplete biomechanical characterization of each stroke without manual segmentation.
-
Enhanced pose representation uses metric world coordinates from MediaPipe Pose Landmarker, which are demonstrated to be critical for cross-player generalization, as replacing them with image-plane landmarks
drops cross-player stroke-type accuracy from 83% to 47% and direction accuracy from 68% to 21%.
-
The system incorporates a
two novel learned tasks plus a rule-based feedback layer,
including dedicated heads for shot direction (cross-court, down-the-line, center) and posture quality (good/bad), which providesdeterministic coaching layer requiring no additional labels.
-
Training configuration uses class-weighted cross-entropy for the posture head, applying a
2.5× upweight for the 'bad' class
to address label imbalance, leading to more reliable identification of poor technique (correctly flagging poor technique is the most coaching-actionable outcome
). -
The pipeline enables
pro-to-amateur stroke-type transfer (83.7% → 82.9%),
making it viable forlow-resource coaching deployments in which professional training data is abundant but labeled amateur data is unavailable.
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs