BEAST3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: I'm Ines.
Marcus: And I'm Marcus, here with Yuki.
Ines: We're looking at the paper: "BEAST3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting".
Marcus: Here's the gist of the paper. Multi-view video recordings are increasingly used to capture the 3D movements of animals in experimental settings, yet extracting rich 3D representations from these recordings remains challenging The authors address these limitations with BEAST3D, a self-supervised pretraining framework that learns 3D visual representations from unlabeled, calibrated multi-view video BEAST3D reconstructs 3D structure with as few as four views by conditioning directly on known camera parameters—unlike general-purpose models, which must estimate camera geometry from dense overlapping viewpoints that are seldom available in lab settings Through comprehensive evaluation across four species, they demonstrate that BEAST3D produces rich, viewpoint-invariant features that transfer effectively to three downstream tasks: novel view synthesis, multi-view pose estimation, and neural encoding The Core Methodology BEAST3D is a self-supervised pretraining framework that learns 3D visual representations from calibrated multi-view animal behavior videos Given only unlabeled, synchronized multi-view images and corresponding camera parameters, BEAST3D predicts a set of 3D Gaussian splats via a transformer trained to reconstruct held-out views through differentiable rendering The model simultaneously segments the animal from the background by distilling masks from a video segmentation model, producing clean foreground representations This framework builds on the 3DGS framework of E-RayZer
35: , but replaces learned components with frozen foundation models and ground truth calibration data The architecture involves image tokenization using a frozen DINOv3 ViT-B/16 as the image encoder Camera tokenization uses Plücker coordinates to provide geometric grounding, which are then fused with image tokens via a two-layer MLP The fused tokens are processed by a geometry transformer that alternates between frame attention and global attention The final output decodes these tokens into per-patch 3D Gaussian parameters via a linear head, followed by differentiable rendering using GSplat
44: Training Signal Construction The training loss L is computed only on the held-out target views T and combines three terms: (i) a photometric loss Ll2 that penalizes pixel-level differences between the ground truth and rendered images, (ii) a perceptual loss Lperc that penalizes abstract feature-level differences between ground truth and rendered images, and (iii) a mask loss Lmask that penalizes differences between the rendered alpha channel and the SAM3 segmentation masks The final loss is a weighted combination of each: L = λl2 Ll2 + λperc Lperc + λmask Lmask They train BEAST3D using the AdamW optimizer with a base learning rate of 5 × 10−5, weight decay of 0.05, and a cosine learning rate schedule with 15% warmup for 800 epochs During each forward pass, they randomly sample Vref = V − 1 reference views and hold out Vtgt = 1 target view Performance Across Downstream Tasks BEAST3D is evaluated on three downstream tasks: novel view synthesis, multi-view pose estimation, and neural encoding In novel view synthesis, BEAST3D produces sharp reconstructions of held-out viewpoints that recover the subject’s silhouette and fine appearance details across all four datasets For pose estimation, BEAST3D achieves the best performance across nearly all datasets, with the exception of Rat7M where VGGT performs best In neural encoding, BEAST3D features outperform 3D keypoints for predicting neural activity in both datasets (Cheese3D and Chickadee), demonstrating that dense 3D representations capture behavioral information that sparse keypoints discard Comparative Analysis with Baselines BEAST3D is compared against several baselines, including E-RayZer, Pose Splatter, and VGGT In novel view synthesis on the InD Test set, BEAST3D outperforms both Pose Splatter and E-RayZer across all datasets For pose estimation, BEAST3D consistently outperforms BEAST across all datasets, demonstrating the benefit of multi-view pretraining However, a striking finding is that multi-view models more broadly (VGGT and E-RayZer) do not consistently outperform single-view DINOv3 and BEAST baselines Inference Efficiency and Scalability The inference compute cost shows that BEAST3D and E-RayZer have comparable compute and memory profiles, with the DINOv3 frontend in BEAST3D adding a roughly constant per-view memory and latency overhead The model's ability to reconstruct 3D structure with as few as four views by conditioning directly on known camera parameters is a key advantage over general-purpose models A current limitation of BEAST3D is its compute requirements: pretraining on a single dataset takes roughly 32 hours on 8 A100 GPUs, placing it out of reach for many labs Broader Implications The BEAST3D framework enables more efficient extraction of meaningful information from multiview video data, potentially accelerating behavioral neuroscience research with several beneficial outcomes By reducing the need for extensive human labeling while improving accuracy, BEAST3D can democratize advanced video analysis capabilities for laboratories with limited resources However, concerns remain regarding surveillance capabilities, bias and fairness across demographic groups, and resource inequality due to high pretraining compute requirements Datasets Used The evaluation spans four datasets spanning species, environments and camera configurations (Cheese3D, Rat7M, Chickadee, and Human3.6M) These datasets include both unconstrained naturalistic behavior and controlled lab settings The dataset construction pipeline involves five stages: cut, downsample, segment, assemble, and resize to produce the calibrated (frame, mask, camera) tuples consumed by BEAST3D Pose Estimation Details For pose estimation training/test sets on Cheese3D and Human3.6M datasets, they constructed a pseudo-labeled dataset from an initial set of 665 instances where each keypoint is labeled in a subset of views depending on anatomical visibility They apply k-means clustering in the space of flattened 3D keypoints for each session to select representative frames for training and testing The final pose estimation training/test sets consist of 450/150 instances for Cheese3D, 455/177 instances for Rat7M, and 1125/425 instances for Human3.6M Neural Encoding Details For neural encoding on Cheese3D and Chickadee datasets, they segment the spike trains into non-overlapping 2.0 s windows and partition them into train, val, or test splits The encoding model is intentionally lightweight and input-agnostic so that it does not itself learn 3D structure that could compensate for a weak input They use the foreground-filtered BEAST3D point cloud as a baseline, contrasting it with multi-view-triangulated 3D keypoints produced by supervised pose estimation models The resulting per-frame point clouds, together with their integer frame indices, are concatenated across the session and serve as the visual input to the encoding model Novel View Synthesis Metrics Novel view synthesis performance is benchmarked using Peak signal-to-noise ratio (PSNR), Structural similarity index measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) BEAST3D results show substantial improvements in metrics across all datasets in the within-subject evaluation setting compared to baselines Table 5 and Table 6 provide quantitative comparisons of these metrics for both InD Test and OOD Test sets Pose Estimation Evaluation Protocol Pose estimation is evaluated using the limited-data evaluation protocol of the original BEAST paper, where they train three models on random subsets of only 100 labeled instances and evaluate on held-out test subjects Results are reported using difficulty-stratified error curves, which reveal how each method handles keypoints of varying difficulty rather than collapsing performance into a single number Neural Encoding Evaluation Protocol For neural encoding, they split neural activity from each session into non-overlapping 2 s chunks and randomly assign each chunk to a train, val or test split They tune model hyperparameters on the train/val splits and report results on the test split using the Bits Per Spike (BPS) metric BEAST3D features outperform 3D keypoints for predicting neural activity in both datasets, and per-neuron comparisons confirm that this gain is consistent across the population rather than driven by a small subset of well-predicted neurons Ablation Study Findings Ablations show that removing the DINOv3 encoder substantially degrades pose estimation performance (Fig.
Ines: 9) Removing the frustum constraint has minimal effect on most datasets but is critical for Chickadee, where the bird is cropped from large images using a per-frame bounding box Inference Cost Comparison Figure 7 compares inference compute cost vs.
Title and authors: Ines: So we're talking about the paper "BEASTthree dee: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting." It sounds like this framework tackles a big problem in getting reliable three-dimensional information out of animal videos.
Marcus: Yeah, what I find interesting is that it directly addresses the issue where you need many overlapping views for three dee reconstruction, but those views aren't always available in a lab setting <ref:2606.02937#pg1>.
Ines: Exactly. It suggests that we can reconstruct three dee structure with just four views by using known camera parameters, which is a big deal because it bypasses the need to estimate complex camera geometry from dense overlaps <ref:2606.02937#pg1>.
Marcus: Right. And the authors are Yanchen Wang, Lenny Aharon, Wangshu Zhu, Kyle Daruwalla, Linghua Zhang, Jiaru Zou and Selmaan Chettih. They bring a strong background in computer vision and deep learning.
Yuki: From my side, I'm curious how this work connects to the actual biological data they're using; whether these views are capturing enough information about the animal's movement or behavior for population studies.
The paper's summary: Ines: So, what BEASTthree dee actually does is it’s a self-supervised pretraining framework. It learns to make three dee visual representations from videos that don't have labels, but they are calibrated multi-view videos <ref:2606.02937#pg1>.
Marcus: They use a transformer trained to reconstruct unseen views by doing differentiable rendering. That means the model learns how to generate what the scene looks like from different angles based on those few views.
Ines: And they also do something extra—they simultaneously segment the animal from its background by taking masks from a video segmentation model, which gives them clean representations of just the subject.
Yuki: So, it’s not just about reconstructing a shape; it’s about isolating the specific biological entity we care about from all that background noise in the footage.
The paper's improvements: Ines: The authors point out that they replaced some of the learned components with frozen foundation models and ground truth calibration data. They also used a frozen DINOv3 vision transformer as the image encoder, which helps them get these features.
Marcus: That’s smart because it means they aren't training everything from scratch; they are leveraging existing powerful tools to learn what's important about the visual data.
Ines: And they show that these learned features work really well across four different species: mouse, rat, chickadee and human. The key improvement is that these features transfer effectively to three downstream tasks: novel view synthesis, multi-view pose estimation, and neural encoding.
Yuki: I wonder about the neural encoding part. If their three dee representations are good enough, does it mean we can actually start linking specific brain activity patterns back to observable three dee body parts or actions <ref:2606.02937#pg1>?
Conclusion: Ines: To wrap up, BEASTthree dee shows that you can get rich, viewpoint-invariant features from multi-view data even with just a few views by using known camera parameters.
Marcus: And these features work across novel view synthesis, multi-view pose estimation, and neural encoding tasks. It seems to establish BEASTthree dee as a versatile framework for extracting three dee features from these complex videos <ref:2606.02937#pg1>.
Yuki: From a population genetics perspective, I think this is promising because it could eventually allow us to analyze subtle movement patterns across different species using standardized, high-quality three dee data that we can actually compare statistically <ref:2606.02937#pg1>.
Ines: That’s the big picture. So, we see a framework that makes three dee analysis of animal behavior much more accessible when you have video recordings from multiple cameras, even sparse ones <ref:2606.02937#pg1>.
Marcus: It does mean that if you're working in a lab with limited resources, this could significantly speed up the process of getting reliable three dee data for behavioral studies <ref:2606.02937#pg1>.
Yuki: I think the main thing is how well these features actually generalize when we move from one species to another, which is where the comparisons across mouse and human really matter.
Ines: It’s definitely a versatile tool that moves us closer to having richer, more detailed behavioral data without needing massive amounts of manual labeling for every single frame.
Marcus: We’ll keep an eye on how the compute requirements scale up, because they did mention that pretraining on a single dataset takes about thirty-two hours on eight A100 GPUs, which is still quite demanding.
Yuki: It’s interesting that the paper flags that their neural encoding features actually outperform sparse keypoints for predicting neural activity in both the mouse and chickadee datasets. That suggests that dense three dee information captures more behavioral detail than just a few points do <ref:2606.02937#pg1>.
Ines: Exactly, it implies we should look at how we can use these dense representations to decode what specific body parts or appearance details are driving those neural activities. That’s where the real biological insight lies in this work.
Columbia University
q-bio.NC, cs.CV
Submitted: 2026-06-01
Updated: 2026-11-05
Code: https://github.com/facebookresearch/vggt2https:
Project page: https://ppwangyc.github.io/projects/beast3d
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 86/100
The gist: Multi-view video recordings are increasingly used to capture the 3D movements of animals in experimental settings…
Key concepts
- Self-supervised Pretraining
- BEAST3D learns 3D visual representations without needing manual 3D labels. It uses the video data itself to train a model to predict 3D Gaussian splats from images and camera parameters. This allows the model to learn meaningful spatial relationships directly from the raw, unlabeled video footage.
- Gaussian Splatting (GSplat)
- This is a technique used by BEAST3D to represent 3D objects as a collection of semi-transparent, view-dependent 3D shapes called Gaussian splats. These splats are rendered using differentiable rendering, which allows the model to train the representation by comparing its output against real images.
- Viewpoint-Invariant Features
- The features learned by BEAST3D are robust to changes in viewing angle. This means the representation of an animal's 3D structure remains consistent whether viewed from a close angle or a distant one, making these features highly effective for tasks like pose estimation and neural encoding.
Terminology
Summary
Multi-view video recordings are increasingly used to capture the 3D movements of animals in experimental settings, yet extracting rich 3D representations from these recordings remains challenging The authors address these limitations with BEAST3D, a self-supervised pretraining framework that learns 3D visual representations from unlabeled, calibrated multi-view video BEAST3D reconstructs 3D structure with as few as four views by conditioning directly on known camera parameters—unlike general-purpose models, which must estimate camera geometry from dense overlapping viewpoints that are seldom available in lab settings Through comprehensive evaluation across four species, they demonstrate that BEAST3D produces rich, viewpoint-invariant features that transfer effectively to three downstream tasks: novel view synthesis, multi-view pose estimation, and neural encoding
The Core Methodology
BEAST3D is a self-supervised pretraining framework that learns 3D visual representations from calibrated multi-view animal behavior videos Given only unlabeled, synchronized multi-view images and corresponding camera parameters, BEAST3D predicts a set of 3D Gaussian splats via a transformer trained to reconstruct held-out views through differentiable rendering The model simultaneously segments the animal from the background by distilling masks from a video segmentation model, producing clean foreground representations This framework builds on the 3DGS framework of E-RayZer [35], but replaces learned components with frozen foundation models and ground truth calibration data The architecture involves image tokenization using a frozen DINOv3 ViT-B/16 as the image encoder Camera tokenization uses Plücker coordinates to provide geometric grounding, which are then fused with image tokens via a two-layer MLP The fused tokens are processed by a geometry transformer that alternates between frame attention and global attention The final output decodes these tokens into per-patch 3D Gaussian parameters via a linear head, followed by differentiable rendering using GSplat [44]
Training Signal Construction
The training loss L is computed only on the held-out target views T and combines three terms: (i) a photometric loss Ll2 that penalizes pixel-level differences between the ground truth and rendered images, (ii) a perceptual loss Lperc that penalizes abstract feature-level differences between ground truth and rendered images, and (iii) a mask loss Lmask that penalizes differences between the rendered alpha channel and the SAM3 segmentation masks The final loss is a weighted combination of each: L = λl2 Ll2 + λperc Lperc + λmask Lmask They train BEAST3D using the AdamW optimizer with a base learning rate of 5 × 10−5, weight decay of 0.05, and a cosine learning rate schedule with 15% warmup for 800 epochs During each forward pass, they randomly sample Vref = V − 1 reference views and hold out Vtgt = 1 target view
Performance Across Downstream Tasks
BEAST3D is evaluated on three downstream tasks: novel view synthesis, multi-view pose estimation, and neural encoding In novel view synthesis, BEAST3D produces sharp reconstructions of held-out viewpoints that recover the subject’s silhouette and fine appearance details across all four datasets For pose estimation, BEAST3D achieves the best performance across nearly all datasets, with the exception of Rat7M where VGGT performs best In neural encoding, BEAST3D features outperform 3D keypoints for predicting neural activity in both datasets (Cheese3D and Chickadee), demonstrating that dense 3D representations capture behavioral information that sparse keypoints discard
Comparative Analysis with Baselines
BEAST3D is compared against several baselines, including E-RayZer, Pose Splatter, and VGGT In novel view synthesis on the InD Test set, BEAST3D outperforms both Pose Splatter and E-RayZer across all datasets For pose estimation, BEAST3D consistently outperforms BEAST across all datasets, demonstrating the benefit of multi-view pretraining However, a striking finding is that multi-view models more broadly (VGGT and E-RayZer) do not consistently outperform single-view DINOv3 and BEAST baselines
Inference Efficiency and Scalability
The inference compute cost shows that BEAST3D and E-RayZer have comparable compute and memory profiles, with the DINOv3 frontend in BEAST3D adding a roughly constant per-view memory and latency overhead The model's ability to reconstruct 3D structure with as few as four views by conditioning directly on known camera parameters is a key advantage over general-purpose models A current limitation of BEAST3D is its compute requirements: pretraining on a single dataset takes roughly 32 hours on 8 A100 GPUs, placing it out of reach for many labs
Broader Implications
The BEAST3D framework enables more efficient extraction of meaningful information from multiview video data, potentially accelerating behavioral neuroscience research with several beneficial outcomes By reducing the need for extensive human labeling while improving accuracy, BEAST3D can democratize advanced video analysis capabilities for laboratories with limited resources However, concerns remain regarding surveillance capabilities, bias and fairness across demographic groups, and resource inequality due to high pretraining compute requirements
Datasets Used
The evaluation spans four datasets spanning species, environments and camera configurations (Cheese3D, Rat7M, Chickadee, and Human3.6M) These datasets include both unconstrained naturalistic behavior and controlled lab settings The dataset construction pipeline involves five stages: cut, downsample, segment, assemble, and resize to produce the calibrated (frame, mask, camera) tuples consumed by BEAST3D
Pose Estimation Details
For pose estimation training/test sets on Cheese3D and Human3.6M datasets, they constructed a pseudo-labeled dataset from an initial set of 665 instances where each keypoint is labeled in a subset of views depending on anatomical visibility They apply k-means clustering in the space of flattened 3D keypoints for each session to select representative frames for training and testing The final pose estimation training/test sets consist of 450/150 instances for Cheese3D, 455/177 instances for Rat7M, and 1125/425 instances for Human3. p. 7 For
each backbone we train three models on random subsets of only 100 labeled instances and evaluate
on held-out test subjects. Because pose estimation error varies substantially across keypoints—some
are more inherently ambiguous or occluded—we report results using the difficulty-stratified error
curves of [53], which reveal how each method handles keypoints of varying difficulty rather than
collapsing performance into a single number.Where the paper says it
Neural Encoding Details
For neural encoding on Cheese3D and Chickadee datasets, they segment the spike trains into non-overlapping 2.0 s windows and partition them into train, val, or test splits The encoding model is intentionally lightweight and input-agnostic so that it does not itself learn 3D structure that could compensate for a weak input They use the foreground-filtered BEAST3D point cloud as a baseline, contrasting it with multi-view-triangulated 3D keypoints produced by supervised pose estimation models The resulting per-frame point clouds, together with their integer frame indices, are concatenated across the session and serve as the visual input to the encoding model
Novel View Synthesis Metrics
Novel view synthesis performance is benchmarked using Peak signal-to-noise ratio (PSNR), Structural similarity index measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) BEAST3D results show substantial improvements in metrics across all datasets in the within-subject evaluation setting compared to baselines Table 5 and Table 6 provide quantitative comparisons of these metrics for both InD Test and OOD Test sets
Pose Estimation Evaluation Protocol
Pose estimation is evaluated using the limited-data evaluation protocol of the original BEAST paper, where they train three models on random subsets of only 100 labeled instances and evaluate on held-out test subjects Results are reported using difficulty-stratified error curves, which reveal how each method handles keypoints of varying difficulty rather than collapsing performance into a single number
Neural Encoding Evaluation Protocol
For neural encoding, they split neural activity from each session into non-overlapping 2 s chunks and randomly assign each chunk to a train, val or test split They tune model hyperparameters on the train/val splits and report results on the test split using the Bits Per Spike (BPS) metric BEAST3D features outperform 3D keypoints for predicting neural activity in both datasets, and per-neuron comparisons confirm that this gain is consistent across the population rather than driven by a small subset of well-predicted neurons
Ablation Study Findings
Ablations show that removing the DINOv3 encoder substantially degrades pose estimation performance (Fig. 9) Removing the frustum constraint has minimal effect on most datasets but is critical for Chickadee, where the bird is cropped from large images using a per-frame bounding box
Inference Cost Comparison
Figure 7 compares inference compute cost vs.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Multi-view video analysis from sparse views can be performed by BEAST3D, which
reconstruct[s] 3D structure with as few as four views by condition[ing] directly on known camera parameters.
This allows for 3D structure reconstruction even when dense overlapping viewpoints are unavailable in lab settings. -
Pose estimation pipelines can be improved using BEAST3D features, which
produce rich, viewpoint-invariant features that transfer effectively to three downstream tasks: novel view synthesis, multi-view pose estimation, and neural encoding.
This enables the creation ofsparse keypoint trajectories widely used in behavioral analysis.
-
Neural encoding for behavior can be enhanced by using BEAST3D Gaussian splat representations, which are
denser and more expressive than a sparse set of keypoints while remaining spatially grounded, making it more interpretable than opaque CLS tokens.
This allows researchers to askwhich body parts or appearance details drive a given neuron’s activity.
-
Novel view synthesis can be benchmarked by comparing BEAST3D's performance against baselines like E-RayZer and Pose Splatter, demonstrating that
BEAST3D outperforms both baselines across all metrics and datasets
in the within-subject evaluation setting. -
Inference compute cost can be managed by leveraging the architecture, as BEAST3D and E-RayZer have
comparable compute and memory profiles,
with DINOv3 adding aroughly constant per-view memory and latency overhead.
Abstract
Multi-view video recordings are increasingly used to capture the 3D movements of animals in experimental settings, yet extracting rich 3D representations from these recordings remains challenging. Supervised pose estimation requires extensive manual annotation, while general-purpose 3D reconstruction models trained on generic scene datasets fail on the specialized imagery and sparse-view setting of laboratory experiments. We address these limitations with BEAST3D, a self-supervised pretraining framework that learns 3D visual representations from unlabeled, calibrated multi-view video. BEAST3D uses a vision transformer to predict 3D Gaussian splats that reconstruct held-out views through differentiable rendering, while simultaneously segmenting the animal from the background. BEAST3D reconstructs 3D structure with as few as four views by conditioning directly on known camera parameters--unlike general-purpose models, which must estimate camera geometry from dense overlapping viewpoints that are seldom available in lab settings. Through comprehensive evaluation across four species, we demonstrate that BEAST3D produces rich, viewpoint-invariant features that transfer effectively to three downstream tasks: novel view synthesis, which validates the quality of the learned 3D representations; multi-view pose estimation, which provides the sparse keypoint trajectories widely used in behavioral analysis; and neural encoding, which relates 3D behavioral features to simultaneously recorded neural activity. BEAST3D thus establishes a versatile framework for behavioral analysis that leverages 3D structure in modern multi-view laboratory recordings.
Sources
- E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training
- DINOv3
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- DINOv2: Learning Robust Visual Features without Supervision
- Stereo Magnification: Learning View Synthesis using Multiplane Images
- Domain-Adaptive Pretraining Improves Primate Behavior Recognition
- SAM 3: Segment Anything with Concepts
- Decoupled Weight Decay Regularization
- GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splatting
- Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity
Related papers
- Self-Supervised Learning of Local Predictive Directions as a Neural Computation
- Synchronized Hearts, Desynchronized Brains: A Nonlinear Dynamics Perspective for Function and Failure
- Message interaction dynamics covary with brain volume across mammalian connectomes
- Planning as Dynamics Relaxation: Hippocampal Recurrent Network Realizes Optimal Goal-Directed Navigation
- The Platonic brain bridge hypothesis: brain networks as architectural priors for multimodal large language models
- BrainPFN: Amortized Brain-Behaviour Prediction via Prior-Data Fitted Networks