BEAST3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting
summary
The gist
Multi-view video recordings are increasingly used to capture the 3D movements of animals in experimental settings…
In short
BEAST3D is a self-supervised framework that learns 3D visual representations from unlabeled, calibrated multi-view animal videos to reconstruct 3D structure with just four views. It uses a transformer and image tokens to create rich, viewpoint-invariant features that successfully improve novel view synthesis, pose estimation, and neural encoding across four species.
Key concepts
- Self-supervised Pretraining
- BEAST3D learns 3D visual representations without needing manual 3D labels. It uses the video data itself to train a model to predict 3D Gaussian splats from images and camera parameters. This allows the model to learn meaningful spatial relationships directly from the raw, unlabeled video footage.
- Gaussian Splatting (GSplat)
- This is a technique used by BEAST3D to represent 3D objects as a collection of semi-transparent, view-dependent 3D shapes called Gaussian splats. These splats are rendered using differentiable rendering, which allows the model to train the representation by comparing its output against real images.
- Viewpoint-Invariant Features
- The features learned by BEAST3D are robust to changes in viewing angle. This means the representation of an animal's 3D structure remains consistent whether viewed from a close angle or a distant one, making these features highly effective for tasks like pose estimation and neural encoding.
Terminology used across episodes
This episode discusses
- BEAST3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting · Paper Radio
- E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training
- DINOv3
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- DINOv2: Learning Robust Visual Features without Supervision
- Stereo Magnification: Learning View Synthesis using Multiplane Images
- Domain-Adaptive Pretraining Improves Primate Behavior Recognition
- SAM 3: Segment Anything with Concepts
- Decoupled Weight Decay Regularization
- GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splatting
- Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity
The paper
BEAST3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting · Read on arXiv
Columbia University
Multi-view video recordings are increasingly used to capture the 3D movements of animals in experimental settings, yet extracting rich 3D representations from these recordings remains challenging. Supervised pose estimation requires extensive manual annotation, while general-purpose 3D reconstruction models trained on generic scene datasets fail on the specialized imagery and sparse-view setting of laboratory experiments. We address these limitations with BEAST3D, a self-supervised pretraining framework that learns 3D visual representations from unlabeled, calibrated multi-view video. BEAST3D uses a vision transformer to predict 3D Gaussian splats that reconstruct held-out views through differentiable rendering, while simultaneously segmenting the animal from the background. BEAST3D reconstructs 3D structure with as few as four views by conditioning directly on known camera parameters--unlike general-purpose models, which must estimate camera geometry from dense overlapping viewpoints that are seldom available in lab settings. Through comprehensive evaluation across four species, we demonstrate that BEAST3D produces rich, viewpoint-invariant features that transfer effectively to three downstream tasks: novel view synthesis, which validates the quality of the learned 3D representations; multi-view pose estimation, which provides the sparse keypoint trajectories widely used in behavioral analysis; and neural encoding, which relates 3D behavioral features to simultaneously recorded neural activity. BEAST3D thus establishes a versatile framework for behavioral analysis that leverages 3D structure in modern multi-view laboratory recordings.
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: I'm Ines.
Marcus: And I'm Marcus, here with Yuki.
Ines: We're looking at the paper: "BEAST3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting".
Marcus: Here's the gist of the paper. Multi-view video recordings are increasingly used to capture the 3D movements of animals in experimental settings, yet extracting rich 3D representations from these recordings remains challenging The authors address these limitations with BEAST3D, a self-supervised pretraining framework that learns 3D visual representations from unlabeled, calibrated multi-view video BEAST3D reconstructs 3D structure with as few as four views by conditioning directly on known camera parameters—unlike general-purpose models, which must estimate camera geometry from dense overlapping viewpoints that are seldom available in lab settings Through comprehensive evaluation across four species, they demonstrate that BEAST3D produces rich, viewpoint-invariant features that transfer effectively to three downstream tasks: novel view synthesis, multi-view pose estimation, and neural encoding The Core Methodology BEAST3D is a self-supervised pretraining framework that learns 3D visual representations from calibrated multi-view animal behavior videos Given only unlabeled, synchronized multi-view images and corresponding camera parameters, BEAST3D predicts a set of 3D Gaussian splats via a transformer trained to reconstruct held-out views through differentiable rendering The model simultaneously segments the animal from the background by distilling masks from a video segmentation model, producing clean foreground representations This framework builds on the 3DGS framework of E-RayZer
35: , but replaces learned components with frozen foundation models and ground truth calibration data The architecture involves image tokenization using a frozen DINOv3 ViT-B/16 as the image encoder Camera tokenization uses Plücker coordinates to provide geometric grounding, which are then fused with image tokens via a two-layer MLP The fused tokens are processed by a geometry transformer that alternates between frame attention and global attention The final output decodes these tokens into per-patch 3D Gaussian parameters via a linear head, followed by differentiable rendering using GSplat
44: Training Signal Construction The training loss L is computed only on the held-out target views T and combines three terms: (i) a photometric loss Ll2 that penalizes pixel-level differences between the ground truth and rendered images, (ii) a perceptual loss Lperc that penalizes abstract feature-level differences between ground truth and rendered images, and (iii) a mask loss Lmask that penalizes differences between the rendered alpha channel and the SAM3 segmentation masks The final loss is a weighted combination of each: L = λl2 Ll2 + λperc Lperc + λmask Lmask They train BEAST3D using the AdamW optimizer with a base learning rate of 5 × 10−5, weight decay of 0.05, and a cosine learning rate schedule with 15% warmup for 800 epochs During each forward pass, they randomly sample Vref = V − 1 reference views and hold out Vtgt = 1 target view Performance Across Downstream Tasks BEAST3D is evaluated on three downstream tasks: novel view synthesis, multi-view pose estimation, and neural encoding In novel view synthesis, BEAST3D produces sharp reconstructions of held-out viewpoints that recover the subject’s silhouette and fine appearance details across all four datasets For pose estimation, BEAST3D achieves the best performance across nearly all datasets, with the exception of Rat7M where VGGT performs best In neural encoding, BEAST3D features outperform 3D keypoints for predicting neural activity in both datasets (Cheese3D and Chickadee), demonstrating that dense 3D representations capture behavioral information that sparse keypoints discard Comparative Analysis with Baselines BEAST3D is compared against several baselines, including E-RayZer, Pose Splatter, and VGGT In novel view synthesis on the InD Test set, BEAST3D outperforms both Pose Splatter and E-RayZer across all datasets For pose estimation, BEAST3D consistently outperforms BEAST across all datasets, demonstrating the benefit of multi-view pretraining However, a striking finding is that multi-view models more broadly (VGGT and E-RayZer) do not consistently outperform single-view DINOv3 and BEAST baselines Inference Efficiency and Scalability The inference compute cost shows that BEAST3D and E-RayZer have comparable compute and memory profiles, with the DINOv3 frontend in BEAST3D adding a roughly constant per-view memory and latency overhead The model's ability to reconstruct 3D structure with as few as four views by conditioning directly on known camera parameters is a key advantage over general-purpose models A current limitation of BEAST3D is its compute requirements: pretraining on a single dataset takes roughly 32 hours on 8 A100 GPUs, placing it out of reach for many labs Broader Implications The BEAST3D framework enables more efficient extraction of meaningful information from multiview video data, potentially accelerating behavioral neuroscience research with several beneficial outcomes By reducing the need for extensive human labeling while improving accuracy, BEAST3D can democratize advanced video analysis capabilities for laboratories with limited resources However, concerns remain regarding surveillance capabilities, bias and fairness across demographic groups, and resource inequality due to high pretraining compute requirements Datasets Used The evaluation spans four datasets spanning species, environments and camera configurations (Cheese3D, Rat7M, Chickadee, and Human3.6M) These datasets include both unconstrained naturalistic behavior and controlled lab settings The dataset construction pipeline involves five stages: cut, downsample, segment, assemble, and resize to produce the calibrated (frame, mask, camera) tuples consumed by BEAST3D Pose Estimation Details For pose estimation training/test sets on Cheese3D and Human3.6M datasets, they constructed a pseudo-labeled dataset from an initial set of 665 instances where each keypoint is labeled in a subset of views depending on anatomical visibility They apply k-means clustering in the space of flattened 3D keypoints for each session to select representative frames for training and testing The final pose estimation training/test sets consist of 450/150 instances for Cheese3D, 455/177 instances for Rat7M, and 1125/425 instances for Human3.6M Neural Encoding Details For neural encoding on Cheese3D and Chickadee datasets, they segment the spike trains into non-overlapping 2.0 s windows and partition them into train, val, or test splits The encoding model is intentionally lightweight and input-agnostic so that it does not itself learn 3D structure that could compensate for a weak input They use the foreground-filtered BEAST3D point cloud as a baseline, contrasting it with multi-view-triangulated 3D keypoints produced by supervised pose estimation models The resulting per-frame point clouds, together with their integer frame indices, are concatenated across the session and serve as the visual input to the encoding model Novel View Synthesis Metrics Novel view synthesis performance is benchmarked using Peak signal-to-noise ratio (PSNR), Structural similarity index measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) BEAST3D results show substantial improvements in metrics across all datasets in the within-subject evaluation setting compared to baselines Table 5 and Table 6 provide quantitative comparisons of these metrics for both InD Test and OOD Test sets Pose Estimation Evaluation Protocol Pose estimation is evaluated using the limited-data evaluation protocol of the original BEAST paper, where they train three models on random subsets of only 100 labeled instances and evaluate on held-out test subjects Results are reported using difficulty-stratified error curves, which reveal how each method handles keypoints of varying difficulty rather than collapsing performance into a single number Neural Encoding Evaluation Protocol For neural encoding, they split neural activity from each session into non-overlapping 2 s chunks and randomly assign each chunk to a train, val or test split They tune model hyperparameters on the train/val splits and report results on the test split using the Bits Per Spike (BPS) metric BEAST3D features outperform 3D keypoints for predicting neural activity in both datasets, and per-neuron comparisons confirm that this gain is consistent across the population rather than driven by a small subset of well-predicted neurons Ablation Study Findings Ablations show that removing the DINOv3 encoder substantially degrades pose estimation performance (Fig.
Ines: 9) Removing the frustum constraint has minimal effect on most datasets but is critical for Chickadee, where the bird is cropped from large images using a per-frame bounding box Inference Cost Comparison Figure 7 compares inference compute cost vs.
Title and authors: Ines: So we're talking about the paper "BEASTthree dee: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting." It sounds like this framework tackles a big problem in getting reliable three-dimensional information out of animal videos.
Marcus: Yeah, what I find interesting is that it directly addresses the issue where you need many overlapping views for three dee reconstruction, but those views aren't always available in a lab setting <ref:2606.02937#pg1>.
Ines: Exactly. It suggests that we can reconstruct three dee structure with just four views by using known camera parameters, which is a big deal because it bypasses the need to estimate complex camera geometry from dense overlaps <ref:2606.02937#pg1>.
Marcus: Right. And the authors are Yanchen Wang, Lenny Aharon, Wangshu Zhu, Kyle Daruwalla, Linghua Zhang, Jiaru Zou and Selmaan Chettih. They bring a strong background in computer vision and deep learning.
Yuki: From my side, I'm curious how this work connects to the actual biological data they're using; whether these views are capturing enough information about the animal's movement or behavior for population studies.
The paper's summary: Ines: So, what BEASTthree dee actually does is it’s a self-supervised pretraining framework. It learns to make three dee visual representations from videos that don't have labels, but they are calibrated multi-view videos <ref:2606.02937#pg1>.
Marcus: They use a transformer trained to reconstruct unseen views by doing differentiable rendering. That means the model learns how to generate what the scene looks like from different angles based on those few views.
Ines: And they also do something extra—they simultaneously segment the animal from its background by taking masks from a video segmentation model, which gives them clean representations of just the subject.
Yuki: So, it’s not just about reconstructing a shape; it’s about isolating the specific biological entity we care about from all that background noise in the footage.
The paper's improvements: Ines: The authors point out that they replaced some of the learned components with frozen foundation models and ground truth calibration data. They also used a frozen DINOv3 vision transformer as the image encoder, which helps them get these features.
Marcus: That’s smart because it means they aren't training everything from scratch; they are leveraging existing powerful tools to learn what's important about the visual data.
Ines: And they show that these learned features work really well across four different species: mouse, rat, chickadee and human. The key improvement is that these features transfer effectively to three downstream tasks: novel view synthesis, multi-view pose estimation, and neural encoding.
Yuki: I wonder about the neural encoding part. If their three dee representations are good enough, does it mean we can actually start linking specific brain activity patterns back to observable three dee body parts or actions <ref:2606.02937#pg1>?
Conclusion: Ines: To wrap up, BEASTthree dee shows that you can get rich, viewpoint-invariant features from multi-view data even with just a few views by using known camera parameters.
Marcus: And these features work across novel view synthesis, multi-view pose estimation, and neural encoding tasks. It seems to establish BEASTthree dee as a versatile framework for extracting three dee features from these complex videos <ref:2606.02937#pg1>.
Yuki: From a population genetics perspective, I think this is promising because it could eventually allow us to analyze subtle movement patterns across different species using standardized, high-quality three dee data that we can actually compare statistically <ref:2606.02937#pg1>.
Ines: That’s the big picture. So, we see a framework that makes three dee analysis of animal behavior much more accessible when you have video recordings from multiple cameras, even sparse ones <ref:2606.02937#pg1>.
Marcus: It does mean that if you're working in a lab with limited resources, this could significantly speed up the process of getting reliable three dee data for behavioral studies <ref:2606.02937#pg1>.
Yuki: I think the main thing is how well these features actually generalize when we move from one species to another, which is where the comparisons across mouse and human really matter.
Ines: It’s definitely a versatile tool that moves us closer to having richer, more detailed behavioral data without needing massive amounts of manual labeling for every single frame.
Marcus: We’ll keep an eye on how the compute requirements scale up, because they did mention that pretraining on a single dataset takes about thirty-two hours on eight A100 GPUs, which is still quite demanding.
Yuki: It’s interesting that the paper flags that their neural encoding features actually outperform sparse keypoints for predicting neural activity in both the mouse and chickadee datasets. That suggests that dense three dee information captures more behavioral detail than just a few points do <ref:2606.02937#pg1>.
Ines: Exactly, it implies we should look at how we can use these dense representations to decode what specific body parts or appearance details are driving those neural activities. That’s where the real biological insight lies in this work.
More episodes
- 2610.33881-Sensorimotor behavior from an untrained brain-and-nerve-cord connectome
- 2610.40344-Morphological Features of Murine Pulmonary Arterial Networks for Diagnosing Vascular Disease
- Daily Summary for 2026-11-19
- 2609.10193-Are You Learning Biological Signal or Shortcuts? Auditing and Mitigating Bias in Protein-Protein Interaction Datasets
- 2610.38236-Generative discovery of chemical reaction networks
- 2507.11176-An Interpretable AI framework Quantifying Traditional Chinese Medicine Principles Towards Enhancing and Integrating with Modern Biomedicine
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2504.03590-Towards a Unified Framework for Determining Conformational Ensembles of Disordered Proteins
- 2603.15217-A multiscale discrete-to-continuum framework for structured population models