Modeling Fly Social Behavior with Agent-Centric Pose Forecasting

summary

The gist

The gist: This work introduces an agent-centric framework for training autoregressive models of animal behavior by constraining deep networks to receive sensory inputs and produce motor outputs

In short

This work introduces an agent-centric framework to train models of animal behavior by forcing deep networks to mimic a specific animal's sensory inputs and motor outputs. It uses a library called AnimalPoseForecasting to manage complex data representations, allowing researchers to study how animals generate behavior rather than just fitting movement trajectories. The results show this approach captures fine-grained social behaviors accurately.

Key concepts

Agent-Centric Framework: This core idea treats each animal as an independent agent that receives sensory information and produces movements in the body's frame. The network is constrained so that it must solve the same problem as the animal itself, aiming to understand how behavior is generated rather than just predicting movement paths.
AnimalPoseForecasting Library: This general-purpose library manages complex data representations needed for the framework. It handles invertible operations between different views of pose and sensing—such as world-frame, egocentric, and machine learning friendly formats—using composable sequences to connect these representations.
Sensory Input/Output Representations: The input is carefully designed to limit the agent to what the animal can actually perceive. For flies, this includes depth maps for visual information about other flies and summarized tactile data like ceiling height at keypoints. Output choices explore different prediction forms and temporal frames.
Autoregressive Rollout: Generation involves iteratively predicting the next step in a dynamic environment. Because the next step depends on all agents' previous actions, this requires inverting predicted movements back into world coordinates at each step to maintain consistency across the simulation.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 18 episodes
Federated Learning — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Flow Matching — explained in 15 episodes

This episode discusses

The paper

Modeling Fly Social Behavior with Agent-Centric Pose Forecasting · Read on arXiv

HHMI Janelia Research Campus

Understanding animal behavior at an algorithmic level -- what animals attend to, how they form internal representations, and how this maps to action -- remains a central challenge in neuroethology. We explore how well large generative models capture properties of fly social behavior. We combine transformers with agent-centric representations. Models input egocentric sensory observations and output egocentric movements, mirroring biological constraints. Social behavior emerges from agents independently sensing and responding to one another. We demonstrate that this simple model captures many properties of fly behavior, outperforming the world-centric representation and achieving SOTA on MABe 2022-fly. We systematically evaluate properties of the representation important for capturing the behavior distribution through a series of lesioning experiments. These comparisons were enabled by a new, general-purpose library for animal-centric pose forecasting.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Modeling Fly Social Behavior with Agent-Centric Pose Forecasting".

Tom: The gist:

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So, to recap, this paper introduces the "Modeling Fly Social Behavior with Agent-Centric Pose Forecasting" framework. The core thesis is building autoregressive models of animal behavior where every animal is an agent that inputs sensory data and outputs movements in its own body frame.

Jane: They claim this approach allows the network and the animal to solve the same problem, shifting the focus from just fitting trajectories to understanding how an animal generates its behavior.

Lu: The framework requires managing many parallel representations of data alongside ML-specific transformations like discretization, which they support with a library called AnimalPoseForecasting.

Meng: This library handles the necessary transformations between different pose representations—world-frame, egocentric, and ML-friendly ones—through composable and invertible operations.

Lalam: The sensory input is hand-crafted to respect what the animal perceives, like using depth maps for visual information about other flies or summarizing tactile info by looking at ceiling height at keypoints.

Tom: And they also explore output representation choices across three axes: the form of the prediction, whether it's continuous, discrete, or hybrid; the temporal frame of reference, absolute pose versus change in pose; and how you parameterize the pose itself.

Jane: When dealing with dynamic environments where agents interact, generation happens by applying next-step prediction iteratively. This requires inverting every agent’s predicted movement back into a change in world-coordinate pose at each rollout step.

Lu: They show that this library design ensures that the same operations used to map raw pose into the model's egocentric input space during training are inverted at rollout time to map predictions back into world coordinates.

Meng: The evaluation looked at qualitative characterization, quantitatively comparing distributions of behavior features between real and simulated data, and training discriminators to tell real from simulated data.

Lalam: Their key finding is that agent-centric representations enabled by the library are most important for accurate behavior modeling in this work on fly social behavior.

Conclusion: Tom: So, looking at the full paper, "Modeling Fly Social Behavior with Agent-Centric Pose Forecasting," the authors are aiming to use these models as tools for scientific discovery, trying to figure out what internal structures animals implement.

Jane: They show that this agent-centric formulation is essential infrastructure for using data-driven predictive models as actual tools for finding new behaviors.

Lu: The work suggests that mechanistic interpretability, by interrogating a trained model, is a promising path toward this goal, letting scientists extract patterns that they haven't previously articulated.

Meng: From an engineering standpoint, it’s interesting because the framework's adaptability was demonstrated by applying it successfully to RatInABox, which is a synthetic neuro-behavior environment.

Lalam: The paper also highlights that discretization plays a vital role in avoiding the pitfall of forecasting no movement when modeling these systems.

Tom: Ultimately, this work pushes us toward using these models not just as simulators but as ways to understand the computations animals are actually performing, which is a big step for ethology and neuroscience.

More episodes

← Home