Autotaxis of a Run-and-Tumble Agent via Reinforcement Learning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Quantitative Biology. The day in quantitative biology: models, measurements and the data behind them.
Ines: We're looking at the paper: "Autotaxis of a Run-and-Tumble Agent via Reinforcement Learning".
Marcus: Here's the gist of the paper. Autotaxis of a Run-and-Tumble Agent via Reinforcement Learning shows that reinforcement learning enhances chemotactic performance in a complex and time-varying chemical landscape The RL agent performs better in the long time limit and improves its performance faster with time when compared to a non-learning agent.
Ines: Modeling the Agent and Environment The study models an autochemotactic agent, specifically an E.
Marcus: coli cell, using a run-and-tumble motion framework.
Ines: The agent's state is defined by whether it is in a 'high' or 'low' state based on the local attractant concentration relative to the average level.
Title and authors: Ines: So we're talking about "Autotaxis of a Run-and-Tumble Agent via Reinforcement Learning" and who the authors are, which is Ramesh Pramanik, Shradha Mishra, and Sakuntala Chatterjee. They set up this study to see if reinforcement learning can improve navigation in an E. coli cell using run-and-tumble motion.
Marcus: Yeah, the paper is about studying a single autochemotactic agent that secretes and senses chemical cues within a chemical environment that changes dynamically through secretion, diffusion, and degradation of those attractant molecules.
Yuki: From my perspective as a population geneticist, I see them using this simple model to test if the underlying mechanisms they observe in E. coli are robust enough to support real-world population dynamics.
Ines: They use a run-and-tumble motion framework where the agent's state is determined by whether it is in a 'high' or 'low' state based on the local attractant concentration relative to the average level.
Marcus: The action space they define is simple: either persist, which means running in its current direction, or reverse, which is tumbling.
Yuki: What I like about this modeling choice is that it simplifies the complex biology down to a manageable set of rules while still capturing the essence of chemotaxis.
Ines: They then introduce reinforcement learning by modeling the decision-making using a Q-learning algorithm with an epsilon greedy strategy for updating its Q matrix.
Marcus: That means the agent learns by evaluating a cost function c S A that links its motility directly to how it interacts with the environment at each time step.
Yuki: And this cost function for tumbling is defined by a memory kernel R of time, which describes past chemical concentrations along the trajectory, meaning tumbling isn't random but history dependent.
Ines: The agent’s policy balances directed movement toward gradients with enough exploration to avoid getting stuck in local concentration maxima.
Marcus: And they detail how the Q matrix is two by two dimensional in this case because of those two actions, and at each time step it evaluates the cost for the chosen action and updates that corresponding element.
Yuki: It feels like they are building a bridge between simple biophysical models of movement and complex decision-making processes needed for survival in a changing niche.
The paper's summary: Ines: To summarize this study on "Autotaxis of a Run-and-Tumble Agent via Reinforcement Learning," the paper shows that reinforcement learning actually enhances the chemotactic performance of this agent in a complex and time-varying chemical landscape.
Marcus: Specifically, they show that when using reinforcement learning, the probability of finding the agent in the favorable region near an attractant peak increases faster with time and reaches a higher value in the long time limit compared to when it performs usual run-tumble chemotaxis without learning.
Yuki: This is significant because it means that even if a cell starts randomly, using this learned behavior helps it find those high-concentration areas more reliably over extended periods.
Ines: They also characterize this performance by varying different parameters like the rates of secretion, degradation, and diffusion to see how they affect localization.
Marcus: The results show that localization increases with the secretion rate gamma and decreases with the degradation rate beta, which tells us about the physical processes that are driving these changes.
Yuki: And there's this non-monotonic dependence on the learning and exploration rates, suggesting an optimal RL strategy exists for balancing how much it learns versus how much it explores.
Ines: They also found an optimum diffusivity D for localization in a homogeneous environment, which is a specific physical parameter that dictates where localization is best achieved when everything is uniform.
Marcus: Overall, the paper demonstrates that the RL agent shows significantly better performance than the run-tumble model without RL, achieving higher occupancy near attractant peaks and reaching this value faster.
Yuki: For someone listening who just wants to know what this means: it means adding a learning mechanism to an E. coli cell's navigation system gives it a real advantage when the chemical landscape is dynamic, not just static.
The paper's improvements: Ines: The paper suggests several ways to improve this work, starting with using reinforcement learning agents to navigate complex, time-varying chemical environments by minimizing a cost or maximizing a reward as described in other literature.
Marcus: They propose an improvement of the system by moving away from waiting for a converged policy and instead using an online learning strategy where the agent continuously updates its Q values based on its current interaction with the environment and selects its action using those presently learned Q values.
Yuki: That continuous updating idea is powerful because it means the agent can react instantly to new chemical cues as they are being generated by itself, rather than waiting for a fixed policy to be fully trained.
Ines: They also suggest optimizing navigation in dynamic chemical landscapes by balancing directed movement toward chemical gradients with sufficient exploration to avoid getting trapped in local concentration maxima, achieving an optimal combination of the learning rate alpha and the exploration parameter epsilon.
Marcus: This points toward finding that optimal combination of the learning rate alpha and the exploration parameter epsilon for a given environment, which is crucial for efficient search.
Yuki: And they also explore how to make tumbling more costly when climbing an attractant gradient by defining the cost c of tumble as Z t minus infinity dt prime R of t minus t prime L of tp.
Ines: That specific cost function for tumbling means that tumbling becomes more expensive when it's on a favorable gradient, which is a direct way to bias the agent toward running up the gradient.
Marcus: And they suggest adapting motility modulation based on temporal integration of chemical cues, specifically by comparing the attractant concentration experienced in the recent past and in the distant past to distinguish between favorable and unfavorable directions.
Yuki: That comparison between recent and distant past is a clever way for an agent to gain a better sense of direction than just looking at what's happening right now.
Conclusion: Ines: So wrapping up the paper "Autotaxis of a Run-and-Tumble Agent via Reinforcement Learning," the main implication is that reinforcement learning provides a significant performance boost for chemotaxis in dynamic chemical settings.
Marcus: They show that this is particularly true when comparing the RL agent to the standard run-tumble model, achieving higher occupancy near attractant peaks and reaching it faster over time.
Yuki: From a population genetics viewpoint, this suggests that adaptive navigation strategies are essential for organisms living in environments where resources are constantly shifting.
Ines: The paper concludes that the optimal search strategy is finding a balance between learning and exploration to efficiently locate the global peak, which is what they found by varying parameters like secretion rate gamma and degradation rate beta.
Marcus: They also confirm that employing reinforcement learning is generally beneficial as it improves long-time performance and search speed in these scenarios.
Yuki: I just want to add that this work confirms that adaptive learning mechanisms can be a powerful tool for spatial navigation in biological systems facing temporal chemical changes.
Ines: That's what we found, looking at the paper "Autotaxis of a Run-and-Tumble Agent via Reinforcement Learning."
Ramesh Pramanik, Shradha Mishra, Sakuntala Chatterjee
Department of Physics of Complex Systems, S.N. Bose National Centre for Basic Sciences, Salt Lake Sector 3, Kolkata 700106, India · Department of Physics, Indian Institute of Technology (BHU) Varanasi
q-bio.CB, cond-mat.stat-mech
Submitted: 2026-11-05
Updated: 2026-11-05
Journal ref: Physical Review E 114, 054402 (2026)
DOI: 10.1103/4xnp-dsn4
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 74/100
The gist: Autotaxis of a Run-and-Tumble Agent via Reinforcement Learning shows that reinforcement learning enhances chemotactic performance in a complex and time-varying chemical landscape The RL agent…
Key concepts
- Run-and-Tumble Motion
- This is the agent's basic movement style, mimicking how bacteria move. The agent runs in one direction for a period before randomly tumbling (changing direction). This simple mechanism forms the foundation of how the cell explores its chemical environment.
- Q-learning Algorithm
- This is the reinforcement learning method used to make decisions. The agent uses Q-learning to update a matrix that estimates the value of taking different actions (like running or tumbling) in specific states based on past experiences, aiming to maximize its reward.
- Chemotactic Performance Metrics
- These are ways to measure how well the agent finds chemical attractants. Key metrics include the width of its final position distribution ($\sigma$), a contrast measure ($\eta$) for periodic environments, and the mean first passage time (MFPT), which tracks how quickly it reaches a target peak.
- Cost Function c[S, A]
- This function defines the 'cost' associated with an action. For tumbling, this cost incorporates a memory kernel that looks at past chemical concentrations. This allows the agent to learn that tumbling is costly when it is currently moving toward a favorable chemical gradient.
Terminology
Summary
Autotaxis of a Run-and-Tumble Agent via Reinforcement Learning shows that reinforcement learning enhances chemotactic performance in a complex and time-varying chemical landscape The RL agent performs better in the long time limit and improves its performance faster with time when compared to a non-learning agent. p. 0 We ask the question whether RL improves the chemotactic performance in a complex and time-varying
chemical landscape. We explicitly compare the long time performance of our RL agent and a non-learning agent to
show that RL indeed yields better performance.Where the paper says it
Modeling the Agent and Environment
The study models an autochemotactic agent, specifically an E. coli cell, using a run-and-tumble motion framework. p. 0 Autotaxis of a Run-and-Tumble Agent via Reinforcement Learning
Ramesh Pramanik1
, Shradha Mishra2 and Sakuntala Chatterjee1
(1) Department of Physics of Complex Systems, S. Bose National
Centre for Basic Sciences, Salt Lake Sector 3, Kolkata 700106, India
(2) Department of Physics, Indian Institute of Technology (BHU) Varanasi 221005, India
We study the run-tumble motion of a single autochemotactic agent that secretes and senses
chemical cues using a reinforcement learning (RL) framework. Bose National
Centre for Basic Sciences, Salt Lake Sector 3, Kolkata 700106, India
(2) Department of Physics, Indian Institute of Technology (BHU) Varanasi 221005, India
We study the run-tumble motion of a single autochemotactic agent that secretes and senses
chemical cues using a reinforcement learning (RL) framework. In a chemical environment that
changes dynamically via secretion, diffusion, and degradation of attractant chemicals, we study the
effectiveness of RL for a successful navigation strategy.Where the paper says it
Where the paper says it
Reinforcement Learning Formulation
The decision-making process is modeled using a Q-learning algorithm with an epsilon-greedy strategy. The agent updates its Q-matrix based on the Q-learning rule, where the cost function c[S, A] links motility to environmental interaction. p. 2 In our case
the Q-matrix is therefore 2 × 2 dimensional. At each time step, the agent evaluates the cost c[S, A] associated with
the chosen action and updates the corresponding Q-matrix element according to the Q-learning rule [29, 73],
Qt+1[S, A] ← Qt[S, A] + α (c[S, A] − Qt[S, A]), (1)
where α is the learning rate. By comparing the attractant concentration along
its trajectory in the recent past and in the distant past, the cell reads off the gradient. This comparison is done with
the help of a memory kernel, which has been measured experimentally [63]. These studies show that there
exists an optimal combination of the learning rate α and the exploration parameter ǫ that yields the most effective
chemotactic response. At this optimum, the agent balances directed movement toward chemical gradients with
sufficient exploration to avoid getting trapped in local concentration maxima of attractant chemical [45].Where the paper says it
Where the paper says it
Where the paper says it
Performance Analysis and Optimal Parameters
The chemotactic performance is characterized by various metrics, including the steady-state position distribution width σ, the contrast measure η for sinusoidal environments, and the mean first passage time (MFPT). The results indicate that localization increases with secretion rate γ and decreases with degradation rate β. p. 1 We study
how the long-time performance of the agent depends on the different parameters and whether there is an optimal set
of parameters that yields the most efficient chemotaxis. In all cases we find the performance increases with secretion
rate and decreases with degradation rate. We also characterize the chemotactic performance of the agent
by varying different parameters like rates of secretion, degradation, and diffusion. Importantly, we
observe a non-monotonic dependence of the chemotactic performance on the learning and exploration
rates, indicating the existence of an optimal RL strategy for the most efficient navigation.Where the paper says it
Where the paper says it
Comparative Results and Conclusion
The RL agent demonstrates significantly better performance than the run-tumble model without RL, achieving higher occupancy near attractant peaks and reaching this value faster. The study investigates three initial profiles: spatially uniform, multiple peaks of equal heights, and multiple peaks of unequal heights. p. 4 Thus the behavior of the agent as well as the shape of the attractant profile depends
on the choice of the initial profile [L](x, 0). We consider three types of initial profiles: a spatially uniform profile, a
profile with multiple peaks of equal height, and a profile with multiple peaks of unequal heights. This is particularly relevant when the chemical landscape features multiple
peaks of unequal heights, and the agent needs to escape the local peaks and find the global peak. We find the most
efficient search strategy involves an optimum balance between learning and exploration by the RL agent. We ask the question whether RL improves the chemotactic performance in a complex and time-varying
chemical landscape. We explicitly compare the long time performance of our RL agent and a non-learning agent to
show that RL indeed yields better performance.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved AI systems can utilize a reinforcement learning agent to navigate complex, time-varying chemical environments by minimizing a cost or maximizing a reward, as described in
an RL algorithm considers an agent that evolves through experience by minimizing a cost or maximizing a reward [25–33].
-
The improved system will enhance chemotactic performance by employing an online learning strategy where the agent
continuously updates its Q-values based on its current interaction with the environment and selects its action using the presently learned Q-values
rather than waiting for a converged policy. -
The system can optimize navigation in dynamic chemical landscapes by balancing directed movement toward chemical gradients with sufficient exploration to avoid getting trapped in local concentration maxima, achieving an
optimal combination of the learning rate α and the exploration parameter ǫ.
-
The agent's behavior is governed by a cost function where "c[S, tumble] = Z t−∞ dt′ R(t − t′) L" for tumbling, making tumbling more costly when climbing an attractant gradient.
-
The system can adapt its motility modulation based on temporal integration of chemical cues, specifically by comparing
the attractant concentration experienced in the recent past and in the distant past
to distinguish between favorable and unfavorable directions. -
The AI can be tuned to find an optimal search strategy by varying parameters like deposition rate γ, degradation rate β, chemical diffusivity D, learning rate α, and exploration parameter ǫ.
-
The improved system can quantify chemotactic performance using measures such as the standard deviation σ of the steady-state position distribution P(x) or a contrast measure η to assess how effectively the agent localizes near attractant peaks in different environments.
Abstract
We study the run-tumble motion of a single autochemotactic agent that secretes and senses chemical cues using a reinforcement learning (RL) framework. In a chemical environment that changes dynamically via secretion, diffusion, and degradation of attractant chemicals, we study the effectiveness of RL for a successful navigation strategy. We show that RL enhances chemotactic performance. Starting from a random initial position, the probability of finding the agent in the favorable region (near the attractant peak) increases faster with time and reaches a higher value in the long time limit when RL is used, compared to the case when the agent performs usual run-tumble chemotaxis without using RL. We also characterize the chemotactic performance of the agent by varying different parameters like rates of secretion, degradation, and diffusion. Importantly, we observe a non-monotonic dependence of the chemotactic performance on the learning and exploration rates, indicating the existence of an optimal RL strategy for the most efficient navigation. We explain these observations as a result of the interplay between the pre-existing attractant environment and the agent-generated attractant profile.
Related papers
- Optimality theory of stigmergic collective information processing by chemotactic cells
- A Phase-field Model for Apoptotic Cell Death
- Single-cell directional sensing at ultra-low chemoattractant concentrations from extreme first-passage events
- Randomness-aware multiscale models of glioma invasion and treatment