From Pixels to Shelf: An Integrated Robotic System for Autonomous Supermarket Stocking with a Mobile Manipulator

The gist Autonomous stocking in retail environments, particularly supermarkets, presents challenges due to dynamic human interactions, constrained spaces, and diverse product geometries; this paper introduces an efficient modular robotic system for autonomous shelf stocking Page 1.

System Integration and Architecture

The proposed system integrates commercially available hardware with a scalable algorithmic architecture Page 3. A major contribution of this work is the system integration of off-the-shelf hardware and ROS2-based perception, planning, and control into a single deployable platform for retail environments Page 1. The architecture leverages Behavior Trees (BTs) for task planning, fine-tuned vision models for object detection, and a two-step Model Predictive Control (MPC) framework for precise shelf navigation using ArUco markers Page 1. This approach addresses critical retail-specific issues such as precise product manipulation, accurate localization, and robust navigation Page 2.

Key Algorithmic Components

The system employs several mature robotic methods to meet the constraints of retail stocking Page 3. These components include:

((

BTs are used to model complex, highlevel decision processes and task planning due to their modular, reactive, and fault-tolerant nature Page 3. They consist of action nodes executing specific actions; control nodes managing the execution flow, including sequence nodes (→) that execute child nodes sequentially until one fails, and fallback (selector) nodes (?) that do so until one succeeds; and conditional check nodes evaluating conditions to adjust behavior Page 3.

((

Model Predictive Control (MPC) is used as part of the control strategy to solve the last-meter navigation problem since it natively enforces motion and safety constraints for coupled robot base-head control while remaining real-time on consumer hardware Page 3. The MPC optimization follows min U c(x,u) (1a) s.t. xk+1 = xk + T · f(xk,uk), (1b) where g(x, u) represents joint state and input constraints Page 3.

((

The Kalman Filter (KF) is used to accurately locate the robot within cluttered and partially observable environments, such as grocery store aisles Page 3. The KF performs a twostep predict-update cycle, which is used to estimate noisy processes, e.g., visual pose estimation, reliably Page 3.

((

Visual Models include YOLO (You Only Look Once) for object detection and SAM2 (Segment Anything Model 2) for segmentation and tracking Page 3. YOLO11n-cls is used for its strong speed/accuracy trade-off, while SAM2 is used due to its combined masking and video tracking capabilities Page 3.

Navigation and Perception Pipeline

The planning architecture operates through a two-step concurrent process designed to dynamically manage the robot’s behavior Page 4. Navigation utilizes the open-source ROS2 Nav2 package, which uses a Model Predictive Path Integral (MPPI) controller Page 4. To improve obstacle perception, a Spatio-Temporal Voxel Layer (STVL) is introduced to the navigation stack, which uses stereo-depth data from the head camera to detect obstacles imperceptible to lidar Page 4. Docking involves an automatic sequence where the robot navigates to a staging location, rotates 180◦, and uses a visual servo controller with ArUco tag detection to compute target coordinates for base velocities Page 4.

Manipulation Strategy

The manipulation pipeline involves distinct stages for picking and placing items Page 5.

((

Picking requires the target product identified via a Global Trade Item Number (GTIN) and uses a pipeline involving segmentation with SAM2, filtering with the grocery YOLO model, and potentially GPT-4o if a dedicated model is unavailable Page 6. From valid segments, point clouds are generated, and known product dimensions guide an optimization that computes accurate bounding boxes Page 6.

((

The arm control uses centroid-based velocity control for the base and vertical lift velocity is determined by the object’s distance from the camera, derived as the 99-th percentile depth of masked points Page 6.

((

Placing involves querying the store’s planogram to locate shelf positions, moving to the area using ArUco tags, and using an IK controller similar to [39] to position the gripper safely above the target location Page 6. The arm lowers the product until effort sensors signal shelf contact, triggering a gentle product release Page 6.

Experimental Validation and Performance

The system was validated in a mock supermarket environment under overnight closed-door operation to match retailer practice Page 6. The reliability evaluation focused on success rates, achieving an object-level success rate exceeding 98% across 724 individual stocking operations Page 6. Performance comparisons against teleoperation and human workers show that the autonomous system was slower than teleoperation, largely due to sequential task execution Page 7. The joint performance index shows that autonomous stocking performs worse than human workers but surpasses teleoperation Page 7. A simple time decomposition suggests that ≈ 80% of the gap can be attributed to hardware constraints and ≈ 20% to software limitations Page 7. Future enhancements to hardware capabilities and algorithmic optimizations promise significant steps toward deployment in retail automation Page 8.

How it works

The paper details the following key aspects of the system's operation:

1. Behavior Trees (BTs) are used for task planning, modeling complex, highlevel decision processes and task planning due to their modular, reactive, and fault-tolerant nature Page 3.

2. Model Predictive Control (MPC) is employed for last-meter navigation since it natively enforces motion and safety constraints for coupled robot base-head control while remaining real-time on consumer hardware Page 3.

3. A Kalman Filter (KF) is implemented to improve the accuracy and robustness of the ArUco-based localization, especially in close-range settings where lidar fails Page 3.

4. Perception relies on YOLO for object detection and SAM2 for segmentation and tracking, with fine-tuned models used to guarantee fast and accurate detection Page 6.

5. Manipulation uses centroid-based velocity control for the base and a depth-derived velocity for the arm, guided by planogram-aware Inverse Kinematics (IK) placement Page 6.

This integrated approach provides a robust pipeline that achieves 98.3% object-level success, including pick and place operations and shelf navigation Page 8. The system demonstrates practical viability in supermarket environments while highlighting hardware constraints as the main focus of improvement toward human-level stocking efficiency Page 8.

The paper is organized as follows: Section II introduces foundational background in BTs, MPC, KF, and vision-based segmentation and classification tools Page 2.

1. Bold Header: Adaptive Task Re-prioritization via Behavior Trees

The existing Behavior Tree (BT) architecture should be enhanced to allow for more dynamic re-prioritization of tasks based on real-time environmental feedback, moving beyond simple sequential execution. This would enable the system to dynamically manage the robot’s behavior by allowing higher-priority tasks, such as immediate obstacle avoidance or emergency docking, to interrupt lower-priority stocking routines.

2. Bold Header: Enhanced Perception for Product Identification Robustness

Improve product identification reliability by integrating a more comprehensive fallback mechanism when dedicated product models are unavailable. The paper notes that failures occur when products lacking a dedicated YOLO model, so implementing a more sophisticated visual servoing strategy or leveraging GPT-4o with richer contextual prompts could reduce these classification errors.

3. Bold Header: Integrated Cost-Benefit Optimization Metric

Develop a quantitative metric beyond the current joint performance index to explicitly model operational risk associated with hardware failure versus task success. This would allow for proactive maintenance scheduling by identifying which components (e.g., the camera system vs. the manipulator) are contributing most significantly to overall performance degradation, directly addressing the hardware/software gap decomposition.

4. Bold Header: Predictive Constraint Management in MPC

Refine the Model Predictive Control (MPC) framework to incorporate explicit prediction of potential state constraints related to product geometry and shelf loading density. This would allow the MPC cost function to anticipate necessary trajectory adjustments before collision or constraint violation occurs, potentially leading to smoother, safer maneuvers in densely packed shelves.

Autonomous stocking in retail environments, particularly supermarkets, presents challenges due to dynamic human interactions, constrained spaces, and diverse product geometries. This paper introduces an efficient modular robotic system for autonomous shelf stocking, integrating commercially available hardware with a scalable algorithmic architecture. A major contribution of this work is the system integration of off-the-shelf hardware and ROS2-based perception, planning, and control into a single deployable platform for retail environments. Our solution leverages Behavior Trees (BTs) for task planning, fine-tuned vision models for object detection, and a two-step Model Predictive Control (MPC) framework for precise shelf navigation using ArUco markers. Laboratory experiments replicating realistic supermarket conditions demonstrate reliable performance, achieving over 98% success in pick-and-place operations across a total of more than 700 stocking events. However, our comparative benchmarks indicate that the performance and cost-effectiveness of current autonomous systems remain inferior to that of human workers, which we use to highlight key improvement areas and quantify the progress still required before widespread commercial deployment can realistically be achieved.