Vision-Guided Reorientation and Color Sorting of Writing Utensils Using a Low-Cost 4-DoF Arm

summary

The gist

Low-cost four-degree-of-freedom arms can reliably pick and place pens using visual intelligence and clever motion planning…

In short

This work demonstrates how a low-cost four-degree-of-freedom arm can reliably pick and sort pens using only visual intelligence and motion planning, eliminating the need for extra mechanical joints. The system uses a pipeline involving bounding box detection, pixel-to-robot coordinate conversion, and color classification to identify pen type and color without needing a wrist rotation axis.

Key concepts

Four-Degree-of-Freedom Arm
This is a simple robot arm with four joints designed to handle positioning (x, y, z) and the gripper's state. Crucially, it lacks an independent axis for wrist rotation. The paper shows that perception and planning can compensate for this limitation to perform complex tasks like pen manipulation.
Oriented Bounding Box Detection (OBB)
This is an AI technique used to accurately find the rectangular boundaries of objects, like pens, in a camera image. It helps the system precisely locate the pen's position and orientation within the visual data. This step is essential for mapping pixel locations to real-world robot coordinates.
Pixel-to-Robot Coordinate Projection
This process translates 2D information from the camera image (pixels) into 3D coordinates that the robot arm understands. By using calibrated geometry, the system can determine exactly where in space a detected pen is located relative to the arm's base, enabling precise movement planning.
Color Classification via HSV/LAB Heuristic
The system identifies pen colors by sampling pixels within the detected bounding box and analyzing their Hue, Saturation, and Value (HSV) or Lab color space. A fixed heuristic maps these sampled colors to predefined bins (e.g., mapping 'pink' to 'red'), ensuring consistent sorting regardless of lighting changes.

Terminology used across episodes

This episode discusses

The paper

Vision-Guided Reorientation and Color Sorting of Writing Utensils Using a Low-Cost 4-DoF Arm · Read on arXiv

Anirudh Rangarajan, Bibit Bianchini

Dougherty Valley High School · University of Pennsylvania

Low-cost four-degree-of-freedom (DoF) arms are among the most accessible robotic platforms. But they are one joint short for picking up in situations where objects are at arbitrary orientations, a task that appears to require five degrees of freedom: the planar position (x and y), the height (z), a wrist rotation to align the gripper with the object, and gripper actuation, of which a four-DoF arm lacks the wrist rotation. This work uses perception and motion planning so that such an arm, a roughly 200 Waveshare RoArm-M2-S under a fixed overhead camera, can detect and color-sort writing utensils without that joint. A YOLO11n-OBB (You Only Look Once, oriented bounding box) detector locates each writing utensil; camera intrinsics and an ArUco reference pose convert its pixel coordinates to robot coordinates; and a color classifier labels it. The detected orientation angle determines the motion strategy: utensils close to the arm's fixed approach direction are picked up directly, and those at steeper angles are reoriented via corrective sweeps until they are graspable, after which they are picked up and sorted into the assigned color bin. In 196 single-utensil trials, the 98 that started beyond 45 degrees from that direction all became graspable within four sweeps (130 total). Overall, 176 (89.8%) ended in the correct bin; 19 of 20 failures were slipped or missed grasps, one a mislabeled color. For the seven utensils tested in this setup, corrective sweeps brought every steeply misaligned utensil into a graspable orientation on an arm with no wrist joint.

DOI: 10.70251/HYJR2348.467488

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Vision-Guided Reorientation and Color Sorting of Writing Utensils Using a Low-Cost 4-DoF Arm".

Dev: Here's the gist of the paper. Low-cost four-degree-of-freedom arms can reliably pick and place pens using visual intelligence and clever motion planning, rather than additional mechanical degrees of freedom The arm’s four joints address position (x, y, z) and gripper state but provide no independent axis for wrist rotation This work shows that perception and motion planning can enable such an arm, a roughly 200 Waveshare RoArm-M2-S, under a fixed overhead camera to detect and color-sort writing utensils without that joint System Pipeline Stages The system pipeline has three stages: oriented bounding box detection, pixel-to-robot coordinate projection, and motion strategy selection based on detected pen angle The color classifier labels each pen as blue, red, green, or grayscale by sampling the hue, saturation, and value (HSV) of pixels within the bounding box and selecting the dominant hue Perception and Color Classification

Title and authors: Rosa: So we're looking at the paper titled "Vision-Guided Reorientation and Color Sorting of Writing Utensils Using a Low-Cost four-DoF Arm," and the authors are Anirudh Rangarajan and Bibit Bianchini <ref:2608.15968#pg1>. It tackles the idea of using vision and planning to do something that normally needs more mechanical joints.

Dev: It’s interesting because they are working with a four degree of freedom arm, which is pretty low cost, but they know that picking up an object at any angle really requires five degrees of freedom to get it right.

Rosa: Exactly. The core idea here is that this paper shows you can use perception and motion planning instead of adding more mechanical joints to handle those extra degrees of freedom. It’s about overcoming a limitation in the hardware by using software intelligence instead.

Taro: I'm curious about how they handle that orientation problem without a wrist rotation joint being present on the arm itself. Does their vision system just give them enough information to compensate?

Dev: They use something called a YOLO11n-OBB detector to find each pen and get its orientation angle directly from the image. Then, they use those pixel coordinates and camera calibration data, along with an ArUco reference tag, to convert that into robot coordinates.

Rosa: That mapping step is crucial because it turns the visual information into something the arm understands in terms of where it needs to move in space. The paper shows they can detect and color-sort these pens without that extra wrist rotation joint by planning a path around the object based on its detected angle.

The paper's summary: Rosa: To summarize what they did, the vision system detects the pen's orientation, and then a color classifier labels it as blue, red, green, or grayscale by looking at specific colors within that bounding box. That’s how they sort them into four bins.

Dev: And their motion planning strategy is based on that orientation data. They decide whether to try a direct perpendicular grasp or use a series of nudges with short lateral sweeps of the pen to push it into a better position for grasping, depending on the detected angle.

Taro: So, if the vision gives you an angle, the planner decides whether to go straight in or sweep around it? That sounds like they are using perception to drive a decision-making process that dictates physical motion.

Rosa: Right. And they also have some improvements suggested in the paper that make this whole system more robust than just the basic setup described on page two, especially when dealing with real world issues.

Dev: They address a few things, like making the color classification work even if you're not seeing the perfect shade of blue or red by mapping those colors to the nearest bin. They also focus on keeping their robot coordinates accurate even when things drift mechanically over time.

The paper's improvements: Rosa: One improvement they point out is that they can use a fixed heuristic for color labeling, which means if the pen color is slightly off from the target, like a pink instead of pure red, it still gets sorted reliably by mapping it to the nearest bin.

Dev: That’s important because ambient light or how a pen is finished on one side can change how you see its hue in real time. Another big improvement they suggest is keeping calibration artifacts like the robot pose data saved so that the measurements stay accurate relative to the workspace throughout operation, preventing accumulated errors from ruining everything.

Taro: I want to ask about handling when things go wrong during motion. They mention introducing real time orientation feedback, maybe by running that object detector again between steps or using visual servoing directly on the pen axis instead of just planning a fixed path.

Rosa: That moves the system from an open loop where it tries a sweep and hoping for the best to something more closed loop where it can correct for slippage or bouncing as it happens during the motion.

Dev: They also suggest adapting how much they nudge based on what they observe about the motion, like inferring contact force or just seeing how fast things are moving to adjust those sweep distances instead of just using fixed steps, which handles friction differences better.

Conclusion: Rosa: So to wrap up, this paper shows that a relatively low cost four degree of freedom arm can pick and place pens by using vision to figure out the orientation and then motion planning to compensate for the missing joint. They use YOLO11n-OBB detection for orientation, a color classifier based on HSV values, and motion planning that chooses between direct grasp or nudging sweeps.

Dev: The main implication is that you don't necessarily need a five degree of freedom arm if your task doesn't actually require all those extra degrees of freedom, and perception can fill the gap effectively. The improvements focus on making it work better in messy real world situations with lighting variations and physical disturbances like slippage.

Taro: From my side, what stands out is how they turn that orientation data into a concrete action—either a direct pick or a series of sweeps—which shows how perception can directly influence the physical behavior of the robot.

Rosa: Exactly. And I think this kind of vision-guided approach is super useful because it opens up possibilities for picking up things in orientations we didn't even initially plan for, which is key for flexible manipulation tasks outside a perfect lab setting.

Dev: It sets a baseline for how perception and planning can combine to solve underactuation problems on simple hardware, which helps us think about safety cases where hardware constraints are tight.

Taro: It’s about making the system graceful when the world doesn't cooperate with our initial assumptions about object placement.

More episodes

← Home