Vision-Guided Reorientation and Color Sorting of Writing Utensils Using a Low-Cost 4-DoF Arm

arXiv:2608.15968 · cs.RO · Submitted 2026-08-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Vision-Guided Reorientation and Color Sorting of Writing Utensils Using a Low-Cost 4-DoF Arm".

Dev: Here's the gist of the paper. Low-cost four-degree-of-freedom arms can reliably pick and place pens using visual intelligence and clever motion planning, rather than additional mechanical degrees of freedom The arm’s four joints address position (x, y, z) and gripper state but provide no independent axis for wrist rotation This work shows that perception and motion planning can enable such an arm, a roughly 200 Waveshare RoArm-M2-S, under a fixed overhead camera to detect and color-sort writing utensils without that joint System Pipeline Stages The system pipeline has three stages: oriented bounding box detection, pixel-to-robot coordinate projection, and motion strategy selection based on detected pen angle The color classifier labels each pen as blue, red, green, or grayscale by sampling the hue, saturation, and value (HSV) of pixels within the bounding box and selecting the dominant hue Perception and Color Classification

Title and authors: Rosa: So we're looking at the paper titled "Vision-Guided Reorientation and Color Sorting of Writing Utensils Using a Low-Cost four-DoF Arm," and the authors are Anirudh Rangarajan and Bibit Bianchini <ref:2608.15968#pg1>. It tackles the idea of using vision and planning to do something that normally needs more mechanical joints.

Dev: It’s interesting because they are working with a four degree of freedom arm, which is pretty low cost, but they know that picking up an object at any angle really requires five degrees of freedom to get it right.

Rosa: Exactly. The core idea here is that this paper shows you can use perception and motion planning instead of adding more mechanical joints to handle those extra degrees of freedom. It’s about overcoming a limitation in the hardware by using software intelligence instead.

Taro: I'm curious about how they handle that orientation problem without a wrist rotation joint being present on the arm itself. Does their vision system just give them enough information to compensate?

Dev: They use something called a YOLO11n-OBB detector to find each pen and get its orientation angle directly from the image. Then, they use those pixel coordinates and camera calibration data, along with an ArUco reference tag, to convert that into robot coordinates.

Rosa: That mapping step is crucial because it turns the visual information into something the arm understands in terms of where it needs to move in space. The paper shows they can detect and color-sort these pens without that extra wrist rotation joint by planning a path around the object based on its detected angle.

The paper's summary: Rosa: To summarize what they did, the vision system detects the pen's orientation, and then a color classifier labels it as blue, red, green, or grayscale by looking at specific colors within that bounding box. That’s how they sort them into four bins.

Dev: And their motion planning strategy is based on that orientation data. They decide whether to try a direct perpendicular grasp or use a series of nudges with short lateral sweeps of the pen to push it into a better position for grasping, depending on the detected angle.

Taro: So, if the vision gives you an angle, the planner decides whether to go straight in or sweep around it? That sounds like they are using perception to drive a decision-making process that dictates physical motion.

Rosa: Right. And they also have some improvements suggested in the paper that make this whole system more robust than just the basic setup described on page two, especially when dealing with real world issues.

Dev: They address a few things, like making the color classification work even if you're not seeing the perfect shade of blue or red by mapping those colors to the nearest bin. They also focus on keeping their robot coordinates accurate even when things drift mechanically over time.

The paper's improvements: Rosa: One improvement they point out is that they can use a fixed heuristic for color labeling, which means if the pen color is slightly off from the target, like a pink instead of pure red, it still gets sorted reliably by mapping it to the nearest bin.

Dev: That’s important because ambient light or how a pen is finished on one side can change how you see its hue in real time. Another big improvement they suggest is keeping calibration artifacts like the robot pose data saved so that the measurements stay accurate relative to the workspace throughout operation, preventing accumulated errors from ruining everything.

Taro: I want to ask about handling when things go wrong during motion. They mention introducing real time orientation feedback, maybe by running that object detector again between steps or using visual servoing directly on the pen axis instead of just planning a fixed path.

Rosa: That moves the system from an open loop where it tries a sweep and hoping for the best to something more closed loop where it can correct for slippage or bouncing as it happens during the motion.

Dev: They also suggest adapting how much they nudge based on what they observe about the motion, like inferring contact force or just seeing how fast things are moving to adjust those sweep distances instead of just using fixed steps, which handles friction differences better.

Conclusion: Rosa: So to wrap up, this paper shows that a relatively low cost four degree of freedom arm can pick and place pens by using vision to figure out the orientation and then motion planning to compensate for the missing joint. They use YOLO11n-OBB detection for orientation, a color classifier based on HSV values, and motion planning that chooses between direct grasp or nudging sweeps.

Dev: The main implication is that you don't necessarily need a five degree of freedom arm if your task doesn't actually require all those extra degrees of freedom, and perception can fill the gap effectively. The improvements focus on making it work better in messy real world situations with lighting variations and physical disturbances like slippage.

Taro: From my side, what stands out is how they turn that orientation data into a concrete action—either a direct pick or a series of sweeps—which shows how perception can directly influence the physical behavior of the robot.

Rosa: Exactly. And I think this kind of vision-guided approach is super useful because it opens up possibilities for picking up things in orientations we didn't even initially plan for, which is key for flexible manipulation tasks outside a perfect lab setting.

Dev: It sets a baseline for how perception and planning can combine to solve underactuation problems on simple hardware, which helps us think about safety cases where hardware constraints are tight.

Taro: It’s about making the system graceful when the world doesn't cooperate with our initial assumptions about object placement.

Anirudh Rangarajan, Bibit Bianchini

Dougherty Valley High School · University of Pennsylvania

cs.RO

Submitted: 2026-08-16

Updated: 2026-11-04

Comments: Accepted manuscript in the authors' formatting, 23 pages, 7 figures, 1 table. Published in the American Journal of Student Research, 2026;4(6):74-88. v1 was titled "Tabletop Pen Manipulation With a Vision-Guided 4-DoF Arm"

Journal ref: Am J Stud Res. 2026;4(6):74-88

DOI: 10.70251/HYJR2348.467488

Code: https://github.com/Anirudhpro/4DoF_vision_robotic_pen_sorting

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 78/100

The gist: Low-cost four-degree-of-freedom arms can reliably pick and place pens using visual intelligence and clever motion planning…

Key concepts

Four-Degree-of-Freedom Arm
This is a simple robot arm with four joints designed to handle positioning (x, y, z) and the gripper's state. Crucially, it lacks an independent axis for wrist rotation. The paper shows that perception and planning can compensate for this limitation to perform complex tasks like pen manipulation.
Oriented Bounding Box Detection (OBB)
This is an AI technique used to accurately find the rectangular boundaries of objects, like pens, in a camera image. It helps the system precisely locate the pen's position and orientation within the visual data. This step is essential for mapping pixel locations to real-world robot coordinates.
Pixel-to-Robot Coordinate Projection
This process translates 2D information from the camera image (pixels) into 3D coordinates that the robot arm understands. By using calibrated geometry, the system can determine exactly where in space a detected pen is located relative to the arm's base, enabling precise movement planning.
Color Classification via HSV/LAB Heuristic
The system identifies pen colors by sampling pixels within the detected bounding box and analyzing their Hue, Saturation, and Value (HSV) or Lab color space. A fixed heuristic maps these sampled colors to predefined bins (e.g., mapping 'pink' to 'red'), ensuring consistent sorting regardless of lighting changes.

Terminology

Summary

Low-cost four-degree-of-freedom arms can reliably pick and place pens using visual intelligence and clever motion planning, rather than additional mechanical degrees of freedom

System Architecture

The system pipeline has three stages: oriented bounding box detection, pixel-to-robot coordinate projection, and motion strategy selection based on detected pen angle The color classifier labels each pen as blue, red, green, or grayscale by sampling the hue, saturation, and value (HSV) of pixels within the bounding box and selecting the dominant hue Hardware setup includes a fixed overhead camera with an ArUco tag placed on the workspace to calibrate the extrinsics before beginning operations The end-effector features a single rotating jaw joint that is commanded in radians over about 118 degrees of travel

Perception Pipeline

The perception pipeline involves several steps to identify and characterize the writing utensil A YOLO11n-OBB detector locates each writing utensil, camera intrinsics and an ArUco reference pose convert its pixel coordinates to robot coordinates, and a color classifier labels it Color is read from the pixels inside the detected box using a custom rule built on OpenCV’s color conversions The system fills the oriented box into a mask and slightly shrinks it to exclude border pixels, which avoids errors at the box’s edges

Motion Strategy Selection

The motion planner uses the orientation to either attempt a direct perpendicular grasp or to perform one or more series of nudges, using short lateral sweeps of the pen to push it into a graspable orientation The arm cannot turn its gripper to meet a pen at an arbitrary angle: the direction it approaches from is fixed by where the pen sits on the table The system selects the motion with a single threshold: STANDARD if penRadialAngle < 45◦, COMPLEX otherwise A standard motion succeeds when the arm grasps the pen and releases it into the bin, assigned by its detected color A complex motion succeeds when its sweep leaves the pen below the 45-degree threshold, the condition that lets the following standard grasp pick it up Most initially misaligned pens were corrected in a single sweep pass (75 of 98), with steeper pens requiring up to four passes

Limitations and Future Work

Many of the shortcomings observed in this study stem from the system still being a prototype Pen surface finish and mass led to variability in friction, meaning identical sweep or nudge motions did not move all utensils by the same distance Color labeling is a separate stage from detection, and color errors came from that fixed gamut rather than the training set Future work could include introducing real-time orientation feedback during motion to turn open-loop sweeps into closed-loop behaviors The system's performance depends on the detector’s quality and the gripper’s mechanics

Conclusion

This work demonstrated that a low-cost 4-DoF arm can reliably pick and place pens using visual intelligence and clever motion planning, rather than additional mechanical degrees of freedom By combining YOLO11n-OBB with calibrated pixel-to-robot geometry, the system chooses between two simple motion policies It uses a direct pick for pens that are already somewhat aligned with the radial line from the robot origin to the pen center, and a sweep-then-pick strategy for highly misaligned pens Across 326 logged motions over seven pens, 196 were standard pick-and-place motions, and 130 were corrective sweep passes While the final performance still depends on the detector’s quality and the gripper’s mechanics, the results indicate that software-driven perception and planning can compensate for fewer DoFs by enabling compensating motions to be used sequentially to complete complex tasks while maintaining the lower cost and simplicity of 4-DoF hardware

Conflict of Interest

The authors declare no conflicts of interest related to this work During the preparation of this work, the author(s) used Claude (Anthropic) for programming assistance, Grammarly for language editing, and Canva for image editing The AI disclosure statement notes that the author(s) carefully reviewed and edited the content as necessary

References

The references include documentation for the hardware, such as Waveshare. RoArm-M2-S and OpenCV for ArUco markers The paper also cites research on fiducial markers and YOLO11 models used in the study The work relies on a custom rule built on OpenCV’s color conversions for estimating pen color The final operating sequence is orchestrated by the script full run. p. 5

Where the paper says it

py, this browser view runs in a compact desktop app-window (a Chrome appmode window served by a small local web server): during setup it provides buttons to jog the arm onto the ArUco tag and confirm its position and to capture the calibration photo, and during sorting it adds an AUTO toggle, a manual trigger, and a stop button beside the live detection feed. Estimating Pen Color Color is read from the pixels inside the detected box using a custom rule built on OpenCV’s color conversions.

p. 5
Where the paper says it

py, this browser view runs in a compact desktop app-window (a Chrome appmode window served by a small local web server): during setup it provides buttons to jog the arm onto the ArUco tag and confirm its position and to capture the calibration photo, and during sorting it adds an AUTO toggle, a manual trigger, and a stop button beside the live detection feed. Estimating Pen Color Color is read from the pixels inside the detected box using a custom rule built on OpenCV’s color conversions.

p. 5
Where the paper says it

py, this browser view runs in a compact desktop app-window (a Chrome appmode window served by a small local web server): during setup it provides buttons to jog the arm onto the ArUco tag and confirm its position and to capture the calibration photo, and during sorting it adds an AUTO toggle, a manual trigger, and a stop button beside the live detection feed. Estimating Pen Color Color is read from the pixels inside the detected box using a custom rule built on OpenCV’s color conversions.

py The paper also includes a parallel Matplotlib plot to visualize geometry in robot coordinates The authors acknowledge that mechanical drift can break the original alignment between the ArUco-defined world frame and the robot coordinate system The paper suggests future work involving visual servoing to create closed-loop behaviors for motion correction The conclusion emphasizes that software-driven perception and planning can compensate for fewer DoFs by enabling compensating motions

This work demonstrated that a low-cost 4-DoF arm can reliably pick and place pens using visual intelligence and clever motion planning, rather than additional mechanical degrees of freedom By combining YOLO11n-OBB with calibrated pixel-to-robot geometry, the system chooses between two simple motion policies It uses a direct pick for pens that are already somewhat aligned with the radial line from the robot origin to the pen center, and a sweep-then-pick strategy for highly misaligned pens Across 326 logged motions over seven pens, 196 were standard pick-and-place motions, and 130 were corrective sweep passes While the final performance still depends on the detector’s quality and the gripper’s mechanics, the results indicate that software-driven perception and planning can compensate for fewer DoFs by enabling compensating motions to be used sequentially to complete complex tasks while maintaining the lower cost and simplicity of 4-DoF hardware The references include documentation for the hardware, such as Waveshare. RoArm-M2-S and OpenCV for ArUco markers The paper also cites research on fiducial markers and YOLO11 models used in the study The work relies on a custom rule built on OpenCV’s color conversions for estimating pen color The final operating sequence is orchestrated by the script full run.py The paper also includes a parallel Matplotlib plot to visualize geometry in robot coordinates The authors acknowledge that mechanical drift can break the original alignment between the ArUco-defined world frame and the robot coordinate system The paper suggests future work involving visual servoing to create closed-loop behaviors for motion correction The conclusion emphasizes that software-driven perception and planning can compensate for fewer DoFs by enabling compensating motions

The authors declare no conflicts of interest related to this work During the preparation of this work, the author(s) used Claude (Anthropic) for programming assistance, Grammarly for language editing, and Canva for image editing The AI disclosure statement notes that the author(s) carefully reviewed and edited the content as necessary The references include documentation for the hardware, such as Waveshare. RoArm-M2-S and OpenCV for ArUco markers The paper also cites research on fiducial markers and YOLO11 models used in the study The work relies on a custom rule built on OpenCV’s color conversions for estimating pen color The final operating sequence is orchestrated by the script full run.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved AI system can reliably pick and place pens using visual intelligence and clever motion planning, rather than additional mechanical degrees of freedom. This is achieved by combining YOLO11n-OBB with calibrated pixel-to-robot geometry to choose between a direct pick or a sweep-then-pick strategy based on the pen's radial misalignment angle.

  2. Improved AI system can handle color labeling by using a fixed HSV/LAB heuristic that labels colors reliably, even if they fall outside the target gamut by mapping them to the nearest bin, such as pink to red and navy to grayscale. This ensures consistent sorting despite variations in ambient light or surface finish.

  3. Improved AI system can be made more robust against mechanical drift by keeping calibration artifacts like K, dist, rvec, and tvec on disk so that robot coordinates remain millimeter-accurate relative to the workspace during operation. This prevents accumulated pose errors from breaking the alignment between the ArUco-defined world frame and the robot coordinate system.

  4. Improved AI system can respond to disturbances by introducing real-time orientation feedback during motion, such as re-running the OBB detector between waypoints or using visual servoing on the pen axis. This turns current open-loop sweeps into closed-loop behaviors that can correct for slip or bounce as they happen.

  5. Improved AI system can adapt its nudging policies by modulating sweep distance based on observed motion or inferred contact force, which better handles friction differences between pens. This allows the system to reason about contact and adjust maneuvers dynamically rather than relying on fixed 10 mm steps.

Abstract

Low-cost four-degree-of-freedom (DoF) arms are among the most accessible robotic platforms. But they are one joint short for picking up in situations where objects are at arbitrary orientations, a task that appears to require five degrees of freedom: the planar position (x and y), the height (z), a wrist rotation to align the gripper with the object, and gripper actuation, of which a four-DoF arm lacks the wrist rotation. This work uses perception and motion planning so that such an arm, a roughly 200 Waveshare RoArm-M2-S under a fixed overhead camera, can detect and color-sort writing utensils without that joint. A YOLO11n-OBB (You Only Look Once, oriented bounding box) detector locates each writing utensil; camera intrinsics and an ArUco reference pose convert its pixel coordinates to robot coordinates; and a color classifier labels it. The detected orientation angle determines the motion strategy: utensils close to the arm's fixed approach direction are picked up directly, and those at steeper angles are reoriented via corrective sweeps until they are graspable, after which they are picked up and sorted into the assigned color bin. In 196 single-utensil trials, the 98 that started beyond 45 degrees from that direction all became graspable within four sweeps (130 total). Overall, 176 (89.8%) ended in the correct bin; 19 of 20 failures were slipped or missed grasps, one a mislabeled color. For the seven utensils tested in this setup, corrective sweeps brought every steeply misaligned utensil into a graspable orientation on an arm with no wrist joint.

Related papers