Autonomous Apple Fruitlet Sizing and Growth Rate Tracking using Computer Vision
summary
The gist
In this paper, we present a computer vision-based approach to measure the sizes and growth rates of apple fruitlets because measuring growth rates is important because it allows apple growers to…
In short
This system uses computer vision to measure apple fruitlet sizes and growth rates autonomously for growers. It captures images of fruit clusters, segments individual fruitlets using Mask-RCNN, and estimates depth using RAFT-Stereo. This allows for automated sizing and growth tracking, helping growers optimize chemical thinning for better yields.
Key concepts
- Mask-RCNN
- This is a type of instance segmentation network used to identify and outline every individual apple fruitlet within an image. It works by first drawing bounding boxes around potential objects and then refining those boxes to create precise masks, ensuring that the system can accurately separate one fruitlet from another.
- RAFT-Stereo
- This is a state-of-the-art deep learning network used for stereo matching, which estimates the distance (disparity) between corresponding points in two images. It is more accurate than older methods, allowing the system to build detailed 3D point clouds of the fruitlets from 2D pictures.
- Louvain Community Detection
- This is a multi-stage method used to group related fruitlets together based on their spatial relationships in the image. It helps the system automatically identify which detected fruitlets belong to the intended target cluster for sizing, even when they are crowded.
- Growth Rate Tracking
- The system tracks how much apple fruitlets have grown over time by measuring their size changes between different images. This data is crucial because it tells growers precisely when to apply chemical thinners to maximize the final harvest yield.
Terminology used across episodes
This episode discusses
- Autonomous Apple Fruitlet Sizing and Growth Rate Tracking using Computer Vision · Paper Radio
- Computer Vision with Deep Learning for Plant Phenotyping in Agriculture: A Survey
- Fruit Mapping with Shape Completion for Autonomous Crop Monitoring
- Real-time self-adaptive deep stereo
- Mask R-CNN
- RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching
- Viewpoint Planning for Fruit Size and Position Estimation
The paper
Autonomous Apple Fruitlet Sizing and Growth Rate Tracking using Computer Vision · Read on arXiv
Harry Freeman, Mohamad Qadri, Abhisesh Silwal, Paul O’Connor, Zachary Rubinstein, Daniel Cooley, George Kantor
Carnegie Mellon University Robotics Institute · University of Massachusetts Amherst Stockbridge School of Agriculture
In this paper, we present a computer vision-based approach to measure the sizes and growth rates of apple fruitlets. Measuring the growth rates of apple fruitlets is important because it allows apple growers to determine when to apply chemical thinners to their crops in order to optimize yield. The current practice of obtaining growth rates involves using calipers to record sizes of fruitlets across multiple days. Due to the number of fruitlets needed to be sized, this method is laborious, time-consuming, and prone to human error. With images collected by a hand-held stereo camera, our system, segments, clusters, and fits ellipses to fruitlets to measure their diameters. The growth rates are then calculated by temporally associating clustered fruitlets across days. We provide quantitative results on data collected in an apple orchard, and demonstrate that our system is able to predict abscise rates within 3.5% of the current method with a 6 times improvement in speed, while requiring significantly less manual effort. Moreover, we provide results on images captured by a robotic system in the field, and discuss the next steps required to make the process fully autonomous.
DOI: 10.1016/j.compag.2026.112611
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Autonomous Apple Fruitlet Sizing and Growth Rate Tracking using Computer Vision".
Dev: Here's the gist of the paper. In this paper, we present a computer vision-based approach to measure the sizes and growth rates of apple fruitlets because measuring growth rates is important because it allows apple growers to determine when to apply chemical thinners to their crops in order to optimize yield.
Rosa: System Overview The system is designed to replicate the process by both sizing the fruitlets and autonomously identifying which fruitlets in the image belong to the cluster intended to be sized.
Dev: Images are captured in the field using an illumination-invariant flash stereo camera, and once images of all clusters are acquired, the data is passed through our sizing and growthrate tracking pipelines.
Rosa: In the first stage, fruitlets belonging to target clusters are extracted and sized, where all fruitlets in each image are segmented using a Mask-RCNN instance segmentation network, and disparities are extracted using RAFT-Stereo to build point clouds of each fruitlet.
Paper summary: Rosa: So we're looking at this paper called "Autonomous Apple Fruitlet Sizing and Growth Rate Tracking using Computer Vision." The main idea is that they've developed a computer vision system to measure the size and growth rates of apple fruitlets. They claim that tracking these growth rates helps apple growers know exactly when to use chemical thinners on their crops so they can get the best yield.
Dev: It sounds like it’s addressing a problem with traditional methods, which involves using calipers to record sizes over several days. The authors say that this current practice is really laborious and takes too much time, and it's prone to human error because you have so many fruitlets to measure.
Taro: So the thesis here is that instead of those slow manual measurements, they can use images captured by a hand-held stereo camera to segment, cluster, and fit ellipses on the fruitlets. Then they calculate growth rates by linking those clustered fruitlets across different days.
Rosa: And what's really interesting is how they handle the data collection part. They capture images of all the clusters, and then their system identifies which ones belong to the target group autonomously before starting to size them.
Dev: The methodology seems pretty involved. They use a Mask-RCNN network to segment every fruitlet in each image, and then they use something called RAFT-Stereo to get the disparity information needed for building point clouds of each fruitlet.
Taro: So, the segmentation is done by Mask-RCNN following a two stage process where bounding box annotations are used first, and then a subset of fruitlets is segmented to train the mask head. And they use RAFT-Stereo instead of something like SGBM for disparity estimation because it's supposed to be better on benchmarks.
Rosa: They also use a multi-stage Louvain Community Detection approach on those point clouds to cluster the fruitlets and figure out which ones are in the target cluster. The sampling strategy they used is pretty specific too, they chose seven clusters per tree across ten trees for a total of seventy clusters instead of what was originally suggested <ref:2212.01506#pg3,7 clusters per tree across 10 trees for a total of 70>.
Paper summary: Dev: That's a change from the original model which suggested using fifteen clusters per tree across seven trees for a total of one hundred and five clusters <ref:2212.01506#pg3,using 15 clusters per tree across 7 trees for a total of>. They made that adjustment because they felt it allowed them to collect data on a wider range of fruitlet sizes and appearances, which is good for testing their system's performance on more diverse fruitlets.
Taro: They also used AprilTags to hang on the selected clusters so they can identify those specific clusters and the individual fruitlets within them. This helps with tracking during the experiment.
Rosa: The results they show are pretty strong when compared to existing methods, claiming their system can predict abscise rates within three point five percent of what's currently done, which is a six times improvement over the current method.
Dev: That's a big improvement in accuracy if it holds up in real-world conditions. They also mention that when comparing their full end-to-end pipeline against ground truth, they get an average KA AAPD of three point seven five percent and an E2E AAPD of three point three three percent.
Taro: It’s interesting that they also show some comparisons where they use the classical disparity estimation method, SGBM, or a naive association method and their results are significantly worse, with the SGBM pipeline showing an average KA AAPD and E2E AAPD of ten point five percent and ten point nine percent.
Rosa: So it seems like this computer vision approach is much more accurate than the older methods when you look at the data they collected in an apple orchard. But there are always caveats with these kinds of systems.
Dev: Right, Rosa, what about the practical side? Taro mentioned something earlier about limitations when we talk about deployment outside the lab.
Taro: Well, they noted that the system doesn't work well when it uses a robot following a pre-set path because it isn't reasoning about the environment like a human with a hand-held camera. It can get confused between a fruitlet that has fallen off and one that was missed by detection.
Paper summary: Rosa: That makes sense, because if the robot is just following its path blindly, it doesn't have that contextual awareness. They also said it struggles to size smaller fruitlets when they are fully occluded or large ones when they are only partially occluded.
Dev: So the system is good for controlled setups with a camera setup where you know what’s in the frame, but if you move to a fully autonomous field operation, that reasoning about where to capture an image gets tricky.
Taro: Exactly. It can't distinguish between a physical detachment and a detection failure, which impacts how accurately it predicts those abscise rates since the growth model considers fruitlets that have already fallen off during the measurement period.
Rosa: So for now, it seems like this computer vision approach is very promising for getting accurate measurements quickly compared to hand-caliper work, but there’s still a lot of work needed before it's ready to be used by growers in the field.
Dev: It sounds like the paper "Autonomous Apple Fruitlet Sizing and Growth Rate Tracking using Computer Vision" presents a solid computer vision pipeline for measuring apple fruitlets. The authors focus on using image segmentation and stereo matching, specifically Mask-RCNN and RAFT-Stereo, to overcome the labor issues of manual caliper measurements.
Rosa: They essentially show how you can automate the sizing and growth rate tracking process by letting the computer handle the heavy lifting of segmenting fruitlets and calculating their sizes from stereo images.
Dev: The implications here are that growers could get much faster data on fruitlet development, which directly translates to better decisions about when to apply chemicals to optimize yield. It moves the measurement process away from being slow and error-prone for a more automated, real-time decision support system.
Taro: But as we saw with the limitations, this isn't a plug-and-play solution for complex field operations yet because it doesn't fully model how a robot needs to reason about an uncertain environment.
Rosa: So the next step for this kind of work is definitely making sure that reasoning about those uncertain situations—like when fruitlets fall off—is handled gracefully so the system can be truly autonomous in a changing orchard environment.
Conclusion: Rosa: So, this paper is about using computer vision to measure apple fruitlet sizes and growth rates in the field so growers can better time their thinning applications for a bigger harvest.
Dev: It’s by some authors focusing on how you can automate that process using images from a camera system.
Rosa: They built this whole pipeline to figure out which fruitlets belong to the target group automatically before they even start measuring them.
Dev: The main idea is taking those images and using advanced networks, like Mask-RCNN for identifying the fruitlets and RAFT-Stereo for figuring out their precise sizes from the stereo cameras.
Rosa: They also use a way to cluster these fruitlets, which helps them sort out the ones they actually need to size based on what they’re looking for.
Dev: And what they show is that this method is significantly faster than using manual calipers, cutting the time down six times on average.
Rosa: It also claims a very low error rate when comparing their measurements against the real truth, like three point seven five percent average absolute percentage difference.
Dev: That’s pretty good compared to older methods that might have had errors over ten percent or more, which is what they showed with the SGBM comparison.
Rosa: So what this means for growers is they can get much quicker data on how fast their fruitlets are growing, which helps them optimize when to thin the trees.
Dev: It moves the measurement from being a slow, manual task to something that can give you near real-time data support for your decisions.
Rosa: But there are still limitations they mentioned, especially when you try to move it onto a robot in the field where things get messy.
Dev: Exactly. They said if a robot is just blindly following a path, it struggles to handle situations where fruitlets fall off or are only partially hidden by other fruitlets.
Rosa: So for now, this is very promising for controlled settings, but making it work reliably in the actual orchard environment with complex visual challenges is the next big hurdle.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies