Autonomous Apple Fruitlet Sizing and Growth Rate Tracking using Computer Vision
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Autonomous Apple Fruitlet Sizing and Growth Rate Tracking using Computer Vision".
Dev: Here's the gist of the paper. In this paper, we present a computer vision-based approach to measure the sizes and growth rates of apple fruitlets because measuring growth rates is important because it allows apple growers to determine when to apply chemical thinners to their crops in order to optimize yield.
Rosa: System Overview The system is designed to replicate the process by both sizing the fruitlets and autonomously identifying which fruitlets in the image belong to the cluster intended to be sized.
Dev: Images are captured in the field using an illumination-invariant flash stereo camera, and once images of all clusters are acquired, the data is passed through our sizing and growthrate tracking pipelines.
Rosa: In the first stage, fruitlets belonging to target clusters are extracted and sized, where all fruitlets in each image are segmented using a Mask-RCNN instance segmentation network, and disparities are extracted using RAFT-Stereo to build point clouds of each fruitlet.
Paper summary: Rosa: So we're looking at this paper called "Autonomous Apple Fruitlet Sizing and Growth Rate Tracking using Computer Vision." The main idea is that they've developed a computer vision system to measure the size and growth rates of apple fruitlets. They claim that tracking these growth rates helps apple growers know exactly when to use chemical thinners on their crops so they can get the best yield.
Dev: It sounds like it’s addressing a problem with traditional methods, which involves using calipers to record sizes over several days. The authors say that this current practice is really laborious and takes too much time, and it's prone to human error because you have so many fruitlets to measure.
Taro: So the thesis here is that instead of those slow manual measurements, they can use images captured by a hand-held stereo camera to segment, cluster, and fit ellipses on the fruitlets. Then they calculate growth rates by linking those clustered fruitlets across different days.
Rosa: And what's really interesting is how they handle the data collection part. They capture images of all the clusters, and then their system identifies which ones belong to the target group autonomously before starting to size them.
Dev: The methodology seems pretty involved. They use a Mask-RCNN network to segment every fruitlet in each image, and then they use something called RAFT-Stereo to get the disparity information needed for building point clouds of each fruitlet.
Taro: So, the segmentation is done by Mask-RCNN following a two stage process where bounding box annotations are used first, and then a subset of fruitlets is segmented to train the mask head. And they use RAFT-Stereo instead of something like SGBM for disparity estimation because it's supposed to be better on benchmarks.
Rosa: They also use a multi-stage Louvain Community Detection approach on those point clouds to cluster the fruitlets and figure out which ones are in the target cluster. The sampling strategy they used is pretty specific too, they chose seven clusters per tree across ten trees for a total of seventy clusters instead of what was originally suggested <ref:2212.01506#pg3,7 clusters per tree across 10 trees for a total of 70>.
Paper summary: Dev: That's a change from the original model which suggested using fifteen clusters per tree across seven trees for a total of one hundred and five clusters <ref:2212.01506#pg3,using 15 clusters per tree across 7 trees for a total of>. They made that adjustment because they felt it allowed them to collect data on a wider range of fruitlet sizes and appearances, which is good for testing their system's performance on more diverse fruitlets.
Taro: They also used AprilTags to hang on the selected clusters so they can identify those specific clusters and the individual fruitlets within them. This helps with tracking during the experiment.
Rosa: The results they show are pretty strong when compared to existing methods, claiming their system can predict abscise rates within three point five percent of what's currently done, which is a six times improvement over the current method.
Dev: That's a big improvement in accuracy if it holds up in real-world conditions. They also mention that when comparing their full end-to-end pipeline against ground truth, they get an average KA AAPD of three point seven five percent and an E2E AAPD of three point three three percent.
Taro: It’s interesting that they also show some comparisons where they use the classical disparity estimation method, SGBM, or a naive association method and their results are significantly worse, with the SGBM pipeline showing an average KA AAPD and E2E AAPD of ten point five percent and ten point nine percent.
Rosa: So it seems like this computer vision approach is much more accurate than the older methods when you look at the data they collected in an apple orchard. But there are always caveats with these kinds of systems.
Dev: Right, Rosa, what about the practical side? Taro mentioned something earlier about limitations when we talk about deployment outside the lab.
Taro: Well, they noted that the system doesn't work well when it uses a robot following a pre-set path because it isn't reasoning about the environment like a human with a hand-held camera. It can get confused between a fruitlet that has fallen off and one that was missed by detection.
Paper summary: Rosa: That makes sense, because if the robot is just following its path blindly, it doesn't have that contextual awareness. They also said it struggles to size smaller fruitlets when they are fully occluded or large ones when they are only partially occluded.
Dev: So the system is good for controlled setups with a camera setup where you know what’s in the frame, but if you move to a fully autonomous field operation, that reasoning about where to capture an image gets tricky.
Taro: Exactly. It can't distinguish between a physical detachment and a detection failure, which impacts how accurately it predicts those abscise rates since the growth model considers fruitlets that have already fallen off during the measurement period.
Rosa: So for now, it seems like this computer vision approach is very promising for getting accurate measurements quickly compared to hand-caliper work, but there’s still a lot of work needed before it's ready to be used by growers in the field.
Dev: It sounds like the paper "Autonomous Apple Fruitlet Sizing and Growth Rate Tracking using Computer Vision" presents a solid computer vision pipeline for measuring apple fruitlets. The authors focus on using image segmentation and stereo matching, specifically Mask-RCNN and RAFT-Stereo, to overcome the labor issues of manual caliper measurements.
Rosa: They essentially show how you can automate the sizing and growth rate tracking process by letting the computer handle the heavy lifting of segmenting fruitlets and calculating their sizes from stereo images.
Dev: The implications here are that growers could get much faster data on fruitlet development, which directly translates to better decisions about when to apply chemicals to optimize yield. It moves the measurement process away from being slow and error-prone for a more automated, real-time decision support system.
Taro: But as we saw with the limitations, this isn't a plug-and-play solution for complex field operations yet because it doesn't fully model how a robot needs to reason about an uncertain environment.
Rosa: So the next step for this kind of work is definitely making sure that reasoning about those uncertain situations—like when fruitlets fall off—is handled gracefully so the system can be truly autonomous in a changing orchard environment.
Conclusion: Rosa: So, this paper is about using computer vision to measure apple fruitlet sizes and growth rates in the field so growers can better time their thinning applications for a bigger harvest.
Dev: It’s by some authors focusing on how you can automate that process using images from a camera system.
Rosa: They built this whole pipeline to figure out which fruitlets belong to the target group automatically before they even start measuring them.
Dev: The main idea is taking those images and using advanced networks, like Mask-RCNN for identifying the fruitlets and RAFT-Stereo for figuring out their precise sizes from the stereo cameras.
Rosa: They also use a way to cluster these fruitlets, which helps them sort out the ones they actually need to size based on what they’re looking for.
Dev: And what they show is that this method is significantly faster than using manual calipers, cutting the time down six times on average.
Rosa: It also claims a very low error rate when comparing their measurements against the real truth, like three point seven five percent average absolute percentage difference.
Dev: That’s pretty good compared to older methods that might have had errors over ten percent or more, which is what they showed with the SGBM comparison.
Rosa: So what this means for growers is they can get much quicker data on how fast their fruitlets are growing, which helps them optimize when to thin the trees.
Dev: It moves the measurement from being a slow, manual task to something that can give you near real-time data support for your decisions.
Rosa: But there are still limitations they mentioned, especially when you try to move it onto a robot in the field where things get messy.
Dev: Exactly. They said if a robot is just blindly following a path, it struggles to handle situations where fruitlets fall off or are only partially hidden by other fruitlets.
Rosa: So for now, this is very promising for controlled settings, but making it work reliably in the actual orchard environment with complex visual challenges is the next big hurdle.
Harry Freeman, Mohamad Qadri, Abhisesh Silwal, Paul O’Connor, Zachary Rubinstein, Daniel Cooley, George Kantor
Carnegie Mellon University Robotics Institute · University of Massachusetts Amherst Stockbridge School of Agriculture
cs.RO, cs.CV
Submitted: 2022-12-03
Updated: 2026-11-06
DOI: 10.1016/j.compag.2026.112611
Code: https://github.com/facebookresearch/detectron2
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 85/100
The gist: In this paper, we present a computer vision-based approach to measure the sizes and growth rates of apple fruitlets because measuring growth rates is important because it allows apple growers to…
Key concepts
- Mask-RCNN
- This is a type of instance segmentation network used to identify and outline every individual apple fruitlet within an image. It works by first drawing bounding boxes around potential objects and then refining those boxes to create precise masks, ensuring that the system can accurately separate one fruitlet from another.
- RAFT-Stereo
- This is a state-of-the-art deep learning network used for stereo matching, which estimates the distance (disparity) between corresponding points in two images. It is more accurate than older methods, allowing the system to build detailed 3D point clouds of the fruitlets from 2D pictures.
- Louvain Community Detection
- This is a multi-stage method used to group related fruitlets together based on their spatial relationships in the image. It helps the system automatically identify which detected fruitlets belong to the intended target cluster for sizing, even when they are crowded.
- Growth Rate Tracking
- The system tracks how much apple fruitlets have grown over time by measuring their size changes between different images. This data is crucial because it tells growers precisely when to apply chemical thinners to maximize the final harvest yield.
Terminology
Summary
In this paper, we present a computer vision-based approach to measure the sizes and growth rates of apple fruitlets because measuring growth rates is important because it allows apple growers to determine when to apply chemical thinners to their crops in order to optimize yield. p. 0 1
Autonomous Apple Fruitlet Sizing and Growth Rate
Tracking using Computer Vision
Harry Freeman 1∗
, Mohamad Qadri 1
, Abhisesh Silwal 1
, Paul O’Connor 2
, Zachary Rubinstein 1
,
Daniel Cooley 2
, and George Kantor 1
Abstract—In this paper, we present a computer vision-based
approach to measure the sizes and growth rates of apple fruitlets. Measuring the growth rates of apple fruitlets is important
because it allows apple growers to determine when to apply
chemical thinners to their crops in order to optimize yield.Where the paper says it
System Overview
The system is designed to replicate the process by both sizing the fruitlets and autonomously identifying which fruitlets in the image belong to the cluster intended to be sized. Images are captured in the field using an illumination-invariant flash stereo camera, and once images of all clusters are acquired, the data is passed through our sizing and growthrate tracking pipelines. p. 1 Images are captured in the field using an illuminationinvariant flash stereo camera [28]. Once images of all clusters
are acquired, the data is passed through our sizing and growthrate tracking pipelines (Fig. In the first stage, the fruitlets
belonging to the target clusters are extracted and sized. All
fruitlets in each image are segmented using a Mask-RCNN
[29] instance segmentation network, and disparities are extracted using RAFT-Stereo [30] to build point clouds of each
fruitlet. All
fruitlets in each image are segmented using a Mask-RCNN
[29] instance segmentation network, and disparities are extracted using RAFT-Stereo [30] to build point clouds of each
fruitlet. The point clouds are used to cluster the fruitlets using a
multi-stage Louvain Community Detection [31] approach, and
the fruitlets corresponding to the target cluster are identified.Where the paper says it
Where the paper says it
Where the paper says it
Fruitlet Cluster Sampling
The Fruitlet Growth Model specifies that growers select the number of trees and clusters per tree they want to size. Our system uses 7 clusters per tree across 10 trees for a total of 70 clusters instead of the original suggestion. p. 2 Fruitlet Cluster Sampling
The Fruitlet Growth Model specifies that growers choose
the number of trees and the number of clusters per tree they
want to size. While the original model suggests using 15
clusters per tree across 7 trees for a total of 105 clusters, we
instead use 7 clusters per tree across 10 trees for a total of 70
clusters. Additionally, spreading clusters
over more trees allows us to collect data on a wider range of
fruitlet sizes and appearances. This is advantageous as it allows
us to evaluate our system’s performance on a more diverse
and representative set of fruitlets. Because our objective is to
compare predicted thinner response between computer vision
and hand-caliper growth rates, and not compare against actual
abscise rates, this is a fair sampling strategy for the purpose
of this work. We hang AprilTags [33] to identify the selected clusters and
fruitlets.Where the paper says it
Where the paper says it
Where the paper says it
Segmentation and Disparity Estimation
We replace the MADNet disparity and pix2pix segmentation networks with RAFTStereo and Mask-RCNN respectively. RAFT-Stereo is a state-of-the-art deep learning-based stereo matching network that outperforms traditional disparity generation methods, such as SGBM, on benchmarks such as the Middleburry Stereo Evaluation. p. 2 Segmentation and Disparity Estimation
Two differences between our sizing approach and the one
presented by Qadri [26] is we replace the MADNet [27]
disparity and pix2pix [37] segmentation networks with RAFTStereo [30] and Mask-RCNN [29] respectively. RAFT-Stereo
is a state-of-the-art deep learning-based stereo matching network that outperforms traditional disparity generation methods, such as SGBM [38], on benchmarks such as the Middleburry Stereo Evaluation7
. 4
not need to be retrained between datasets or when switching
out the stereo camera being used. In addition, Mask-RCNN
is a more standard and widely adopted instance segmentation
network compared to pix2pix, which is traditionally used for
pair-to-pair image translation. To account for the difficulties in obtaining ground truth
segmented data, the Mask-RCNN network is trained following
a two stage process. In the first stage, bounding box annotations are used to train the bounding box regression and
classification heads.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The system achieves significant speed improvements compared to manual methods, with RAFT-Stereo consuming a majority of the time with an average of 4.32s. This pipeline was able to size fruitlets 6 times faster compared to hand-caliper measurements. p. 6 The response we received is that it takes
approximately 30s per cluster, which we confirmed when collecting ground truth measurements. Based on this preliminary
study, our computer vision pipeline was able to size fruitlets 6
times faster compared to hand-caliper measurements. Known and
E2E association results are provided to demonstrate the performance should the grower choose to manually or autonomously
associate the fruitlets respectively. Among the six clusters, our method achieves the lowest
average absolute percentage difference compared to ground
truth using both known associations and the full end-to-end
pipeline, with 3. Among the six clusters, our method achieves the lowest
average absolute percentage difference compared to ground
truth using both known associations and the full end-to-end
pipeline, with 3. 33% average KA AAPD and
E2E AAPD respectively.Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The system successfully predicts similar abscise rates with only one to two single stereo image pairs per cluster, without any human effort required to label fruitlets or take caliper measurements. The advantage our system brings is a faster and less labor intensive approach that produces comparable results. p. 8 Most notably, we are
able to predict similar abscise rates with only one to two
single stereo image pairs per cluster, without any human effort
required to label fruitlets or take caliper measurements. The
advantage our system brings is a faster and less labor intensive
approach that produces comparable results. The
advantage our system brings is a faster and less labor intensive
approach that produces comparable results. While our approach produces promising results, there is still
work needed in order to make it adoptable by growers.Where the paper says it
Where the paper says it
The performance of our full Fruitlet Growth Measurement Pipeline (FGMP) against one that uses the classical disparity estimation method SGBM or a naive association method shows that our method achieves the lowest average absolute percentage difference compared to ground truth using both known associations and the full end-to-end pipeline. The SGBM-FGMP pipeline achieved an average KA AAPD and E2E AAPD of 10. p. 6 Replacing RAFT with SGBM performed the worst. The SGBM-FGMP pipeline acheived an average KA AAPD
and E2E AAPD of 10. The SGBM-FGMP pipeline acheived an average KA AAPD
and E2E AAPD of 10. 9% respectively, with AAPD
values greater than 15% for both the 2021 datasets of the Fuji
and Gala varietals.Where the paper says it
Where the paper says it
The system fails to successfully size smaller, fully occluded fruitlets and large, partially occluded fruitlets when using a robot following a pre-set path because it is not reasoning about the environment as a human is with the hand-held camera. This is because the robot is naively following pre-set path and not reasoning about where to capture an image. p. 7 Across all days, the sizes produced by the computer vision
pipeline are larger on average than those produced by the
caliper method, and the distributions are much narrower. This
is because the robot is naively following pre-set path and not
reasoning about where to capture an image. Lastly, one drawback
of our approach is the system cannot distinguish between
a fruitlet that fell off and a missed detection. This affects
thinner response predictions as the Fruitlet Growth Model
considers fruitlets that have abscised during the sizing period.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Next-best-view planning integration allows robots to
take high quality images of all fruitlets in the cluster
by utilizing techniques like region of interest exploration, addressing current limitations where robots fail to size smaller, fully occluded fruitlets. -
The system can distinguish between a
fruitlet that fell off and a missed detection
by dedicating future work to this distinction, which is crucial for accurately replicating the Fruitlet Growth Model's prediction of persist and abscise rates. -
Lightweight models can be integrated to replace the backbone of Mask-RCNN, potentially allowing the system to run in real-time on mobile devices in the field, overcoming current computational resource constraints.
Abstract
In this paper, we present a computer vision-based approach to measure the sizes and growth rates of apple fruitlets. Measuring the growth rates of apple fruitlets is important because it allows apple growers to determine when to apply chemical thinners to their crops in order to optimize yield. The current practice of obtaining growth rates involves using calipers to record sizes of fruitlets across multiple days. Due to the number of fruitlets needed to be sized, this method is laborious, time-consuming, and prone to human error. With images collected by a hand-held stereo camera, our system, segments, clusters, and fits ellipses to fruitlets to measure their diameters. The growth rates are then calculated by temporally associating clustered fruitlets across days. We provide quantitative results on data collected in an apple orchard, and demonstrate that our system is able to predict abscise rates within 3.5% of the current method with a 6 times improvement in speed, while requiring significantly less manual effort. Moreover, we provide results on images captured by a robotic system in the field, and discuss the next steps required to make the process fully autonomous.
Sources
- Computer Vision with Deep Learning for Plant Phenotyping in Agriculture: A Survey
- Fruit Mapping with Shape Completion for Autonomous Crop Monitoring
- Real-time self-adaptive deep stereo
- Mask R-CNN
- RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching
- Viewpoint Planning for Fruit Size and Position Estimation
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation