Deep learning for real‑time fruit detection and orchard fruit load estimation: benchmarking of ‘MangoYOLO’

Article


Koirala, A, Walsh, K.B, Wang, Z and McCarthy, C. 2019. "Deep learning for real‑time fruit detection and orchard fruit load estimation: benchmarking of ‘MangoYOLO’." Precision Agriculture. 20 (6), pp. 1107-1135. https://doi.org/10.1007/s11119-019-09642-0
Article Title

Deep learning for real‑time fruit detection and orchard fruit load estimation: benchmarking of ‘MangoYOLO’

ERA Journal ID5325
Article CategoryArticle
AuthorsKoirala, A (Author), Walsh, K.B (Author), Wang, Z (Author) and McCarthy, C
Journal TitlePrecision Agriculture
Journal Citation20 (6), pp. 1107-1135
Number of Pages28
Year2019
PublisherSpringer
Place of PublicationUnited States
ISSN1385-2256
1573-1618
Digital Object Identifier (DOI)https://doi.org/10.1007/s11119-019-09642-0
Web Address (URL)https://link.springer.com/article/10.1007/s11119-019-09642-0
Abstract

The performance of six existing deep learning architectures were compared for the task of detection of mango fruit in images of tree canopies. Images of trees (n = 1 515) from across five orchards were acquired at night using a 5 Mega-pixel RGB digital camera and 720 W of LED flood lighting in a rig mounted on a farm utility vehicle operating at 6 km/h. The two stage deep learning architectures of Faster R-CNN(VGG) and Faster R-CNN(ZF), and the single stage techniques YOLOv3, YOLOv2, YOLOv2(tiny) and SSD were trained both with original resolution and 512 × 512 pixel versions of 1 300 training tiles, while YOLOv3 was run only with 512 × 512 pixel images, giving a total of eleven models. A new architecture was also developed, based on features of YOLOv3 and YOLOv2(tiny), on the design criteria of accuracy and speed for the current application. This architecture, termed ‘MangoYOLO’, was trained using: (i) the 1 300 tile training set, (ii) the COCO dataset before training on the mango training set, and (iii) a daytime image training set of a previous publication, to create the MangoYOLO models ‘s’, ‘pt’ and ‘bu’, respectively. Average Precision plateaued with use of around 400 training tiles. MangoYOLO(pt) achieved a F1 score of 0.968 and Average Precision of 0.983 on a test set independent of the training set, outperforming other algorithms, with a detection speed of 8 ms per 512 × 512 pixel image tile while using just 833 Mb GPU memory per image (on a NVIDIA GeForce GTX 1070 Ti GPU) used for in-field application. The MangoYOLO model also outperformed other models in processing of full images, requiring just 70 ms per image (2 048 × 2 048 pixels) (i.e., capable of processing ~ 14 fps) with use of 4 417 Mb of GPU memory. The model was robust in use with images of other orchards, cultivars and lighting conditions. MangoYOLO(bu) achieved a F1 score of 0.89 on a day-time mango image dataset. With use of a correction factor estimated from the ratio of human count of fruit in images of the two sides of sample trees per orchard and a hand harvest count of all fruit on those trees, MangoYOLO(pt) achieved orchard fruit load estimates of between 4.6 and 15.2% of packhouse fruit counts for the five orchards considered. The labelled images (1 300 training, 130 validation and 300 test) of this study are available for comparative studies.

Keywordsdeep learning, fruit detection, mango, yield estimation
Contains Sensitive ContentDoes not contain sensitive content
ANZSRC Field of Research 2020300802. Horticultural crop growth and development
460304. Computer vision
400799. Control engineering, mechatronics and robotics not elsewhere classified
Public Notes

Files associated with this item cannot be displayed due to copyright restrictions

Byline AffiliationsCentral Queensland University
Centre for Agricultural Engineering
Institution of OriginUniversity of Southern Queensland
Permalink -

https://research.usq.edu.au/item/q5882/deep-learning-for-real-time-fruit-detection-and-orchard-fruit-load-estimation-benchmarking-of-mangoyolo

  • 329
    total views
  • 7
    total downloads
  • 4
    views this month
  • 0
    downloads this month

Export as

Related outputs

Non-visual common root rot disease detection using NIR spectrum and machine learning methods
Xiong, Yiyi, McCarthy, Cheryl, Humpal, Jacob and Percy, Cassandra. "Non-visual common root rot disease detection using NIR spectrum and machine learning methods." 2023 11th International Conference on Agro-Geoinformatics (Agro-Geoinformatics). Wuhan, China 25 - 28 Jul 2023 IEEE (Institute of Electrical and Electronics Engineers). https://doi.org/https://ieeexplore.ieee.org/xpl/conhome/10233256/proceeding
A review on common root rot of wheat and barley in Australia
Xiong, Yiyi, McCarthy, Cheryl, Humpal, Jacob and Percy, Cassandra. 2023. "A review on common root rot of wheat and barley in Australia." Plant Pathology. 72 (8), pp. 1347-1364. https://doi.org/10.1111/ppa.13777
NIR sensing and machine learning to rapidly assess crown rot in wheat
Humpal, Jacob, McCarthy, Cheryl, Percy, Cassandra and Thomasson, J. Alex. 2020. "NIR sensing and machine learning to rapidly assess crown rot in wheat." GRDC Grains Research Update: Driving Profit Through Research. Goondiwindi, Australia 03 - 04 Mar 2020 Australia.
Detection of crown rot in wheat utilising near-infrared spectroscopy: towards remote and robotic sensing
Humpal, Jacob, McCarthy, Cheryl, Percy, Cassy and Thomasson, J. Alex. 2020. "Detection of crown rot in wheat utilising near-infrared spectroscopy: towards remote and robotic sensing." Thomasson, J. Alex and Torres-Rua, Alfonso F. (ed.) Autonomous Air and Ground Sensing Systems for Agricultural Optimization and Phenotyping V (SPIE 2020). Online 27 Apr - 08 May 2020 United States. https://doi.org/10.1117/12.2557949
Machine vision learning for crop disease and quality parameters
Humpal, Jacob, McCarthy, Cheryl, Thomasson, J. Alex and Percy, Cassy. 2018. "Machine vision learning for crop disease and quality parameters." 21st Precision Agriculture Symposium (2018). Adelaide, Australia Australia.
Space Agriculture: Sensing Crops in Space
Humpal, Jacob, McCarthy, Cheryl, Baillie, Craig, Percy, Cassy and Brett, Peter. 2022. "Space Agriculture: Sensing Crops in Space." GRDC Update Papers.
Preliminary evaluation of real-time sensing of harvester losses by machine vision
McCarthy Chery. 2022. "Preliminary evaluation of real-time sensing of harvester losses by machine vision ." 43rd Annual Conference of the Australian Society of Sugar Cane Technologists (ASSCT 2022). Mackay, Australia 19 - 22 Apr 2022 Australia. Australian Society of Sugar Cane Technologists. pp. 108-110
Deep learning - method overview and review of use for fruit detection and yield estimation
Koirala, Anand, Walsh, Kerry B., Wang, Zhenglin and McCarthy, Cheryl. 2019. "Deep learning - method overview and review of use for fruit detection and yield estimation." Computers and Electronics in Agriculture. 162, pp. 219-234. https://doi.org/10.1016/j.compag.2019.04.017
A review of autonomous navigation systems in agricultural environments
Shalal, N., Low, T., McCarthy, C. and Hancock, N.. 2013. "A review of autonomous navigation systems in agricultural environments." SEAg 2013: Innovative Agricultural Technologies for a Sustainable Future. Barton, Australia 22 - 25 Sep 2013
Row and water front detection from UAV thermal-infrared imagery for furrow irrigation monitoring
Long, Derek, McCarthy, Cheryl and Jensen, Troy. 2016. "Row and water front detection from UAV thermal-infrared imagery for furrow irrigation monitoring." 2016 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM 2016). Banff, Canada 12 - 15 Jul 2016 https://doi.org/10.1109/AIM.2016.7576783
Orchard mapping and mobile robot localisation using on-board camera and laser scanner data fusion – Part B: mapping and localisation
Shalal, Nagham, Low, Tobias, McCarthy, Cheryl and Hancock, Nigel. 2015. "Orchard mapping and mobile robot localisation using on-board camera and laser scanner data fusion – Part B: mapping and localisation." Computers and Electronics in Agriculture. 119, pp. 267-278. https://doi.org/10.1016/j.compag.2015.09.026
Orchard mapping and mobile robot localisation using on-board camera and laser scanner data fusion – Part A: tree detection
Shalal, Nagham, Low, Tobias, McCarthy, Cheryl and Hancock, Nigel. 2015. "Orchard mapping and mobile robot localisation using on-board camera and laser scanner data fusion – Part A: tree detection." Computers and Electronics in Agriculture. 119, pp. 254-266. https://doi.org/10.1016/j.compag.2015.09.025
Evaluation of stereovision for extracting plant features
Mohammed Amean, Z., Low, T., McCarthy, C. and Hancock, N.. 2013. "Evaluation of stereovision for extracting plant features." SEAg 2013: Innovative Agricultural Technologies for a Sustainable Future. Barton, Australia 22 - 25 Sep 2013 Canberra, Australia.
A preliminary evaluation of vision and laser sensing for tree trunk detection and orchard mapping
Shalal, Nagham, Low, Tobias, McCarthy, Cheryl and Hancock, Nigel. 2013. "A preliminary evaluation of vision and laser sensing for tree trunk detection and orchard mapping." Katupitiya, Jayantha, Guivant, Jose and Eaton, Ray (ed.) Australasian Conference on Robotics and Automation (ACRA 2013). Sydney, Australia 02 - 04 Dec 2013 Australia.
Automatic plant branch segmentation and classification using vesselness measure
Mohammed Amean, Z., Low, T., McCarthy, C. and Hancock, N.. 2013. "Automatic plant branch segmentation and classification using vesselness measure." Katupitiya, Jayantha, Guivant, Jose and Eaton, Ray (ed.) Australasian Conference on Robotics and Automation (ACRA 2013). Sydney, Australia 02 - 04 Dec 2013 Australia.
Commercialisation of precision agriculture technologies in the macadamia industry
Rees, Steven, Dunn, Mark, Werkman, Peter and McCarthy, Cheryl. 2009. Commercialisation of precision agriculture technologies in the macadamia industry. Toowoomba, Australia. University of Southern Queensland.
Preliminary evaluation of shape and colour image sensing for automated weed identification in sugarcane
McCarthy, Cheryl, Rees, Steven and Baillie, Craig. 2012. "Preliminary evaluation of shape and colour image sensing for automated weed identification in sugarcane." Bruce, R. C. (ed.) 34th Annual Conference of the Australian Society of Sugar Cane Technologists (ASSCT 2012). Palm Cove, Australia 01 - 04 May 2012 Australia.
Development and evaluation of a prototype precision spot spray system using image analysis to target guinea grass in sugarcane
Rees, S. J., McCarthy, C. L., Baillie, C. P., Burgos-Artizzu, X. P. and Dunn, M. T.. 2011. "Development and evaluation of a prototype precision spot spray system using image analysis to target guinea grass in sugarcane." Australian Journal of Multi-Disciplinary Engineering. 8 (2), pp. 97-106.
Automated internode length measurement of cotton plants under field conditions
McCarthy, C. L., Hancock, N. H. and Raine, S. R.. 2009. "Automated internode length measurement of cotton plants under field conditions." Transactions of the ASABE. 52 (6), pp. 2093-2103.
On-the-go machine vision sensing of cotton plant geometric parameters: first results
McCarthy, Cheryl, Hancock, Nigel and Raine, Steven. 2008. "On-the-go machine vision sensing of cotton plant geometric parameters: first results." Billingsley, John and Bradbeer, Robin (ed.) 13th Annual Conference on Mechatronics and Machine Vision in Practice. (M2VIP 2006). Toowoomba, Australia 05 - 07 Dec 2006 Germany. pp. 305-312 https://doi.org/10.1007/978-3-540-74027-8_26
Applied machine vision of plants: a review with implications for field deployment in automated farming operations
McCarthy, C. L., Hancock, N. H. and Raine, S. R.. 2010. "Applied machine vision of plants: a review with implications for field deployment in automated farming operations." Intelligent Service Robotics. 3 (4), pp. 209-217. https://doi.org/10.1007/s11370-010-0075-2
Machine vision-based weed spot spraying: a review and where next for sugarcane?
McCarthy, Cheryl, Rees, Steven and Baillie, Craig. 2010. "Machine vision-based weed spot spraying: a review and where next for sugarcane?" Bruce, R. C. (ed.) 32nd Annual Conference of the Australian Society of Sugar Cane Technologists (ASSCT 2010). Bundaberg, Australia 11 - 14 May 2010 Australia.
Apparatus and infield evaluations of a prototype machine vision system for cotton plant internode length measurement
McCarthy, Cheryl, Hancock, Nigel and Raine, Steven R.. 2010. "Apparatus and infield evaluations of a prototype machine vision system for cotton plant internode length measurement." Journal of Cotton Science. 14 (4), pp. 221-232.
Applied machine vision in agriculture at the NCEA
McCarthy, Cheryl and Billingsley, John. 2009. "Applied machine vision in agriculture at the NCEA." Banhazi, T. M. and Saunders, C. (ed.) SEAg 2009: Agricultural Technologies In a Changing Climate. Brisbane, Australia 13 - 16 Sep 2009 Brisbane, Australia.
Cattle liveweight estimation using machine vision assessment of objective body measurements: first results
McCarthy, C., Billingsley, J., Finch, N., Murray, P. and Gaughan, J.. 2010. "Cattle liveweight estimation using machine vision assessment of objective body measurements: first results." 28th Biennial Australian Society of Animal Production Conference. Armidale, Australia 11 - 15 Jul 2010 Brisbane, Australia.
Development of a prototype precision spot spray system using image analysis and plant identification technology
Rees, Steven, McCarthy, Cheryl, Artizzu, X. P. B., Baillie, Craig and Dunn, Mark. 2009. "Development of a prototype precision spot spray system using image analysis and plant identification technology." Banhazi, T. M. and Saunders, C. (ed.) SEAg 2009: Agricultural Technologies In a Changing Climate. Brisbane, Australia 13 - 16 Sep 2009 Brisbane, Australia.
A preliminary field evaluation of an automated vision-based plant geometry measurement system
McCarthy, Cheryl, Hancock, Nigel and Raine, Steven R.. 2007. "A preliminary field evaluation of an automated vision-based plant geometry measurement system." Prusinkiewicz, Przemyslaw, Hanan, Jim S. and Lane, Brendan (ed.) 5th International Workshop on Functional Structural Plant Models (FSPM07). Napier, New Zealand 04 - 09 Nov 2007
Automated machine vision sensing of plant structural parameters
McCarthy, C. L., Hancock, Nigel and Raine, Steven R.. 2007. "Automated machine vision sensing of plant structural parameters." Biological Sensorics 2007: Critical Technologies for Future Biosystems. Minneapolis, Australia 15 - 17 Jun 2007 St. Joseph, MI, United States.
On-the-go machine vision sensing of cotton plant geometric parameters: first results
McCarthy, Cheryl, Hancock, Nigel and Raine, Steven R.. 2006. "On-the-go machine vision sensing of cotton plant geometric parameters: first results." Billingsley, John (ed.) 13th Annual Conference on Mechatronics and Machine Vision in Practice. (M2VIP 2006). Toowoomba, Australia 05 - 07 Dec 2006 Berlin, Heidelberg.
A preliminary evaluation of machine vision sensing of cotton nodes for automated irrigation control
McCarthy, Cheryl, Hancock, Nigel and Raine, Steven R.. 2006. "A preliminary evaluation of machine vision sensing of cotton nodes for automated irrigation control." Irrigation Australia 2006: Irrigation Association of Australia National Conference and Exhibition. Brisbane, Australia 09 - 11 May 2006 Brisbane.