Automatic Fruit Plucking Machine Using Deep Learning

Abstract

The Automatic Fruit Plucking Machine project aims to develop an innovative automated system to revolutionize fruit harvesting. By integrating computer vision, robotic arm manipulation, and YOLO deep learning algorithms, the machine can accurately identify and pluck ripe fruits. It reduces labor costs, enhances harvest quality, and improves productivity. The machine\'s core is a trained machine learning algorithm that combines image processing and deep learning models for real-time fruit detection. Through iterative testing and refinement, the machine\'s performance continually improves. This project offers a sustainable and efficient solution for fruit producers to meet market demands and reduce manual labor reliance.

Introduction

I. INTRODUCTION

The agricultural industry plays a vital role in feeding the growing global population. Within this industry, fruit production has witnessed significant growth, driven by increasing consumer demand for fresh and nutritious fruits. However, fruit harvesting remains a labor-intensive and time-consuming process, often relying on manual labor to identify and pluck ripe fruits from trees or plants. This reliance on human labor poses several challenges, including high costs, labor shortages, and inefficiencies in the harvesting process. To overcome these challenges and improve efficiency in fruit harvesting, the concept of an Automatic Fruit Plucking Machine has emerged.

This innovative solution leverages advanced technologies such as computer vision, robotics, and machine learning to automate the fruit plucking process. The goal is to develop a machine capable of identifying and selectively harvesting ripe fruits while maintaining their quality and minimizing damage.

The Automatic Fruit Plucking Machine project aims to address the pressing need for an efficient and cost-effective solution to fruit harvesting. By combining the capabilities of computer vision and machine learning algorithms, the machine can accurately identify ripe fruits by analyzing their color, size, and shape. This technology eliminates the manual inspection process, reducing the time required for harvesting and increasing overall productivity. Furthermore, the project focuses on the design and integration of a robotic arm equipped with specially designed grippers that mimic the dexterity and delicacy of human hands. This robotic arm can pluck fruits gently, minimizing bruising and damage, thus enhancing the quality of the harvest. The use of automation also helps reduce labor costs and reliance on human labor, addressing labor shortages and increasing operational efficiency. The integration of machine learning algorithms into the Automatic Fruit Plucking Machine enables real-time decision-making based on fruit recognition. Through continuous learning and optimization, the machine becomes more accurate and efficient over time, ensuring consistent performance and improved harvest quality. Overall, the Automatic Fruit Plucking Machine project presents a transformative solution for fruit producers, addressing the challenges of labor costs, labor shortages, and inefficiencies in fruit harvesting. By automating the process, this technology offers increased productivity, reduced costs, and improved harvest quality, benefiting both farmers and consumers alike. With its potential to revolutionize the fruit industry, this project holds great promise for the future of fruit production.

II. LITERATURE REVIEW

1. An automated fruit harvesting robot by using deep learning [Onishi, Y., Yoshida, T., Kurita, H.]

This study [1] introduces an innovative approach to automate fruit harvesting using a robot arm. The method employs the Single Shot MultiBox Detector for accurate fruit detection and a stereo camera system to determine fruit positions in 3D. Inverse kinematics calculates joint angles, allowing the robot arm to move precisely to the target fruit. Harvesting is performed by twisting the hand axis. Promising results show over 90% successful fruit detection, with the robot arm harvesting a fruit in just 16 seconds. This research significantly advances automation and labor-saving techniques in fruit cultivation by combining computer vision, robotic manipulation, and inverse kinematics, potentially revolutionizing fruit harvesting, reducing labor needs, and improving overall efficiency.

2. Autonomous Fruit Picking Machine: A Robotic Apple Harvester [Baeten, J., Donné, K., Boedrij, S., Beckers, W., Claesen, E. (2008).]

The Autonomous Fruit Picking Machine (AFPM) represents a significant advancement in apple harvesting automation [2]. The project aimed to create a machine capable of autonomously harvesting apples. Two primary approaches to robotic apple harvesting were considered: bulk harvesting and apple by apple harvesting.

Bulk harvesting requires uniform fruit ripeness and specific tree conditions, while apple by apple harvesting allows for selective picking without these constraints. To implement the latter effectively, a non-damaging gripper is crucial, preserving apple quality and tree integrity.

Various gripper designs, including low-cost inflatable grippers, have been proposed in previous research. The vision system is another critical component, with this approach positioning the camera within the gripper for easier calibration and control. While navigation systems for orchard rows exist, the project focused on demonstrating the AFPM's feasibility and functionality rather than autonomous navigation.

The AFPM prototype, constructed using state-of-the-art components, includes an innovative gripper designed for apple harvesting and image-based control strategies. Field experiments were conducted to assess the machine's performance.

In summary, the Autonomous Fruit Picking Machine promises to revolutionize apple harvesting by automating the process and enhancing efficiency.

3. A Proposal for Automatic Fruit Harvesting by Combining a Low-Cost Stereovision Camera and a Robotic Arm [Font, Davinia & Pallejà, Tomàs & Tresanchez, Marcel & Runcan, David & Javier Moreno, Javier & Martinez, Dani & Teixido, Merce & Palacín, Jordi. (2014).]

The proposed paper [3] introduces an automatic fruit harvesting system that combines a low-cost stereo-vision camera and a robotic arm.

The stereo-vision camera is utilized to detect crucial information such as color, distance, and position of the fruit, while the robotic arm is responsible for mechanically plucking the fruits. The system is based on a prototype that includes a harvesting robot with a cylindrical shape and three degrees of freedom.

The robot approaches the target fruit from the side of the path. Notably, the system achieves lower power consumption by utilizing ARDUINO NANO, DC motors, and Motor Drivers. The system has been tested successfully in laboratory conditions with uniform illumination applied to the fruits.

As a future endeavor, the system will be further tested and enhanced in real-world outdoor farming conditions, and there is potential for individual development of both the robot and the moving platform.

4. Fruit Detachment and Classification method for Strawberry Harvesting Robot

In reference to [7], the paper "Fruit Detachment and On-line Classification for Harvesting Robots: A Focus on Ground-Grown Strawberries" presents an innovative approach to improve fruit harvesting efficiency, particularly for ground-grown strawberries. It utilizes the OHTA color space-based image segmentation algorithm for strawberry recognition and calculates their orientation using principal inertia axis.

The research introduces selective picking based on ripeness and shape classification, supported by a histogram matching method. Impressive experimental results, such as a 93% accuracy in strawberry stem detection and over 90% accuracy in ripeness and shape quality assessment, underscore the practicality and potential field application of this approach, making it a significant contribution to harvesting robotics.

The design of the Automatic Fruit Plucking Machine project encompasses several key aspects.

1. Robotic Arm: The project involves the design and integration of a robotic arm specifically tailored for fruit harvesting. The robotic arm is equipped with joints and grippers that mimic human hand movements, allowing for delicate and precise fruit plucking without causing damage. The design takes into consideration factors such as reach, flexibility, and strength to ensure efficient and effective fruit harvesting.
2. Computer Vision System: A stereo camera system of two ESP 32 cameras is incorporated into the design to capture images of the fruits. The design includes the positioning and calibration of the cameras to accurately detect the color, distance, and position of the fruits. The YOLO (you only look once) computer vision algorithm and Triangulation techniques are implemented to analyze the captured images and identify ripe fruits for harvesting.
3. Machine Learning Algorithms: The design includes the integration of YOLO v4(you only look once) algorithm to train the system to distinguish between ripe and unripe fruits based on visual cues. These algorithms enable real-time fruit recognition and decision-making, improving the accuracy and efficiency of the fruit harvesting process.
4. Microcontroller: The design incorporates a Arduino UNO (microcontroller) to coordinate the movements of the robotic arm based on the detected fruit positions. Inverse kinematics calculations are performed to determine the joint angles required for precise positioning and plucking. The control system ensures synchronized and accurate movements of the robotic arm to successfully harvest the fruits.

Overall, the design of the Automatic Fruit Plucking Machine combines the elements of a specialized robotic arm with rover, a computer vision system, machine learning algorithms, and a control system. This integrated design allows for efficient and gentle fruit harvesting by accurately identifying ripe fruits, controlling the robotic arm movements, and delicately plucking the fruits without causing damage.

C. Theory

The system in figure 2 aims to automate the process of detecting ripe lemons on trees using a stereo camera and utilizing a robotic arm with 4 degrees of freedom to pluck them. The methodology involves the use of YOLO v4 (You Only Look Once version 4) algorithm for lemon detection, 2D-to-3D conversion of lemon positions using triangulation, and inverse kinematics for positioning the robotic arm's end effector accurately for plucking.

1. Lemon Detection using deep learning

The YOLO v4 algorithm is employed for real-time object detection, specifically to detect ripe lemons in captured stereo camera images. A threshold of 0.9 is set to ensure accurate lemon detection, ensuring that only ripe lemons are considered for plucking.

The YOLO (You Only Look Once) algorithm stands out as an optimal choice for our automatic fruit plucking machine project due to its remarkable real-time object detection capabilities. Renowned for its speed, accuracy, and adaptability, YOLO swiftly processes images, crucial for swiftly identifying fruits of diverse sizes on trees. The multi-resolution detection feature ensures precise localization, a pivotal aspect for automating the fruit plucking process. Its efficiency, comprehensive documentation, and strong community support make it an ideal candidate for integration into our fruit plucking machine, augmenting its ability to locate and harvest ripe fruits efficiently and accurately.

V. ACKNOWLEDGMENT

We extend our heartfelt appreciation to Prof. Anil Kadu, our esteemed guide and mentor, for his invaluable support, guidance, and expertise throughout this project. His insights and encouragement significantly contributed to the success of our research. We are grateful for his mentorship and the opportunities provided to us during this academic endeavor.

We would also like to express our gratitude to all those who supported and guided us during this project. Your encouragement and assistance were invaluable, and we appreciate your contributions to our work.

Conclusion

In summary, the Automatic Fruit Plucking Machine represents a significant advancement in fruit harvesting automation. By integrating computer vision, robotic arm manipulation, and machine learning algorithms, the machine achieves accurate fruit detection and gentle harvesting. The machine\'s performance has been extensively tested, demonstrating high accuracy in fruit detection and recognition. The robotic arm\'s precise movements enable efficient and damage-free plucking, resulting in improved fruit quality and longer shelf life. The benefits of the Automatic Fruit Plucking Machine include reduced labor requirements, lower production costs, increased productivity, and improved efficiency for fruit producers. Future improvements may focus on refining machine learning algorithms, optimizing robotic arm control, and incorporating advanced sensors. Further field testing in real-world farming conditions will provide valuable insights, and ongoing research and development will address scalability and cost-effectiveness. Overall, the Automatic Fruit Plucking Machine offers a promising solution for automating fruit harvesting, revolutionizing the industry, and providing significant benefits to fruit producers globally.

References

[1] Onishi, Y., Yoshida, T., Kurita, H. et al. An automated fruit harvesting robot by using deep learning. Robomech J 6, 13 (2019). https://doi.org/10.1186/s40648-019-0141-2. [2] Baeten, J., Donné, K., Boedrij, S., Beckers, W., Claesen, E. (2008). Autonomous Fruit Picking Machine: A Robotic Apple Harvester. In: Laugier, C., Siegwart, R. (eds) Field and Service Robotics. Springer Tracts in Advanced Robotics, vol 42. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-75404-6_51 [3] Font, Davinia & Pallejà, Tomàs & Tresanchez, Marcel & Runcan, David & Javier Moreno, Javier & Martinez, Dani & Teixido, Merce & Palacín, Jordi. (2014). A Proposal for Automatic Fruit Harvesting by Combining a Low-Cost Stereovision Camera and a Robotic Arm. Sensors (Basel, Switzerland). 14. 11557-11579. 10.3390/s140711557. [4] Ashwini K, “Survey Paper on Fruit Picking Robots,” International Journal of Computer Science and Mobile Computing, IJCSMC, pg,96 - 101 Vol. 5, Issue. 1, January 2016 [5] Davinia Font, “A proposal for automatic fruit harvesting by combining a low-cost stereo-vision camera and a Robotic arm,” Mdpi journal, pg, 1- 23, Issue. 30, June 2014. [6] Chiu Y C, Yang P Y, Chen S. Development of the end-effector of a picking robot for greenhouse-grown tomatoes. Applied Engineering in Agriculture, 2013; 29(6): 1001–1009. [7] Guo F, Cao Q X, Masateru N. Fruit detachment and classification method for strawberry harvesting robot. International Journal of Advanced Robotic Systems, 2008; 5(1): 5662. doi: 10.5772/5662. [8] Font D, Pallejà T, Tresanchez M, Runcan D, Moreno J, Martínez D, Teixidó M, Palacín J. A proposal for automatic fruit harvesting by combining a low cost stereovision camera and a robotic arm. Sensors (Basel). 2014 Jun 30;14(7):11557-79. doi: 10.3390/s140711557. PMID: 24984059; PMCID: PMC4168444. [9] Yoshida, T., Onishi, Y., Kawahara, T. et al. Automated harvesting by a dual-arm fruit harvesting robot. Robomech J 9, 19 (2022). https://doi.org/10.1186/s40648-022-00233-9 [10] Smart healthcare system-a brain-like computing approach for analyzing the performance of detectron2 and PoseNet models for anomalous action detection in aged people with movement impairments - Scientific Figure on ResearchGate. Available from: https://www.researchgate.net/figure/Performance-comparison-of-various-object-detection-models_fig5_349960212 [accessed 19 Oct, 2023]

Copyright

Copyright © 2023 Omkar Patil, Shivanand Nemane, Meet Nathwani, Saurabh Patil. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.