Deep Learning Models Used for Crop Analysis: A Review

Abstract

The development and improvement to map agricultural land cover are major challenges for researchers. For the sustainable development of agronomics spatial information about agricultural practices plays a vital role. Remote sensing satellite imagery is a valuable aid in providing and understanding this spatial distribution of agricultural practices. The aim of this paper is to provide a better understanding of capabilities of satellite images for agricultural land cover mapping through the use of deep learning techniques. The global coverage , rich spectral and spatial information and repetitive nature of remote sensing(RS) data have made them effective tools for mapping crop extents and yield prediction. This paper explores wide ranging review of research papers and articles on deep learning algorithms for image processing and predictions in the field of agriculture. The DL algorithms has attained remarkable success in different fields of RS and its use in crop monitoring. This review systematically identified 40 research papers from peer reviewed scientific publications related to sensors, platforms, input features, training data, spatial distribution of study sites. This article provides a concise summary of major DL algorithms, including concepts, limitations, implementation, to help researchers in agriculture to gain a holistic picture of major DL techniques quickly.

Introduction

I. INTRODUCTION

Agriculture indeed plays a crucial role in the global economy, especially as the world's population continues to grow. The increasing demands on the agriculture sector necessitate advancements in agricultural technology and modern agricultural practices. These new scientific research areas focus on increasing agricultural productivity while minimizing environmental impact through data-intensive methods.
Modern agricultural processes rely on data produced by various sensors to understand operating conditions such as climatic conditions, soil quality, and the interaction of dynamic crops. This data enables more accurate and faster decision-making, ultimately leading to improved efficiency and sustainability in agriculture.

Crop maps are useful for precision agriculture, the monitoring of farming activities, the preparation of crop statistics and the study of the impact of environmental factors on crops. Data captured by satellites, airplane or unmanned aerial vehicles (UAVs) provide a comprehensive snapshot of our environment. Many researchers have used RS data for crop monitoring, including crop-type classification and yield prediction due to their global coverage, repetitive nature, multispectral information for monitoring crops.

For crop mapping, pixel-based or object-based supervised and unsupervised classification methods have been primarily used over the years. Recently, machine learning (ML) algorithms, such as random forest (RF), decision tree (DT) and support vector machines (SVM), have also been successfully applied for crop mapping and crop-yield estimation.The growing demand for efficient and sustainable agricultural practices has led to the adoption of deep learning algorithms in agriculture for image processing and predictions.

Deep learning is a ML method that can not only map features onto outputs but also learns appropriate features itself, thereby avoiding the need for feature engineering. Deep learning is a reapplication of neural networks, in which multiple layers of neural networks are used for predictions based on available data. The ability of DL models to extract features automatically at different levels of abstraction from RS data to make predictions without the need to simulate complex relationships makes them valuable tools for crop monitoring. This paper comprehensively reviews that used RS data and DL techniques for critical crop-monitoring applications, namely, crop mapping. Crop mapping mainly refers to the classification problem.

II. REVIEW OF LITERATURE

The selection process for the review articles focused on ensuring the inclusion of studies that met specific criteria related to remote sensing imagery and crop identification or classification. A systematic search was conducted on topics such as deep learning (DL) and remote sensing (RS) based crop classification, yield prediction, and related areas. The search criteria included terms like 'Deep Learning' AND 'Remote Sensing', as well as keywords related to crop analysis using RS data, crop mapping, and similar topics. This approach aimed to gather relevant literature that addresses the intersection of deep learning and remote sensing in the context of crop analysis and classification.

The search for literature was conducted across reputable electronic databases including IEEE, Springer, Scopus, Web-of-Science, Elsevier, and Taylor & Francis Publishers. These databases are widely recognized for hosting high-quality publications and are considered authoritative sources in the academic community. By searching these databases, the review aimed to ensure the inclusion of studies that adhere to rigorous standards of research and publication.The emphasis on advanced computational techniques for extracting meaningful information from imaging data reflects the importance of technological advancements in enhancing our understanding of crop patterns and characteristics.

III. OVERVIEW OF DEEP LEARNING ALGORITHMS

Deep learning is a machine learning method inspired by the structure of the human brain, involving the training of neural networks with multiple layers. Machine learning, enables computers to perform tasks by learning from data without explicit programming. It is particularly valuable when relationships between variables cannot be efficiently described using traditional linear models. In deep learning, multiple layers learn data representation at different abstraction levels, allowing for the learning of complex functions with sufficient data and layers representing features at various levels of abstraction. Convolutional Neural Networks (CNNs), also known as ConvNets, have become the preferred deep neural network model in Computer Vision applications.

Convolutional neural networks (CNNs) consist of multiple layers of artificial neurons and are commonly used in computer vision tasks. These networks utilize filters, such as convolution and pooling layers, to extract features from input images. Each layer in a CNN highlights different features, creating hierarchical representations of the data. The convolution layer acts as a feature extractor, while the pooling layer reduces dimensionality and helps prevent overfitting. Fully connected layers, similar to biological neurons, calculate weighted sums of inputs to produce activation values. In a ConvNet, each layer generates multiple activation functions when an image is inputted, which are then passed to subsequent layers for further processing.

Recurrent Neural Networks (RNNs) are a specialized type of neural network designed to effectively model and predict sequential data. Unlike traditional feedforward networks, RNNs have the unique ability to capture temporal information, making them well-suited for tasks involving sequences. In standard neural networks, each input and output is treated as independent, but in scenarios where understanding the context of previous inputs is essential, such as predicting the next word in a sentence, RNNs shine. By incorporating a Hidden Layer, RNNs can retain and utilize information from prior inputs, enabling them to remember sequential patterns. The Hidden state within an RNN stores crucial details about a sequence, and the network's Memory component ensures that relevant information is preserved throughout the computation process. RNNs apply the same weights and biases to each input, ensuring consistent processing across all inputs and hidden layers.

The introduction of the Gated Recurrent Unit (GRU) in 2014 by Cho et al. presented a simpler alternative to the well-known Long Short-Term Memory (LSTM) networks. GRU, similar to LSTM, is a type of recurrent neural network (RNN) designed to handle sequential data like text, speech, and time-series data. The concept behind the GRU revolves around the use of gating mechanisms, which allow for selective updates to the network's hidden state at every time step. These gating mechanisms play a vital role in regulating the flow of information into and out of the network. In the case of GRU, there are two gating mechanisms: the reset gate and the update gate. The reset gate determines the degree to which the previous hidden state should be forgotten, aiding in the management of long-term dependencies. On the other hand, the update gate determines the extent to which the new input should influence the updated hidden state. The final output of the GRU is then calculated based on this updated hidden state.

Multilayer Perceptron’s (MLPs) are fundamental to deep learning technology, as they are a type of feed-forward neural network with multiple layers of perceptron’s. These perceptron’s contain various activation functions and are structured with connected input and output layers of equal number, with a hidden layer in between. MLPs are commonly utilized in developing image and speech recognition systems, as well as translation software. The operation of MLPs involves inputting data into the input layer, where neurons form connections that pass in a single direction. The weights of the input data are determined between the hidden layer and the input layer. Activation functions are used in MLPs to identify which nodes are activated. MLPs are primarily employed in training models to understand the correlations between layers in order to achieve the desired output from a given dataset.

IV. ANALYSIS OF THE LITERATURE

 A full-text read was conducted on the 40 articles that were identified. The articles were analysed to determine and explore their essential aspects such as the architecture of the DL, and its frameworks, RS data, training data, site and scale, assessment measures and performance and findings.

A. Sensors and Platforms Used

Satellite, aerial, and UAV sensors have been utilized to collect remote sensing data for crop mapping and yield prediction. Many crop-mapping studies have relied on satellite and aerial imagery to validate the effectiveness of their models. Satellite imagery is particularly advantageous due to its easy accessibility, as satellites regularly capture data and providers handle initial pre-processing tasks. This accessibility allows users to concentrate on application development rather than data pre-processing. Additionally, the remote sensing data mentioned are freely available, including through platforms like the Google Earth Engine, making data management and pre-processing more accessible for researchers in the field of agriculture.

B. Input Features

In crop-mapping studies utilizing deep learning architectures, various types of data are commonly used as input features, including optical data (RGB), multispectral data, radar data, thermal data, or a combination of these data sources. Some studies incorporate time-series enhanced vegetation index and normalized difference vegetation index derived from remote sensing data into their crop-mapping models. While computer-vision convolutional neural network (CNN) models are traditionally designed for three-channel RGB images, when transferring these models to remote sensing applications, the data must be formatted in a three-channel RGB format, limiting the use of additional multispectral bands. Multi-temporal data, crucial for distinguishing between crop types and accurately estimating yield, provide information on various crop growth stages.

C. Architecture

The deep-learning applications for crop mapping and yield prediction predominantly utilize architectures such as CNN, RNN, DNN, AEs, Transformer, and hybrid models. Among these, CNN is the most popular architecture, accounting for approximately 58% of the reviewed studies. CNN is particularly well-suited for array data like remote sensing data. For crop mapping, early approaches involved using CNN for feature extraction and scene classification. RNN models, on the other hand, were preferred for yield prediction, with over 40% of the studies utilizing RNNs. RNNs, especially LSTM, are effective in learning temporal characteristics for crop mapping and yield estimation from multitemporal images. Hybrid models that combine multiple architectures are also employed to learn spatial, spectral, and temporal features for enhanced decision-making. These hybrid models merge features from different networks or use the output of one architecture as input for another, enabling the joint modeling of spatial context and temporal information from multitemporal images.

D. Frameworks

Deep-learning frameworks are essential software libraries designed to facilitate the implementation of deep learning models. These frameworks come with pre-built structures that make it easier and more accessible to deploy deep learning architectures. Some of the most popular deep-learning frameworks include Caffe, Theano, TensorFlow, PyTorch, CNTK, and MatConvNet, which are known for their convolutional architectures that enable fast feature embedding. These frameworks are equipped with robust GPU backends that enable the training of networks with billions of parameters.
Among these frameworks, TensorFlow stands out as one of the most widely used frameworks for crop mapping and yield prediction using deep learning. Developed by researchers from the Google Brain Team, TensorFlow is a machine learning and deep neural network framework that supports multiple GPUs and CPUs. It is written in Python and also offers interfaces in R and JavaScript for broader usability.
Keras, on the other hand, is a high-level neural network API written in Python that runs on top of TensorFlow or Theano. Keras APIs are known for their intuitive and straightforward nature, leading to rapid adoption. TensorFlow has integrated Keras, providing users with a versatile library that combines the power of TensorFlow with the simplicity of Keras' interface.

Facebook’s AI-research laboratory developed PyTorch, it has gained a user community in recent years. Caffe is written in C++ with a Python interface and is also popular in computer vision because it incorporates various CNN frameworks and datasets. Deep neural networks are also built in Scikit-learn, a ML library. 

E. Crop Type

In crop-yield prediction studies, deep learning (DL) techniques were predominantly applied to corn and soybeans. While most studies focused on predicting the yield of a single crop, some also attempted to predict the yield of multiple crops without differentiation. In terms of crop mapping, the majority of studies detected multiple crops, with rice being the most commonly mapped single crop. The prevalence of rice as a staple food crop, along with its distinct phenological characteristics reflected in sensor data, likely contribute to its high rate of detection in crop mapping studies.

F. Training Data

The accuracy and generalization ability of a deep learning model are heavily influenced by the quality and quantity of the training data used. Insufficient training data can lead to over fitting and impact the model's prediction accuracy. In the context of crop mapping, traditional methods involved collecting crop-type labels through labour-intensive field visits. Following field surveys, the cropland-data layer (CDL) served as a primary training source for crop-classification models. Additionally, visual image interpretation of higher-resolution images was another method used for training data. These approaches highlight the importance of obtaining high-quality and diverse training data to enhance the performance of deep learning models in agricultural applications

G. Scale of the Output

The scale of output in crop mapping studies is directly influenced by the resolution of the input and target data. Typically, each pixel or group of pixels is assigned a crop class, with the precision of field boundaries and generalization being dependent on the spatial resolution of remote sensing data. Around 70% of yield prediction studies are conducted at the county level, utilizing county/district-crop-yield statistics. The remaining studies are field-level, using data collected directly from farmers and harvesters. The accuracy of yield predictions is enhanced when precise yield data is available at the appropriate scale. It is worth noting that the platforms used and the scales of the studies are often correlated, reflecting the importance of matching data resolution with the intended analysis scale.

V. CHALLENGES AND FUTURE DIRECTIONS

Deep learning algorithms have shown remarkable success in agriculture, several challenges and limitations still exist. This section discusses challenges related to data scarcity, interpretability, scalability, and computational requirements. Additionally, it outlines potential future directions for improving deep learning algorithms in agriculture, including the integration of multimodal data, addressing domain shift, and increasing explainability.

Data: Data is the most fundamental requirement to build the deep learning models. Many researchers faced the challenges regarding data. From this survey it is observed that, many researchers use data source sites like Kaggel, Meandly, IEEE Data port etc. to get the data to build models.

One has to apply the different pre-processing techniques to make the data suitable for training, testing, and validation testing the model. This might be time consuming process such as Dimensionality problem, Deep learning limitations, Mixed pixel classification etc.

Conclusion

This review paper provides a comprehensive overview of the applications of deep learning algorithms in agriculture for image processing and predictions. Deep learning, particularly convolutional neural networks (CNNs) and recurrent neural networks (RNNs), has demonstrated significant potential in addressing critical challenges faced by the agricultural industry. The review provided an overview of important observations regarding the employed platforms, sensors, input features, architectures, frameworks, training data, spatial distributions of study sites, output scales, assessment criteria and performances. The DL provides a promising solution for crop mapping and yield estimation. The deep learning algorithms in agriculture has paved the way for more accurate and efficient decision-making processes, to improve crop productivity, reduced resource consumption, and enhanced sustainability. The reviewed studies and case studies have showcased the capabilities of deep learning models in handling diverse agricultural data, including images, weather data, and satellite imagery. In conclusion, deep learning algorithms have proven their potential to revolutionize agriculture by enabling accurate image processing and predictions.

References

[1] Zhong L, Hu L, Zhou H (2019) Deep learning based multi-temporal crop classification. Remote Sens Environ 221:430–443. [2] Pallab Bharman a*, Sabbir Ahmad Saad a , Sajib Khan a , Israt Jahan a , Milon Ray a and Milon Biswas ,”Deep Learning in Agriculture: A Review” ,Asian Journal of Research in Computer Science 13(2): 28-47, 2022; ISSN: 2581-8260 [3] Razmjooy, N.; Estrela, V.V. Applications of Image Processing and Soft Computing Systems in Agriculture; IGI Global: Hershey, PA, USA, 2019. [4] Feng, S.; Zhao, J.; Liu, T.; Zhang, H.; Zhang, Z.; Guo, X. Crop Type Identification and Mapping Using Machine Learning Algorithms and Sentinel-2 Time Series Data. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2019, 12, 3295–3306 [5] LeCun, Y.; Bengio, Y.; Hinton, G. Deep Learning. Nature 2015, 521, 436–444. [6] Ma, L.; Liu, Y.; Zhang, X.; Ye, Y.; Yin, G.; Johnson, B.A. Deep Learning in Remote Sensing Applications: A Meta-Analysis and Review. ISPRS J. Photogramm. Remote Sens. 2019, 152, 166–177. [7] Khaki S and Wang L (2019) Crop Yield Prediction Using Deep Neural Networks. Front. Plant Sci. 10: 621.doi: 10.3389/fpls.2019.00621 [8] Zhang, Qian, Yeqi Liu, Chuanyang Gong, Yingyi Chen, and Huihui Yu. 2020. \"Applications of Deep Learning for Dense Scenes Analysis in Agriculture: A Review\" Sensors 20, no. 5: 1520. https://doi.org/10.3390/s20051520 [9] Sarma, Kandarpa Kumar, et al. \"Learning Aided System for Agriculture Monitoring Designed Using Image Processing and IoT-CNN.\" IEEE Access 10 (2022): 41525-41536. [10] Zhu N Y, Liu X, Liu Z Q, Hu K, Wang Y K, Tan J L, et al. Deep learning for smart agriculture: Concepts, tools, applications, and opportunities. Int J Agric&BiolEng, 2018; 11(4): 32–44. [11] K Dokic et al 2020 IOP Conf. Ser.: Earth Environ. Sci. 614 012138 [12] DOI 10.1088/1755-1315/614/1/012138 [13] N.M. Karie, V.R. Kebande, H.S. Venter Diverging deep learning cognitive computing techniques into cyber forensics Forensic SciInt, 1 (2019), pp. 61-67 vol. 1, pp. 61-67doi.org/10.1016/j.fsisyn.2019.03.006 [14] K.G. Liakos, P. Busato, D. Moshou, S. Pearson, D. Bochtis Machine learning in agriculture: a review Sensors (Switzerland), 18 (8) (2018), pp. 1-29 10.3390/s18082674 [15] A. Sharma, A. Jain, P. Gupta, V. Chowdary Machine learning applications for precision agriculture: a comprehensive review IEEE Access, 9 (2021), pp. 4843-4873 10.1109/ACCESS.2020.3048415 [16] D. Sivakumar, K. SuriyaKrishnaan, P. Akshaya, G.V. Anuja, G.T. Devadharshini [17] Computerized growth analysis of seeds using deep learning method [18] Int J Recent TechnolEng (2019) Volume-7Issue-6S5 [19] S. Zhu, L. Zhou, P. Gao, Y. Bao, Y. He, L. Feng Near-infrared hyperspectral imaging combined with deep learning to identify cotton seed varieties Molecules, 24 (2019), p. 3268 10.3390/molecules24183268 [20] L.C. Uzal, et al. Seed-per-pod estimation for plant breeding using deep learning Comput Electron Agricul, 150 (2018), pp. 196-204 [21] Ulzii-Orshikh Dorj et al.An yield estimation in citrus orchards via fruit detection and counting using image processingComput. Electron. Agric.(2017)https://doi.org/10.1016/j.compag.2017.05.019 [22] D. Nkemelu, D. Omeiza, and N. Lubalo,”Deep convolutional neural network for plant seedlings classification”, 2018, arXiv:1811.08404v1 . [23] M.H. Saleem, J. Potgieter, K.M. Arif Plant disease detection and classification by deep learning Plants, 8 (2019), p. 468 [24] E. Jr Piedad, J.I. Larada, G.J. Pojas, L. Vithalie, V. Ferrer Postharvest classification of banana (Musa acuminata) using tier-based machine learning Postharvest BiolTechnol, 145 (2018), pp. 93-100 [25] Microsoft, “What is automated machine learning (AutoML)?” https://docs.microsoft.com/en-US/azure/machine-learning/concept-automated-ml, (Accessed: July 2021). [26] Bouguettaya, A., Zarzour, H., Kechida, A. et al. Deep learning techniques to classify agricultural crops through UAV imagery: a review. Neural Comput & Applic 34, 9511–9536 (2022). [27] Bah MD, Hafiane A, Canals R (2019) Crownet: deep network for crop row detection in uav images. IEEE Access 8:5189–5200. [28] Bayraktar E, Basarkan ME, Celebi N (2020) A low-cost uav framework towards ornamental plant detection and counting in the wild. ISPRS J Photogramm Remote Sens 167:1–11. [29] Chamorro Martinez JA, Cué La Rosa LE, Feitosa RQ et al (2021) Fully convolutional recurrent networks for multidate crop recognition from multitemporal image sequences. ISPRS J Photogramm Remote Sens 171:188–201 [30] Der Yang M, Tseng HH, Hsu YC, et al (2020) Real-time crop classification using edge computing and deep learning. In: 2020 IEEE 17th annual consumer communications & networking conference (CCNC), IEEE, pp 1–4 [31] Ranganathan Krishnamoorthy; Ranganathan Thiagarajan; Shanmugam Padmapriya; Indiran Mohan; Sundaram Arun; Thangaraju Dineshkumar, \"Applications of Machine Learning and Deep Learning in Smart Agriculture,\" , IEEE, 2023, pp.371-395, doi: 10.1002/9781119861850.ch21 [32] Cynthia, Shamse & Hossain, Kazi& Hasan, Md&Asaduzzaman, Md& Das, Amit. (2019). Automated Detection of Plant Diseases Using Image Processing and Faster R-CNN Algorithm. 1-5. 10.1109/STI47673.2019.9068092. [33] Bal, Fatih&Kayaalp, Fatih. (2023). A Novel Deep Learning-Based Hybrid Method for the Determination of Productivity of Agricultural Products: Apple Case Study. IEEE Access. PP. 1-1. 10.1109/ACCESS.2023.3238570. [34] Meshram, Vishal &Patil, Kailas. (2021). FruitNet: Indian fruits image dataset with quality for machine learning applications. Data in Brief. 40. 107686. 10.1016/j.dib.2021.107686. [35] Koklu, Murat &Ünler?en, Muhammed &Ozkan, Ilker Ali & Aslan, Muhammet&Sabanci, Kadir. (2022). A CNN-SVM Study based on selected deep features for grapevine leaves classification. Measurement. 188. 1-10. 10.1016/j.measurement.2021.110425. [36] Behera, Santi &Rath, Amiya &Mahapatra, Abhijeet & Sethy, Prabira. (2020). Identification, classification & grading of fruits using machine learning & computer intelligence: a review. Journal of Ambient Intelligence and Humanized Computing. 10.1007/s12652-020-01865-8. [37] Duong-Trung N, Quach LD, Nguyen MH, et al (2019) A combination of transfer learning and deep learning for medicinal plant classification. In: Proceedings of the 2019 4th international conference on intelligent information technology. Association for computing machinery, New York, NY, USA, ICIIT ’19, p 83-90, [38] Bhosle K, Musande V (2020) Evaluation of cnn model by comparing with convolutional auto encoder and deep neural network for crop classification on hyperspectral imagery. Geocarto International 1–15 . [39] Fawakherji M, Potena C, Bloisi DD, et al (2019) Uav image based crop and weed distribution estimation on embedded gpu boards. In: International conference on computer analysis of images and patterns, Springer, pp 100–108, [40] R. Sujatha, J.M. Chatterjee, N.Z. Jhanjhi, S.N. Brohi Performance of deep learning vs machine learning in plant leaf disease detection MicroprocessMicrosyst, 80 (2021) [41] Adrian J, Sagan V, Maimaitijiang M (2021) Sentinel sar-optical fusion for crop type mapping using deep learning and google earth engine. ISPRS J Photogramm Remote Sens 175:215–235.

Copyright

Copyright © 2024 Nayana S. Ratnaparkhi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.