Forecasting Price of the Crude oil using LSTM based on RNN

Abstract

Oil demand is inelastic; therefore the rise in price is good news for producers because they will see an increase in their revenue. Oil importers, however, will experience increased costs of purchasing oil. Because oil is the largest traded commodity, the effects are quite significant. A rising oil price can even shift economic/political power from oil importers to oil exporters. The crude oil price movements are subject to diverse influencing factors. Our work mainly focuses on applying Recurrent Neural Networks to predict the Crude Oil Price. This decision helps common people to buy crude oil at the proper time. Time series analysis is the best option for this kind of prediction because we are using the previous history of crude oil prices to predict future price of the crude oil. So we would be implementing RNN (Recurrent Neural Network) with LSTM (Long Short Term Memory) to achieve the task. We will be experimenting with different types of models with varying number of epochs, look backs and other tuning methods.

Introduction

I. INTRODUCTION

Prediction of crude oil prices has been a wide topic for ages. People use their intuition and lot of techniques to guess the prices of crude oil. It takes a lot of knowledge about the crude oil to accurately predict it. Predicting the crude oil price is very significant in various economic, political and industrial areas, both for crude oil importer and exporter countries. Since the crude oil is important strategic resource around the globe; it has become the crucial commodity for the world’s economy. Thus, prediction of prices of crude oil has always been considered as a very exciting and challenging task which drew the curiosity of professionals, researchers and organizations all over the world.

Moreover, crude oil volatility has a critical impact on macroeconomic parameters such as such as inflation, unemployment, exchange rate, economic growth of countries whose economy rely heavily on crude oil export or import. Thus, crude oil price prediction can help governments of countries of the world in economic policymaking and make quick and operative economic decisions to hedge against probable risk in these economic parameters. Therefore, forecasting of crude oil prices is quite useful and is also the objective of this paper. In this, we have used LSTM based recurrent neural networks for the purpose of crude oil price prediction.

Recurrent neural networks (RNN) have been proved to be one of the most powerful models for processing time-series based sequential data. LSTM is one of the most successful RNN architectures. LSTM introduces the memory cell, a unit of computation that replaces traditional artificial neurons in the hidden layer of the network. With these memory cells, networks are able to effectively associate memories and input remote in time, hence suit to grasp the structure of data dynamically over time with high prediction capacity.

Recurrent neural network are a type of Neural Network where the output from previous step are fed as input to the current step. In traditional neural networks, all the inputs and outputs are independent of each other, but in cases like when it is required to predict the next word of a sentence, the previous words are required and hence there is a need to remember the previous words. Thus RNN came into existence, which solved this issue with the help of a Hidden Layer. The main and most important feature of RNN is Hidden state, which remembers some information about a sequence.

LSTMs have chain like structure with the repeating module having a different structure. There are four neural network layers which are interacting to each other in a special way. The key to LSTMs is the cell state, which is the horizontal line running through the top of the diagram. The cell state runs straight down the entire chain, with only some minor linear interactions. The information flows along it unchanged. The LSTM does have the ability to remove or add information to the cell state, carefully regulated by structures called gates. Gates are a way to optionally let information through. They are composed out of a sigmoid neural network layer and a point-wise multiplication operation. An LSTM has three of these gates, to protect and control the cell state.

A. Problem Statement

Nowadays, the increased oil prices worldwide are having a great impact on all economic activities. Over the years there has been a fluctuation in petroleum prices, and a close consideration of the demand and supply side effects that sparked these price changes shows there is high probability that these changes will continue in the outlook period and beyond. West African Monetary Agency (2008) concluded that increase in world oil prices have been shown to worsen fiscal deficit positions of oil importing countries like Ghana. For this reason, we believe that if the government can see ahead of monthly petroleum prices, our deficit would not be worsened.

B. Objectives

Based on literature survey conducted the objectives concluded are:

1. To prepare model to predict crude oil prices for next coming days, weeks months and quarters using LSTM based RNN concepts by using synthetic data.
2. To create a dashboard for various prediction activities

???????C. Proposed System

The proposed system uses a simple AI model and data collected for a period of time to use it as the AI knowledge and using that knowledge to predict the price of the crude oil as accurately as possible.

II. LITERATURE SURVEY

Crude oil forecasting is an important topic in financial and economic studies. Many studies have been performed to forecast the prices of crude oil.

The first research about forecasting oil market is proposed by Amano (1987). The author used a small-scale econometric model for oil market prediction. Huntington (1994) utilized a sophisticated econometric model for predicting oil price in the 1980s. In another work, Gulen (1998) applied counteraction analysis to predict the WTI crude oil price. Barone-adesi et al. (1998) suggested a semi-parametric approach based on the filtered historical simulation technique to forecast oil price. Kulkarni and Haidar [8] presented a model for forecasting crude oil spot price direction in the short- term, up to three days ahead based on multilayered feed-forward neural network. They tested the relation between crude oil future prices and spot price. They found the evidence that future prices of crude oil contain new information about oil spot price detection.

Hamdi and Aloui [9] performed a literature survey of numerous studies done on forecasting of crude oil price using artificial neural networks (ANN) until 2014. From the survey, they concluded that crude oil market is the most volatile commodity and forecasting oil price using nonlinear models such as ANN is the most suitable choice.

Abdullah and Zeng [10] proposed Hierarchical Conceptual (HC) and Artificial Neural Networks- Quantitative (ANN-Q) model based on machine learning and computational intelligence techniques to predict the monthly WTI crude oil price for every barrel in USD. The results obtained from their study validated the effectiveness of data selection process by the proposed model which successfully extracts a comprehensive list of key factors that cause the crude oil price market to be volatile.

Chen et al. [11] proposed a crude oil price forecasting model based on the deep learning model. They were able to analyze and model the crude oil price movement using the proposed deep learning model. They used the proposed model to capture the unknown complex non linear characteristics ofthe crude oil price movements. They evaluated the performance of the proposed model using the price data in the WTI crude oil markets.

Huntington (1994) utilized a sophisticated econometric model for predicting oil price in the 1980s.In another work, Gulen (1998) applied co-integration analysis to predict the WTI crude oil price.

Morana (2001) employed a semi-parametric approach investigated by Barone-adesi et al. (1998) to short-term forecast of Brent crude oil price. In another work, Tang and Hammoudeh (2002) utilized nonlinear regression to predict OPEC basket price.

Many Scientist and researcher have come across unique and variation of model for discovering and exploring and forecasting crude oil prices. Apprehensive study related to the forecasting and prediction of prices to be contingent on economic models. Also more study about intelligent and smart algorithm is taken into account. In the short-term it was Observed, variance was inflating by the events which were irregular. One of the feature of the imperfection of all the methodology was that the upcoming movement of oil price were derived from the prior data. A lot of machine learning strategy came into existence for oil price prediction which can be defined by ANN. Scientist have perceived that discussion of a topic done online convey point of view or outlook based on a subject matter, and thus topic models may have consequences on market prices.

But in most cases, machine learning techniques compared to the conventional techniques varies on permanent set of training data to coach a machine learning model followed by series of solicitation of model to the test set. One of the prime data source defines the important and useful information to be inserted in unstructured big data such as data n form of text which can employed for oil cast forecasting. In modern times , a number of studies have made essential offering for prediction in retail in the sphere of text mining. Convolutional neural network is a neural network based on deep learning concept. Its application has been mostly in recognition of image and speech and sentence modeling. The validation from the EMD technique proposes regarding the prime cause behind the fluctuation of price is midterm weighty incident. When out of the blue and unplanned events crash a market, it results from decomposition integration method which decline performance.

III. SYSTEM DESIGN

A. System Architecture

The figure 1 depicts the architectural diagram of proposed system. System design the main aim of this structure incorporated in study can fetch out data from economic news and propose this sets into prognosticate model. Major phases in formulated system include data collection and pre-processing, feature and factor selection and price appraisal and prediction. In the initial hand, news, financial and market data are gathered and processed. In Further aspect, unstructured documents are modified into structured extract by CNN classification.

Data retrieval and pre-processing in data retrieval, datasets can be fetched such of news data, black gold price data and market data. Dataset from news can be retrieved through headlines as it is easier to obtain and justifies in one line. Factors that affect the prediction are expert business, stock market and later business. Sentimental Analysis In this era of modernization, big data is also assisting through study of sentiment analysis which focuses on retrieving data through news and proposing prediction model. In this kind of analysis dictionary-based approach is accounted to gather the data regarding markets and essential factors affecting it. In case of trend prediction, the sentiment and prediction models are considered as variables.

Back Propagation Back-propagation is considered as an algorithm which can be used for the purpose of training feed forward neural networks for prognosticate learning model. This leads to the attainably use gradient methods to teach multi – layer networks, by modifying weights to minimum loss. The process fetch the inputs and outputs and modify its inner state that will be capable enough to calculate the output that will be very precise to the expected output. Back propagation can also be described as "backward propagation of errors." It is a natural function to teach artificial neural networks.

The forecasting can be done with Tensorflow which is followed by getting the data, generating features, generating ML model, training the ML model and testing and predicting with ml model. Movement of the price which can fall and rise can be termed as the outcome of CNN classification. Generally the activity of price can be specified by: Mt = {0 , pt < pt-1 }{1, pt >= pt- 1}

RNN (Recurrent Neural Network) RNN are not same as feed-forward networks. To predict things, it uses internal memory. It is capable of doing things which humans can’t do such as handwriting, speech recognition. Sequential information are added as input to these networks. We assume inputs are independent of each other which is a false assumption. We should know the previous words so that next words can be predicted without any trouble.

???????B. Use case Diagram

A Use Case Diagram is a lot of situations that reflect a client-frame relationship. A use case chart shows the entertainer-to-use relationship. Usage cases and on-screen characters are the two main elements of an usage case diagram. An on-screen character refers to a user or other person connected with the demonstrated process. A use case chart in figure is an out- of - the-box perspective that speaks to some activity each module will perform to complete an errand.

C. ??????????????Dataflow Diagram

Figure 3 shows the dataflow diagram of the proposed crude oil price forecasting system. Following are the steps:

1. Input Data: We need to input the last 10 years collected data as input.
2. Data pre-processing: This step includes

• Import the Libraries.
• Importing the dataset.
• Analyse the data
• Taking care of Missing Data
• Feature Scaling
• Data Visualization
• Splitting Data into Train and Test.

4. Training the RNN and LSTM classifier using pretrained Models: The model will be using the technique of Transfer Learning for training the model - “Feature Extraction from pre-trained model and training a classifier using extracted features”

4. Validation and Testing: Once the model is trained using the train dataset (the sample of data used to fit the model) then validated using validation dataset (The sample of data used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyper parameters.) and finally tested using the test dataset.

5. Oil Price Prediction: Finally we the oil price predicted as the output from our implemented system.

.

IV. IMPLEMENTATION

A. Methodology Of Proposed System

Our work is divided into 4 modules namely,

• Data collection
• Data preprocessing
• Model building
• Application building

1. Data Collection

Deep Learning depends heavily on data, without data, a machine can't learn. It is the most crucial aspect that makes neural network training possible. In Deep Learning projects, we need a training data set. It is the actual data set used to train the model for performing various actions.

2. Data Preprocessing

Data Pre-processing includes the following main tasks

• Import The Libraries

The first step is usually importing the libraries that will be needed in the program. The required libraries to be imported to Python script are:

Numpy: It is an open-source numerical Python library. It contains a multi-dimensional array and matrix data structures. It can be used to perform mathematical operations on arrays such as trigonometric, statistical, and algebraic routines.

Pandas- It is a fast, powerful, flexible, and easy to use open-source data analysis and manipulation tool, built on top of the Python programming language.

Matplotlib- Visualization with python. It is a comprehensive library for creating static, animated, and interactive visualizations in Python.

• Data Visualization

Data visualization is where a given data set is presented in a graphical format. It helps the detection of patterns, trends, and correlations that might go undetected in text-based data. Understanding your data and the relationship presents within it is just as important as any algorithm used to train your machine learning model. In fact, even the most sophisticated machine learning models will perform poorly on data that wasn’t visualized and understood properly. To visualize the dataset we need libraries called Matplotlib and Seaborn. The Matplotlib library is a Python 2D plotting library that allows you to generate plots, scatter plots, histograms, bar charts, etc.

• Splitting Data Into Train And Test

When you are working on a model and you want to train it, you have a dataset. But after training, we have to test the model on some test dataset. For this, you will need a dataset that is different from the training set you used earlier. But it might not always be possible to have so much data during the development phase. In such cases, the solution is to split the dataset into two sets, one for training and the other for testing. But the question is, how do you split the data? For time-series data, the sequence of values is important. A simple method that we can use is to split the ordered dataset into train and test datasets. The code below calculates the index of the split point and separates the data into the training datasets with 65% of the observations that we can use to train our model, leaving the remaining 30% for testing the model.

3. Model Building

• Model Building Includes:
• Import the model building Libraries
• Initializing the model
• Adding LSTM Layers
• Adding Output Layer
• Configure the Learning Process
• Training the model
• Model Evaluation
• Save the Model
• Test the Model
• Adding LSTM Layers

Note for the LSTM layer, units is the number of LSTM neurons in the layer. 50 neurons will give the model high dimensionality, enough to capture the upwards and downward trends. Return sequences is True as we need to add another LSTM layer after the current one. Input shape corresponds to the number of time stamps and the number of indicators.

• Adding Output Layers

The dense layer is a deeply connected neural network layer. It is the most common and frequently used layer. Finally, add the output layer. The output dimension is 1 since we are predicting 1 price each time. Understanding the model is a very important phase to properly use it for training and prediction purposes. Keras provides a simple method, summary to get the full information about the model and its layers.

• Train The Model

Now, let us train our model. RNN weights are updated every 64 stock prices with a batch size of 64. Try more batches and epochs if the loss of the model is not converging.

Arguments:

Epochs: an integer and number of epochs we want to train our model for.

Validation data can be either: an inputs and targets list, a generator. An inputs, targets, and sample_weights list which can be used to evaluate the loss and metrics for any model after any epoch has ended

• Model Evaluation

Finally, we need to check to see how well our model is performing on the test data.

Regression Evaluation Metrics:

1.    Mean Squared Error (MSE):

MSE or Mean Squared Error is one of the most preferred metrics for regression problems. It is simply the average of the squared difference between the target value and the value predicted by the regression model.

 As it squares the differences, it penalizes even a small error which leads to over-estimation of how bad the model is. It is preferred more than other metrics because it is differentiable and hence can be optimized better.

2.    Root Mean Square Error (RMSE):

RMSE is the square root of the averaged squared difference between the target value and the value predicted by the model. It is preferred more in some cases because the errors are first squared before averaging which poses a high penalty on large errors. This implies that RMSE is useful when large errors are undesired.

• Test the Model

Finally, we can generate predictions using the model for both the train and test to visualize the model.

We must shift the predictions so that they align on the x-axis with the original dataset. Once prepared, the data is plotted, showing the original dataset in blue, the predictions for the training dataset in green, the predictions on the unseen test dataset in orange.

• Prediction for the next 10 Days

Now let us predict the price of crude oil for the next 10 days. As the length of the test data is 2876, we are taking the previous 10 days input i.e., from index 2866 -2876 to predict 2867th day output. For predicting the next 10 days crude oil prices we consider n steps=10 We create the input for prediction, index starting from the date 10 days before the first date in the test dataset. Then, reshape the inputs to have only 1 column and predict using model predict predefined function.

???????B. Algorithm

1. Recurrent Neural Network (RNN)

Recurrent neural network are a type of Neural Network where the output from previous step are fed as input to the current step. In traditional neural networks, all the inputs and outputs are independent of each other, but in cases like when it is required to predict the next word of a sentence, the previous words are required and hence there is a need to remember the previous words.

In Recurrent Neural networks, the information cycles through a loop to the middle hidden layer. The input layer ‘x’ takes in the input to the neural network and processes it and passes it onto the middle layer. The middle layer ‘h’ can consist of multiple hidden layers, each with its own activation functions and weights and biases. If you have a neural network where the various parameters of different hidden layers are not affected by the previous layer, the neural network does not have memory, then you can use a recurrent neural network. The Recurrent Neural Network will standardize the different activation functions and weights and biases so that each hidden layer has the same parameters. Then, instead of creating multiple hidden layers, it will create one and loop over it as many times as required. Figure 5 shows the working of RNN.

 2. Long short-term memory (LSTM)

LSTMs have chain like structure with the repeating module having a different structure. There are four neural network layers which are interacting to each other in a special way. Initially LSTM decides what information we’re going to throw away from the cell state. This decision is made by a sigmoid layer called the “forget gate layer”.

Next it decides what new information we’re going to store in the cell state as shown in fig 7. This has two parts. First, a sigmoid layer called the “input gate layer” decides which values are to be updated. Next, a tanh layer creates a vector of new candidate values, C?t, that could be added to the state.

Then it combines these two to create an update to the state. The old cell state, Ct−1, updates into the new cell state Ct. The old state is multiplied by ft, forgetting the things which were decided to forget earlier.Then we add it xC?t. This is the new candidate values, scaled by how much it is decided to update each state value.

This output will be based on the cell state, but will be a filtered version. First, a sigmoid layer runs which decides what parts of the cell state will be output. Then, the cell state through tanh (to push the values to be between −1 and 1) and multiply it by the output of the sigmoid gate to get desired output.

???????

Conclusion

This paper has clears that an LSTM network is better than other traditional neural network for forecasting prices as it aims in using back propagation model. Traditional neural network such as CNN on the other hand predicts the next outgoing but doesn’t necessarily save the previous data or connection which is based on feed-forwarding, in the sense the previous data is not necessary to predict the future data. LSTM focuses on storing the previous data and prediction which is rather encouraging and more approximate. The outcome derived are relatively encouraging. The results show that large look ups do not necessarily improve the accuracy of the predictions of crude oil prices. Hence it can be concluded, the model with single LSTM model is definitely the most accurate.

References

[1] Mohammad Reza Mahdiani and Ehsan Khamehchi, “A modified neural network model for predicting the crude oil price”, Intellectual Economics, vol. 10, no. 2, pp. 71-77, Aug. 2016. [2] Manel Hamdi and Chaker Aloui, \"Forecasting Crude Oil Price Using Artificial NeuralNetworks: A Literature Survey,\" Economics Bulletin, AccessEcon, vol. 35, no. 2, pp. 1339-1359, 2015. [3] [3] Yu Runfang, Du Jiangze and Liu Xiaotao, “Improved Forecast Ability of Oil Market Volatility Based on combined Markov Switching and GARCH-class Model, Procedia Computer Science, vol. 122, pp. 415-422, 2017. [4] K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink and J. Schmidhuber, \"LSTM: A Search Space Odyssey,\" IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 10, pp. 2222-2232,Oct. 2017. [5] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, Nov. 1997. [6] Jammazi, R., Aloui, C.: Crude oil price forecasting: experimental evidence from wavelet decomposition and neural network modeling. Energy Econ. 34(3), 828–841 (2012) [7] S. Moshiri, and F. Foroutan, “Forecasting nonlinear crude oil futures prices,” The Energy Journal vol. 27, pp. 81-95, 2005. [8] Siddhi Vinayak Kulkarni and Imad Haidar, Forecasting Model for Crude Oil Price Using Artificial Neural Networks and Commodity Futures Prices. International Journal of Computer Science and Information Security, vol. 2, no.1, June 2009. [9] Hamdi and Aloui, \"Machine learning approach for crude oil price prediction with Artificial Neural Networks-Quantitative (ANN-Q) model,\" The 2010 International JointConference on Neural Networks (IJCNN), Barcelona, pp. 1-8, 2010. [10] Abdullah and Zeng.: Exploring the core factors and its dynamic effects on oil price: An application on path analysis and BVAR-TVP model. Energy Policy 39(12), 8022–8036 (2011) [11] Chen et al.: Forecasting the crude oil spot price by wavelet neural networks using OECD petroleum inventory levels. New Math. Nat. Comput. 07(02), 281–297 (2011)

Copyright

Copyright © 2022 Dr. Nirmala C R, Abhishek Kumar, Ajay K, Rangaswamy V R, Shivkumar S. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.