A Survey on Stock Market Price Prediction System using Machine Learning Techniques

Abstract

Prediction of stock prices is one of the most researched topics and gathers interest from academia and the industry alike. In the finance world stock trading is one of the most important activities. Stock market prediction is an act of trying to determine the future value of a stock other financial instrument traded on a financial exchange. This paper explains the prediction of a stock using Machine Learning. The technical and fundamental or the time series analysis is used by the most of the stockbrokers while making the stock predictions. The programming language is used to predict the stock market using machine learning is Python. In this paper we propose a Machine Learning (ML) approach that will be trained from the available stocks data and gain intelligence and then uses the acquired knowledge for an accurate prediction. The paper focuses on the use of Linear Regression, Moving Average, K-Nearest Neighbours, Auto ARIMA, Prophet, and LSTM based Machine learning techniques to predict stock values. Factors considered are open, close, low, high and volume. The models are evaluated using standard strategic indicators: RMSE and MAPE. The low values of these two indicators show that the models are efficient in predicting stock closing price. We conducted comprehensive evaluations on frequently used machine learning models and conclude that our proposed solution outperforms due to the comprehensive feature engineering that we built. The system achieves overall high accuracy for stock market price prediction. This work contributes to the stock analysis research community both in the financial and technical domains.

Introduction

I. INTRODUCTION

A. Information of Stock

A stock market is a public market where you can buy and sell shares for publicly listed companies. The stocks, also known as equities, represent ownership in the company. The stock exchange is the mediator that allows the buying and selling of shares. We all have heard the word stock one way or the other. Particularly stock is related with the associates and companies which are commercialized and are to settling in the world of marketization. The other word used for stock is share which is prominently used in day-to-day life. People even term is as an investment plan and it's something people see as a long-term investment that secures and provides an abundant funds during the retirement age.

Market is unpredictable so are the resources and the factors that are taken to drive it off or on the set. It's never been on the same level and the pattern of the same is still unpredictable till the time. Some closeness and prediction method had been derived and approximates values and the rough figures are generated hoping for the best but all of the resource can’t be trusted and are still unpredictable in nature.

B. Stock Market Prediction

Stock market prediction and analysis are some of the most difficult jobs to complete. There are numerous causes for this, including market volatility and a variety of other dependent and independent variables that influence the value of a certain stock in the market. These variables make it extremely difficult for any stock market expert to anticipate the rise and fall of the market with great precision.

However, with the introduction of Machine Learning and its strong algorithms, the most recent market research and Stock Market Prediction advancements have begun to include such approaches in analysing stock market data.

The probable stock market prediction target can be the future stock price or the volatility of the prices or market trend. In the prediction there are two types like dummy and a real time prediction which is used in stock market prediction system. In Dummy prediction they have define some set of rules and predict the future price of shares by calculating the average price.  In the real time prediction compulsory used internet and saw current price of shares of the company.

C. Stock Analysis: Fundamental Analysis vs. Technical Analysis

There are two types to analyse stocks which investors perform before investing in a stock, first is the fundamental analysis, in this analysis investors look at the intrinsic value of stocks, and performance of the industry, economy, political climate etc. to decide that whether to invest or not. On the other hand, the technical analysis it is an evolution of stocks by the means of studying the statistics generated by market activity, such as past prices and volumes.

Table 1: Fundamental Analysis vs. Technical Analysis

Computational advances have led to introduction of machine learning techniques for the predictive systems in financial markets.  In this paper we are using a Machine Learning technique i.e., Linear Regression, Moving Average, K-Nearest Neighbours, Auto ARIMA, Prophet, and LSTM based Machine learning techniques to predict stock values and we are using Python language for programming.

II. LITERATURE SURVEY

"What other people think” has always been an important piece of information for most of us during the decision-making process. The Internet and the Web have now (among other things) made it possible to find out about the opinions and experiences of those in the vast pool of people that are neither our personal acquaintances nor well-known professional critics - that is, people we have never heard of. And conversely, more and more people are making their opinions available to strangers via the Internet. The interest that individual users show in online opinions about products and services, and the potential influence such opinions wield, is something that is driving force for this area of interest. And there are many challenges involved in this process which needs to be walked all over in order to attain proper outcomes out of them. In this survey we analysed basic methodology that usually happens in this process and measures that are to be taken to overcome the challenges being faced.

A. Stock Market Prediction Using Machine Learning

The research work done by V Kranthi Sai Reddy Student, ECM, Sreenidhi Institute of Science and Technology, Hyderabad, India. In the finance world stock trading is one of the most important activities. Stock market prediction is an act of trying to determine the future value of a stock other financial instrument traded on a financial exchange. This paper explains the prediction of a stock using Machine Learning. The technical and fundamental or the time series analysis is used by the most of the stockbrokers while making the stock predictions. The programming language is used to predict the stock market using machine learning is Python. In this paper we propose a Machine Learning (ML) approach that will be trained from the available stocks data and gain intelligence and then uses the acquired knowledge for an accurate prediction. In this context this study uses a machine learning technique called Support Vector Machine (SVM) to predict stock prices for the large and small capitalizations and in the three different markets, employing prices with both daily and up-to-the-minute frequencies.

B. Forecasting the Stock Market Index Using Artificial Intelligence Techniques

The research work done by Lufuno Ronald Marwala A dissertation submitted to the Faculty of Engineering and the Built Environment, University of the Witwatersrand, Johannesburg, in fulfilment of the requirements for the degree of Master of Science in Engineering. The weak form of Efficient Market hypothesis (EMH) states that it is impossible to forecast the future price of an asset based on the information contained in the historical prices of an asset. This means that the market behaves as a random walk and as a result makes forecasting impossible. Furthermore, financial forecasting is a difficult task due to the intrinsic complexity of the financial system. The objective of this work was to use artificial intelligence (AI) techniques to model and predict the future price of a stock market index. Three artificial intelligence techniques, namely, neural networks (NN), support vector machines and neuro-fuzzy systems are implemented in forecasting the future price of a stock market index based on its historical price information. Artificial intelligence techniques have the ability to take into consideration financial system complexities and they are used as financial time series forecasting tools. Two techniques are used to benchmark the AI techniques, namely, Autoregressive Moving Average (ARMA) which is linear modelling technique and random walk (RW) technique. The experimentation was performed on data obtained from the Johannesburg Stock Exchange. The data used was a series of past closing prices of the All-Share Index. The results showed that the three techniques have the ability to predict the future price of the Index with an acceptable accuracy. All three artificial intelligence techniques outperformed the linear model. However, the random walk method out performed all the other techniques. These techniques show an ability to predict the future price however, because of the transaction costs of trading in the market, it is not possible to show that the three techniques can disprove the weak form of market efficiency. The results show that the ranking of performances support vector machines, neuro-fuzzy systems, multilayer perceptron neural networks is dependent on the accuracy measure used.

C. Indian stock market prediction using artificial neural networks on tick data

The research work done by Dharmaraja Selvamuthu, Vineet Kumar and Abhishek Mishra Department of Mathematics, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India. A stock market is a platform for trading of a company’s stocks and derivatives at an agreed price. Supply and demand of shares drive the stock market. In any country stock market is one of the most emerging sectors. Nowadays, many people are indirectly or directly related to this sector. Therefore, it becomes essential to know about market trends. Thus, with the development of the stock market, people are interested in forecasting stock price. But, due to dynamic nature and liable to quick changes in stock price, prediction of the stock price becomes a challenging task. Stock m Prior work has proposed effective methods to learn event representations that can capture syntactic and semantic information over text corpus, demonstrating their effectiveness for downstream tasks such as script event prediction. On the other hand, events extracted from raw texts lacks of common-sense knowledge, such as the intents and emotions of the event participants, which are useful for distinguishing event pairs when there are only subtle differences in their surface realizations. To address this issue, this paper proposes to leverage external common-sense knowledge about the intent and sentiment of the event.

Experiments on three event-related tasks, i.e., event similarity, script event prediction and stock market prediction, show that our model obtains much better event embeddings for the tasks, achieving 78% improvements on hard similarity task, yielding more precise inferences on the subsequent events under given contexts, and better accuracies in predicting the volatilities of the stock market 1. Markets are mostly a nonparametric, non-linear, noisy and deterministic chaotic system (Ahangar et al. 2010). As the technology is increasing, stock traders are moving towards to use Intelligent Trading Systems rather than fundamental analysis for predicting prices of stocks, which helps them to take immediate investment decisions. One of the main aims of a trader is to predict the stock price such that he can sell it before its value decline, or buy the stock before the price rises. The efficient market hypothesis states that it is not possible to predict stock prices and that stock behaves in the random walk. It seems to be very difficult to replace the professionalism of an experienced trader for predicting the stock price. But because of the availability of a remarkable amount of data and technological advancements we can now formulate an appropriate algorithm for prediction whose results can increase the profits for traders or investment firms. Thus, the accuracy of an algorithm is directly proportional to gains made by using the algorithm.

D. The Stock Market and Investment

The research work done by Manh Ha Duong Boriss Siliverstovs. Investigating the relation between equity prices and aggregate investment in major European countries including France, Germany, Italy, the Netherlands and the United Kingdom. Increasing integration of European financial markets is likely to result in even stronger correlation between equity prices in different European countries.

This process can also lead to convergence in economic development across European countries if developments in stock markets influence real economic components, such as investment and consumption. Indeed, our vector autoregressive models suggest that the positive correlation between changes equity prices and investment is, in general, significant. Hence, monetary authorities should monitor reactions of share prices to monetary policy and their effects on the business cycle.

E. Automated Stock Price Prediction Using Machine Learning

The research work done by Mariam Moukalled Wassim El-Hajj Mohamad Jaber Computer Science Department American University of Beirut. Traditionally and in order to predict market movement, investors used to analyse the stock prices and stock indicators in addition to the news related to these stocks.

Hence, the importance of news on the stock price movement. Most of the previous work in this industry focused on either classifying the released market news as (positive, negative, neutral) and demonstrating their effect on the stock price or focused on the historical price movement and predicted their future movement.

In this work, we propose an automated trading system that integrates mathematical functions, machine learning, and other external factors such as news’ sentiments for the purpose of achieving better stock prediction accuracy and issuing profitable trades. Particularly, we aim to determine the price or the trend of a certain stock for the coming end-of-day considering the first several trading hours of the day.

To achieve this goal, we trained traditional machine learning algorithms and created/trained multiple deep learning models taking into consideration the importance of the relevant news. Various experiments were conducted, the highest accuracy (82.91%) of which was achieved using SVM for Apple Inc. (AAPL) stock.

F. Stock Price Correlation Coefficient Prediction with ARIMALSTM Hybrid Model

The research work done by Hyeong Kyu Choi, B.A Student Dept. of Business Administration Korea University Seoul, Korea. Predicting the price correlation of two assets for future time periods is important in portfolio optimization. We apply LSTM recurrent neural networks (RNN) in predicting the stock price correlation coefficient of two individual stocks. RNN’s are competent in understanding temporal dependencies.

The use of LSTM cells further enhances its long-term predictive properties. To encompass both linearity and nonlinearity in the model, we adopt the ARIMA model as well. The ARIMA model filters linear tendencies in the data and passes on the residual value to the LSTM model.

The ARIMA-LSTM hybrid model is tested against other traditional predictive financial models such as the full historical model, constant correlation model, single-index model and the multi-group model.

In our empirical study, the predictive ability of the ARIMA-LSTM model turned out superior to all other financial models by a significant scale. Our work implies that it is worth considering the ARIMALSTM model to forecast correlation coefficient for portfolio optimization.

G. Forecasting Directional Movements of Stock Prices for Intraday Trading using LSTM and Random Forests

The research work done by Pushpendu Ghosh, Ariel Neufeld, Jajati Keshari Sahoo Department of Computer Science & Information Systems, BITS Pilani K.K. Birla Goa campus, India division of Mathematical Sciences, Nanyang Technological University, Singapore department of Mathematics, BITS Pilani K.K. Birla Goa campus, India. We employ both random forests and LSTM networks (more precisely CuDNNLSTM) as training methodologies to analyse their effectiveness in forecasting out of-sample directional movements of constituent stocks of the S&P 500 from January 1993 till December 2018 for intraday trading. We introduce a multi-feature setting consisting not only of the returns with respect to the closing prices, but also with respect to the opening prices and intraday returns. As trading strategy, we use Krauss et al. (2017) and Fischer & Krauss (2018) as benchmark and, on each trading day, buy the 10 stocks with the highest probability and sell short the 10 stocks with the lowest probability to outperform the market in terms of intraday returns – all with equal monetary weight. Our empirical results show that the multi-feature setting provides a daily return, prior to transaction costs, of 0.64% using LSTM networks, and 0.54% using random forests. Hence, we outperform the single feature setting in Fischer & Krauss (2018) and Krauss et al. (2017) consisting only of the daily returns with respect to the closing prices, having corresponding daily returns of 0 .41% and of 0 .39% with respect to LSTM and random forests, respectively. 1 Keywords: Random Forest, LSTM, Forecasting, Statistical Arbitrage, Machine learning, Intraday trading.

III. PROBLEM STATEMENT

The stock market appears in the news every day. You hear about it every time it reaches a new high or a new low. The rate of investment and business opportunities in the Stock market can increase if an efficient algorithm could be devised to predict the short-term price of an individual stock. Previous methods of stock predictions involve the use of Artificial Neural Networks and Convolution Neural Networks which has an error loss at an average of 20%. In this project, we will see if there is a possibility of devising a model using Recurrent Neural Network which will predict stock price with a less percentage of error. And if the answer turns to be YES, we will also see how reliable and efficient will this model be.

A. Goal and Objectives

1. To determine the future value of a company stock or other financial instrument traded on an exchange.
2. In the current emerging competitive market, predicting the stock returns as well as the company's financial status in advance will provide more benefits for the investors in order to invest confidently.
3. Forecasting and predicting the trends of market.
4. To help the investors to understand when and what stocks can be purchased for the growth of their investment.

B. Statement of Scope

Stock market prediction is the act of trying to determine the future value of a company stock or other financial instrument traded on an exchange. The successful prediction of a stock's future price could yield significant profit. This project requires investigating trends in stock market and factors affecting the stock prices. This project aims at predicting stock market by using financial news, Analyst opinions and quotes in order to improve quality of output. It proposes a novel method for the prediction of the stock market closing price.

C. Software Context

To develop algorithms, we require Python 3.5 in Google Colab. It is used for data pre-processing, model training and prediction.

D. Major Constraints

Generally speaking, forecasting the price of a stock is comparatively easier than predicting a stock price, which is much more difficult. For prediction, the model needs to have multiple inputs. This can include major political and economic events and assessing their impact on the stock price. Also, aspects that are tough to capture, such as the market’s mood, psychology, and herd mentality, need to be quantified and fed to the model. If the user can do so, then for sure, the model can predict the stock price. The reason is that even when the components involved for successfully predicting the stock value range from physical to psychological, driven by human motives that can often be irrational, there are still patterns that the Machine Learning model can capture. What needs to be understood is that the volatility in the market is because stock prices are solely influenced by humans (it’s a market, after all), and humans are often unpredictable and irrational. Still, their unpredictability and irrationality also have patterns that a machine learning model can capture. Suppose the country’s president where the stock is being traded is pro-free market and loses the election. In that case, people can panic and sell stocks leading to a sudden drop in the price, whereas if another anti-capitalist president loses, the market will react oppositely, and the stock prices can shoot up.

E. Methodologies of Problem Solving and Efficiency Issues

The high-level architecture of our proposed solution could be separated into three parts. First is the feature selection part, to guarantee the selected features are highly effective. Second, we look into the data and perform the dimensionality reduction. And the last part, which is the main contribution of our work is to build a prediction model of target stocks.

IV. PROPOSED METHODOLOGY

Stock is unpredicted and liberal in nature. The follow of the same is impressive and reluctant in nature. Finding the predictability and getting the nearest is the best hit goal for the same. The exact and accurate estimation of the same is never-less possible. There are various constrains that in-fluctuate the pricing and the rate of stock. Those constrains had to be taken in consideration before jumping to the conclusion and report derivation.

Here as described in the figure above, the proposed system will have an input from the dataset which will be extracted featured wise and Classified underneath. The classification technique used is supervised and the various techniques of machine level algorithms are implemented on the same. Training Dataset are created for training the machine and the test cases are derived and implemented to carry out the visualization and the plotting’s. The result generated are passed and visualized in the graphical form.

A. Moving Average

‘Average’ is easily one of the most common things we use in our day-to-day lives. For instance, calculating the average marks to determine overall performance, or finding the average temperature of the past few days to get an idea about today’s temperature – these all are routine tasks we do on a regular basis. So, this is a good starting point to use on our dataset for making predictions.

The predicted closing price for each day will be the average of a set of previously observed values. Instead of using the simple average, we will be using the moving average technique which uses the latest set of values for each prediction. In other words, for each subsequent step, the predicted values are taken into consideration while removing the oldest observed value from the set.

B. Linear Regression

The most basic machine learning algorithm that can be implemented on this data is linear regression. The linear regression model returns an equation that determines the relationship between the independent variables and the dependent variable.

The equation for linear regression can be written as:

C. K-Nearest Neighbours

Another interesting ML algorithm that one can use here is KNN (k nearest neighbours). Based on the independent variables, KNN finds the similarity between new data points and old data points.

D. Auto ARIMA

ARIMA is a very popular statistical method for time series forecasting. ARIMA models take into account the past values to predict the future values. There are three important parameters in ARIMA:

1. p (past values used for forecasting the next value)
2. q (past forecast errors used to predict the future values)
3. d (order of differencing)

Parameter tuning for ARIMA consumes a lot of time. So, we will use auto ARIMA which automatically selects the best combination of (p, q, d) that provides the least error.

E. Prophet

There are a number of time series techniques that can be implemented on the stock prediction dataset, but most of these techniques require a lot of data pre-processing before fitting the model. Prophet, designed and pioneered by Facebook, is a time series forecasting library that requires no data pre-processing and is extremely simple to implement. The input for Prophet is a data frame with two columns: date and target (ds and y).

F. Long Short-Term Memory (LSTM)

LSTMs are widely used for sequence prediction problems and have proven to be extremely effective. The reason they work so well is because LSTM is able to store past information that is important, and forget the information that is not. LSTM has three gates:

1. The input gate: The input gate adds information to the cell state.
2. The forget gate: It removes the information that is no longer required by the model.
3. The output gate: Output Gate at LSTM selects the information to be shown as output.

V. PROJECT PURPOSE

Stock market prediction is a prediction system software that illuminate the risk that undergoes during the investment in stock market. It predicts the stock rates and its rate of exchange acknowledging the basic understanding and the statistical analysis in front of users. The main theme of the project is to predict the turning curves and bring the predictability method and undergo the process and algorithms to conclude to a viable resource source. This project helps in bridging the resources and empowering the people to know and trade the most out of stock and understand the generation and the vulnerabilities that has to be seen and predicted. The enhancement of the same is done with the resource graph which makes a user or the customer to analyses the same and take the needs and important details before dealing and consider those things for the yield that the person is willing to invest on. Forecasting of the stock prediction is done for the upcoming week. The predictability itself is a challenge and that’s the main purpose of the project.

VI. RESULT

VII. FUTURE ENHANCEMENT

Stock Market are the best alternative for business to grow and it’s a side way income for the individuals who are ready to invest and earn from the same. The term stock had been in picture ever since and it’s growing in bulk every day. There are thousands of investors investing on the same and making the fortune out of it. The project can be further continued to gain the effectiveness of the prediction with addition implementations of the content that can involve real time scenario and the way of executing and processing the real time scenario. Various constraints have to be added and performance of the same can be acylated in the future time for the effective results. The expected form of the display is graph whereas from the same the more appearance and setting of the display can be integrated and a pie-chart and a custom graph can further be implemented on the same.

VIII. ACKNOWLEDGEMENT

I would like to express my deep gratitude to Professor Mrs. A.S. Khandagale, our project guide, for their patient guidance, enthusiastic encouragement and useful critiques of this research work.

I would also like to thank Mrs. V.R. Palandurkar, for her advice and assistance in keeping our progress on schedule.

I would also like to extend my thanks to the technicians of the laboratory of the Information Technology department for their help in offering me the resources in running the program.

Finally, I wish to thank my parents for their support and encouragement throughout my study.

Conclusion

Stock Market are the best alternative for business to grow and it’s a side way income for the individuals who are ready to invest and earn from the same. The term stock had been in picture ever since and it’s growing in bulk every day. There are thousands of investors investing on the same and making the fortune out of it. The project can be further continued to gain the effectiveness of the prediction with addition implementations of the content that can involve real time scenario and the way of executing and processing the real time scenario. Various constraints have to be added and performance of the same can be acylated in the future time for the effective results. The expected form of the display is graph whereas from the same the more appearance and setting of the display can be integrated and a pie-chart and a custom graph can further be implemented on the same.

References

[1] N. Babu and B. E. Reddy, “Selected Indian stock predictions using a hybrid ARIMA-GARCH model,” 2014 Int. Conf. Adv. Electron. Comput. Commun. ICAECC 2014, 2015. [2] M. D. Shubhrata, D. Kaveri, T. Pranit, and S. Bhavana, “Stock Market Prediction and Analysis Using Naïve Bayes,” Int. J. Recent Innov. Trends Comput. Commun., vol. 4, no.11, pp. 121–124, 2016. [3] X. Ding, Y. Zhang, T. Liu, and J. Duan, “Deep Learning for Event-Driven Stock Prediction,” no. Ijcai, pp. 2327–2333, 2015. [4] K. A. Smith and J. N. D. Gupta, “Neural networks in business: Techniques and applications for the operations researcher,” Comput. Oper. Res., vol. 27, no. 11–12, pp.1023–1044, 2000. [5] M. S. Babu, N. Geethanjali, and P. B. Satyanarayana, “Clustering Approach to Stock Market Prediction,” vol. 1291, pp. 1281– 1291, 2012. [6] A. A, Adebiyi, A. K, Charles, A. O, Marion, and O. O, Sunday, “Stock Price Prediction using Neural Network with Hybridized Market Indicators,” J. Emerg. Trends Comput. Inf. Sci., vol. 3, no. 1, pp. 1–9, 2012. [7] S. Kayode, “Stock Trend Prediction Using Regression Analysis – A Data Mining Approach,” ARPN J. Syst. Softw., vol. 1, no. 4, pp. 154–157, 2011. [8] A. Sharma, D. Bhuriya, and U. Singh, “Survey of Stock Market Prediction Using Machine Learning Approach,” pp. 506–509, 2017. [9] H. L. Siew and M. J. Nordin, “Regression techniques for the prediction of stock price trend,” ICSSBE 2012 - Proceedings, 2012 Int. Conf. Stat. Sci. Bus. Eng. \"Empowering Decis. Mak. with Stat. Sci., pp. 99–103, 2012. [10] B. Sivakumar and K. Srilatha, “A novel method to segment blood vessels and optic disc in the fundus retinal images,” Res. J. Pharm. Biol. Chem. Sci., vol. 7, no. 3, pp. 365–373, 2016. [11] M. P. Mali, H. Karchalkar, A. Jain, A. Singh, and V. Kumar, “Open Price Prediction of Stock Market using Regression Analysis,” Ijarcce, vol. 6, no. 5, pp. 418–421, 2017. [12] L. Khaidem, S. Saha, and S. R. Dey, “Predicting the direction of stock market prices using random forest,” vol. 0, no. 0, pp. 1– 20, 2016. [13] T. Manojlovi? and I. Štajduhar, “Predicting stock market trends using random forests: A sample of the Zagreb stock exchange,” 38th Int. Conv. Inf. Commun. Technol. Electron. Microelectron. MIPRO 2015, no. May, pp. 1189–1193, 2015. [14] S. B. Imandoust and M. Bolandraftar, “Forecasting the direction of stock market index movement using three data mining techniques: The case of Tehran Stock Exchange,” Int. J. Eng. Res. Appl., vol. 4, no. 6, pp. 106–117, 2014.

Copyright

Copyright © 2022 Mr. Yash Kadam, Mr. Sujay Kulkarni, Mr. Suyog Lonsane, Prof. Anjali S. Khandagale. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.