Applications of Artificial Neural Networks to Solve Ordinary Differential Equations

Abstract

Applications of Neural Networks to numerical problems have gained increasing interest. Solving Ordinary Differential Equations can be realized with simple artificial neural network architectures. In this paper, first step is training a Neural Network to satisfy the condition required by a differential equation and then finding a function whose derivative satisfies the Ordinary Differential Equation condition. The method of solving differential equation is implemented in Python Programming using the TensorFlow library.

Introduction

I. INTRODUCTION

Differential equations play an important role in various fields such as Science and Engineering. Numerical approaches to solve the differential equation have been studied by different researchers. Several methods have been developed to solve the differential equations. Some of the methods to produce a solution in the form of array that contains the value of the solution at a selected group of points and the other methods use basis functions to find the solution in analytic form. However, there is a need to develop more efficient and universal method to solve the differential equations. The connection between Mathematics and physical phenomena is often formulated by differential equations which combine the function and their derivatives to produce a mathematical model of real world problems [10].

With the emerging field of Computer Science and Scientific Computing, Artificial Neural Networks is considered as one of such methods. Algorithms based on ANN have been proposed for solving first order differential equations. Later on the algorithm was developed based on feedforward neural networks for solving ordinary differential equations [7,8]. Lagaris et al. proposed algorithms for solving Partial differential equations on regular and irregular domains. The focus on algorithm is to use output of a single layer neural network to construct numerical solutions that satisfy the boundary value problem. This paper is organized as follows. Section II discusses the related work, Section III focuses on Ordinary Differential Equations, Section IV presents Artificial Neural Networks, Section V discusses about Mathematical Formulation for Neural Network Function, Section VI shows Python Implementation and Results and finally the Conclusion is given in Section VII.

II. RELATED WORK

Issac Elias Lagaris et al., have presented a method to solve Initial Value and Boundary Value Problems using Artificial Neural Networks for solving Ordinary Differential equation and Partial Differential Equations. The idea of solving an Ordinary Differential Equation using a Neural Network was first described by them [1]. Marco Di Giovanni et al., have designed a novel algorithm to find the solutions of a differential equation with non-unique solution using neural networks and tested the accuracy of applying neural networks to the non-linear differential equation [9].

Liam L.H. Lan and Dem’s Werth have explored the method of using Neural Networks to find the solution of differential equations and found that the Neural Networks was able to perform better than the typical numerical solutions for differential equation such as for highly oscillating solutions [4]. Toni Schneidereit and Michael Breus have investigated several parameters and constant versus random weight initialization for two solution methods for solving Ordinary Differential Equations using Artificial Neural Networks [3]. Susmita Mall and S. Chakraverty have investigated the solution of Ordinary Differential Equations with initial conditions using Regression based algorithm and compared the different Neural Network architectures corresponding to different regression models [5].

Enze Shi and Chuanju Xu have discussed the possible forms of loss function in an Artificial Neural Networks method for solving differential equations and investigated their efficiency through examples [2]. Forough Arabshahi et al., have presented a novel approach for solving differential equations using Neural Networks [6].

III. ORDINARY DIFFERENTIAL EQUATIONS (ODE)

An Ordinary Differential Equation is a relation between a function, its derivatives and the variable upon which they depend. The general form of an Ordinary Differential Equation is given by

IV. ARTIFICIAL NEURAL NETWORKS (ANN)

An Artificial Neural Network (ANN) is a mathematical model that tries to simulate the structure and functionalities of biological neural networks. Basic building block of every artificial neural network is artificial neuron which is a simple mathematical model or function. Such a model has three simple sets of rules: multiplication, summation and activation. At the entrance of artificial neuron the inputs are weighted what means that every input value is multiplied with individual weight. In the middle section of artificial neuron is sum function that sums all weighted inputs and bias. At the exit of artificial neuron the sum of previously weighted inputs and bias is passing through activation function that is also called transfer function [11].

Artificial Neural Network (ANN) model involves computations and mathematics, which simulate the human–brain processes. Many of the recently achieved advancements are related to the artificial intelligence research area such as image and voice recognition, robotics using ANN. The ANN models have the specific architecture format, which is inspired by a biological nervous system. Like the structure of the human brain, the ANN models consist of neurons in a complex and nonlinear form. The neurons are connected to each other by weighted links. All the processes in ANN models, such as data collection and analysis, network structure design, number of hidden layers, network simulation, and weights/bias trade-off are computed through learning and training methods. A neural network is simply a collection of Neurons which is also known as activations that are connected through various layers. It attempts to learn the mapping of input data to output data on being provided a training set.

The training of the neural network later facilitates the predictions made by it on a testing data of the same distribution. This mapping is attained by a set of trainable parameters called weights, distributed over different layers. The weights are learned by the backpropagation algorithm whose aim is to minimize a loss function. A loss function measures how distant the predictions made by the network are from the actual values. Every layer in a neural network is followed by an activation layer that performs some additional operations on the neurons.

The Loss Function is one of the important components of Neural Networks. Loss is nothing but a prediction error of Neural Net. And the method to calculate the loss is called Loss Function. In simple words, the Loss is used to calculate the gradients. Gradients are used to update the weights of the Neural Net. Tensorflow library has various inbuilt loss functions for different objectives. Mean Squared Error loss is used for regression tasks. This loss is calculated by taking the mean of squared differences between actual and predicted values.

Mathematically speaking, any neural network architecture aims at finding any mathematical function y=fx that can map attributes (x)  to output (y) . The accuracy of mapping of this function differs depending on the distribution of the dataset and the architecture of the network employed. The Universal Approximation Theorem tells us that Neural Networks has a kind of universality. For any function fx, there is a network that can approximately approach the result. This result holds for any number of inputs and outputs.

V. MATHEMATICAL FORMULATION FOR NEURAL NETWORK FUNCTION

VI. PYTHON IMPLEMENTATION AND RESULTS

To implement the described method in Python using TensorFlow library, we will use a low level design to avoid a number of possible optimizations provided in the library. Our focus is to implement the solution of ordinary differential equation by using Neural Network. Let us consider the first order differential equation

Conclusion

In this paper, ANN is applied to find the solution of first order ordinary differential equation. We have implemented Python programming to find the solution. In this approach, we found that ANN was able to perform better than numerical solvers for differential equation. The proposed method has been explored by solving the differential equation to obtain the accuracy. We have discussed possible forms of loss functions in an ANN method for solving differential equation and investigated their efficiency through a number of numerical examples.

References

[1] Isaac Elias Lagaris, Aristidis Likas and Dimitrios I. Fotiadis, Artificial Neural Networks for Solving Ordinary and Partial Differential Equations, IEEE Transactios On Neural networks, Vol. 9, No. 5, September 1998, pp. 987 – 1000. [2] Enzi Shi and Chuanju Xu, A Comparative Investigation of Neural Networks in Solving Differential Equations, Journal of Algorithms and Computational Technology, Vol. 15, 2021, pp. 1 – 15. [3] Toni Schneidereit and Michael Dreub, Solving Ordinary Differential Equations Using Artificial Neural Networks – A Study on the Solution Variance, Proceedings of ALGORITMY 2020, pp. 21 – 30 . [4] Liam L.H Lau and Denis Werth, ODEN: A framework to Solve Ordinary Differential Equations using Artificial Neural Networks, arXiv:2005, 14090V2[physics.comp-ph] june 2020, pp. 1 – 10 [5] Susmita Mall and S.Chakraverty, Comparison of Artificial Neural Network Architecture in Solving Ordinary Differential Equations, Advances in Artificial Neural Systems, Hindawi Publishing corporation, Vol. 2013, pp. 1 -12. [6] Forough Arabshahi, Sameer Singh and Animashree Anandkumar, Towards Solving Differential equations through Neural Programming, ICML workshop Neural Abstract Machines and Program Induction, V2, 2018, pp.1 – 4 . [7] Lee H and Kang IS Neural Algorithm for Solving Differential equations, Journal of Computational Physics, 1990; Vol. 91, pp. 110 – 131 . [8] Meade AJ and Fernandez AA, The Numerical Solution of linear Ordinary Differential Equations by feedforward neural networks, Mathematical and Computer Modelling, 1994, Vol. 19, 1994, pp. 1- 25. [9] Malek A. and Shekari Beidokhti R., Numerical solution for higher order differential equations using a hybrid neural network optimization method, Applied mathematics and Computation, Vol. 183, 2006, pp.260 – 271 . [10] Michoski C., Milosavljevic M., Oliver T.,et al., Solving Differential Equations using Deep neural networks, Neurocomputing 2020; Vol. 399, pp. 193 – 212. [11] J.M. Zurada, Introduction to Artificial Neural Network, West Publishing, 1994.

Copyright

Copyright © 2022 Arunachalam Sundaram . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.