Deep Learning for Solving Partial Differential Equations: A Review of Literature

Abstract

Partial Differential Equations (PDEs) are fundamental in modeling various phenomena in physics, engineering, and finance. Traditional numerical methods for solving PDEs, such as finite element and finite difference methods, often face limitations when applied to high-dimensional and complex systems. In recent years, deep learning has emerged as a promising alternative for approximating solutions to PDEs, offering potential improvements in both efficiency and scalability. This paper provides a comprehensive review of the literature on deep learning-based methods for solving PDEs, focusing on key approaches such as Physics-Informed Neural Networks (PINNs), deep Galerkin methods, and neural operators. These methods leverage the expressiveness of neural networks to capture underlying physics while avoiding the curse of dimensionality associated with classical techniques. We explore the theoretical foundations, advantages, and limitations of these deep learning models, along with their applications in diverse fields like fluid dynamics, quantum mechanics, and financial modeling. Additionally, this review examines recent advancements in hybrid models that combine traditional numerical methods with deep learning approaches to enhance accuracy and stability. Through this review, we highlight key trends and open challenges in the field, paving the way for future research at the intersection of deep learning and computational mathematics.

Introduction

I. INTRODUCTION

Partial Differential Equations (PDEs) play a crucial role in modeling complex systems across a wide range of scientific and engineering disciplines, including fluid dynamics, heat transfer, electromagnetism, and financial modeling. Traditionally, solving PDEs has relied on numerical methods like finite element, finite difference, and spectral methods. While effective, these classical approaches often struggle with high-dimensional problems, complex geometries, and computational inefficiencies, particularly as models become more intricate. In recent years, deep learning has emerged as a transformative tool in computational mathematics, providing new ways to tackle PDEs that are intractable using traditional methods. Neural networks, with their ability to approximate highly complex functions, have demonstrated potential in solving PDEs by bypassing some of the limitations of conventional techniques. Deep learning-based approaches such as Physics-Informed Neural Networks (PINNs), deep Galerkin methods, and neural operators are redefining how we approach PDE solutions, especially for problems involving high-dimensional spaces or uncertain boundary conditions. PINNs, for example, incorporate known physical laws directly into the network’s architecture, enabling more accurate and stable solutions. Additionally, these methods are naturally suited for handling large datasets and can efficiently generalize across various problem domains. As deep learning continues to evolve, it offers a promising pathway for solving PDEs, with applications extending to real-time simulations, inverse problems, and scenarios where data-driven models are integrated with theoretical physics. This paper aims to provide a comprehensive review of the literature on deep learning techniques for solving PDEs, discussing key methodologies, their advantages over traditional methods, and their applications in different fields. By analyzing the current state of research, this review also highlights ongoing challenges and future directions for enhancing the performance, accuracy, and scalability of deep learning approaches in the context of solving PDEs.

.

II. PREVIOUS WORKS

High-dimensional Partial Differential Equations (PDEs) have posed significant computational challenges due to the infeasibility of mesh-based methods in high dimensions. The paper [1] propose a solution using a deep neural network to approximate PDE solutions by training the network to satisfy the differential operator, initial condition, and boundary conditions. Their algorithm is meshfree, training on randomly sampled time and space points instead of forming a mesh. Tested on high-dimensional free boundary PDEs, the algorithm accurately solves problems in up to 200 dimensions. It is also applied to a Hamilton–Jacobi–Bellman PDE and Burgers' equation, where it generalizes solutions across a continuum of boundary and physical conditions.

The method, termed the "Deep Galerkin Method (DGM)," approximates solutions with a neural network, similar to how Galerkin methods use basis functions. Additionally, the authors prove a theorem on the approximation power of neural networks for certain quasilinear parabolic PDEs.

The paper [2] presents an enhanced method for solving partial differential equations (PDEs) using neural networks that incorporate physical information, improving upon existing physics-informed neural networks (PINNs). By integrating the physical laws inherent in PDEs as a form of regularization within the neural networks, the method encourages better learning of solutions from limited observational data. The authors conduct experimental analyses on three significant PDEs, achieving promising results due to the neural networks' strong function approximation capabilities combined with the embedded physical information. They anticipate that this approach will significantly impact the study of PDE solutions and advance the field of scientific computing. However, challenges remain, including optimizing the introduction of physical information into the networks and addressing potential non-convergence issues in loss function optimization, which will be the focus of future research. The authors also plan to compare the performance of PINN-based methods with finite element methods (FEM) in terms of accuracy and computational efficiency.

This paper [3] introduces a deep learning approach for extracting nonlinear partial differential equations (PDEs) from spatio-temporal datasets, utilizing recent advancements in automatic differentiation to create efficient algorithms for learning infinite-dimensional dynamical systems via deep neural networks. The authors acknowledge the reliance on black-box solvers to validate their method, highlighting the need for developing general-purpose PDE solvers that can match the maturity of traditional techniques like finite element, finite difference, and spectral methods. The paper identifies several open questions for future research, such as the challenge of parameter-dependent PDEs that may undergo bifurcations and the potential for collecting data across various parameter values to infer parameterized equations. Additionally, the authors propose exploring convolutional architectures to address the complexity of high-dimensional PDEs frequently encountered in dynamic programming, optimal control, and reinforcement learning. They believe their framework can be extended to these high-dimensional cases and is scalable to big data scenarios through mini-batch processing. The authors also express interest in expanding their methodology to encompass nonlinear PDEs that do not fit the standard form and to stochastic PDEs, utilizing independent realizations of noise processes as training data. They note that the challenge of selecting appropriate measurements from a dynamical system remains significant, suggesting that time-delay coordinates could provide additional variables in the infinite-dimensional context. Finally, they propose applying their framework to real-world datasets, such as sea surface temperature data from Geostationary Operational Environmental Satellites (GOES).

This research work [4] addresses the longstanding challenge of developing algorithms to solve high-dimensional partial differential equations (PDEs), which is complicated by the "curse of dimensionality." The authors propose a deep learning-based approach that reformulates general high-dimensional parabolic PDEs using backward stochastic differential equations, approximating the gradient of the unknown solution with neural networks in a manner akin to deep reinforcement learning, where the gradient serves as the policy function. Numerical results demonstrate the effectiveness of the proposed algorithm on various examples, including the nonlinear Black–Scholes equation, the Hamilton–Jacobi–Bellman equation, and the Allen–Cahn equation, showing promising accuracy and cost efficiency in high dimensions. This approach opens new possibilities in fields such as economics, finance, operational research, and physics by enabling a comprehensive consideration of all participating agents, assets, resources, or particles simultaneously, rather than relying on simplified assumptions about their interrelationships.

The paper [5] presents physics-informed neural networks (PINNs) that are designed to address supervised learning tasks while respecting the physical laws governed by general nonlinear partial differential equations. The authors focus on two primary applications: data-driven solutions and data-driven discovery of PDEs. To accommodate different data arrangements, they develop two types of algorithms: continuous time models, which act as data-efficient spatio-temporal function approximators, and discrete time models, which utilize highly accurate implicit Runge-Kutta time-stepping schemes with unlimited stages. The effectiveness of this framework is validated through a range of classical problems in fluid dynamics, quantum mechanics, reaction-diffusion systems, and the propagation of nonlinear shallow-water waves. Also the research work  [6] investigates a novel algorithm for solving high-dimensional parabolic partial differential equations (PDEs) and backward stochastic differential equations (BSDEs). This approach draws an analogy between BSDEs and reinforcement learning, where the gradient of the solution functions as the policy, and the loss function represents the error between the prescribed terminal condition and the BSDE solution. The policy function is approximated using a neural network, similar to techniques in deep reinforcement learning. Numerical results obtained with TensorFlow demonstrate the algorithm's efficiency and accuracy when applied to several 100-dimensional nonlinear PDEs from physics and finance, including the Allen–Cahn equation, the Hamilton–Jacobi–Bellman equation, and a nonlinear pricing model for financial derivatives.

Researchers [7] have provided an overview of physics-informed neural networks (PINNs), a recent advancement in applying deep learning to solve partial differential equations (PDEs). PINNs integrate a PDE into the neural network's loss function using automatic differentiation, making the algorithm versatile for various types of PDEs, including integro-differential, fractional, and stochastic equations. Additionally, PINNs facilitate the solution of inverse problems with the same ease as forward problems. The authors introduce a new residual-based adaptive refinement (RAR) method to enhance training efficiency. For educational purposes, the paper compares the PINN algorithm with traditional finite element methods. They also present DeepXDE, a Python library designed for both classroom use and research applications in computational science and engineering. DeepXDE can solve forward problems with initial and boundary conditions, as well as inverse problems with additional measurements, while supporting complex geometries through constructive solid geometry techniques. The library promotes compact user code that closely resembles mathematical formulations. The authors demonstrate the capabilities of PINNs and the user-friendliness of DeepXDE through five examples, highlighting its contribution to the advancement of the scientific machine learning field. A similar research work [8] addresses the challenge of efficiently solving partial differential equations (PDEs), which are prevalent in mathematics, physics, and engineering for modeling natural phenomena and dynamical systems. The authors develop a deep learning-based numerical method combined with small sample learning (SSL) to approximate PDE solutions using a deep feedforward neural network. This network is trained to satisfy the PDEs along with the initial and boundary conditions. The method is framed as an optimization problem that minimizes a specially designed cost function, incorporating the residuals of the differential equations, initial/boundary conditions, and a limited number of observations. The proposed approach is demonstrated through various benchmark problems in mathematical physics, including the Burgers equations, Schrödinger equations, Buckley-Leverett equation, Navier-Stokes equation, and carburizing diffusion equations relevant to material science. The results show that the algorithm is effective, flexible, and robust, achieving good predictive accuracy without relying on trial solutions.

The work [9] explores the application of deep neural networks (DNNs) for solving partial differential equations (PDEs), highlighting their potential due to the ability to learn complex solution-related features. However, the authors note that the training process for DNNs often involves extensive iterations and requires large datasets, which limits their use in complex physical scenarios. To overcome these challenges, the authors propose utilizing transfer learning for DNN-based PDE solving tasks. They conduct transfer experiments on the Helmholtz and Navier-Stokes equations by creating subtasks with varying source terms and Reynolds numbers. A series of experiments are performed to assess the generalizability of features across different equations. The results indicate that the transfer learning approach significantly enhances the accuracy of predicted solutions, achieving a maximum performance improvement of 97.3% on commonly used surrogate models, despite the differences in the underlying PDE systems. Paper [10] introduces a novel framework for addressing the computational challenges of solving nonlinear evolution partial differential equations by utilizing deep learning techniques. By approximating latent solutions with a deep neural network and incorporating physical laws as constraints during training, the proposed paradigm demonstrates its effectiveness, particularly in the context of the Burgers' equation. The results indicate that the framework not only successfully captures complex patterns in data but also significantly reduces training time for soliton solutions compared to other initial conditions. This approach has the potential to advance the field of computational science, offering a robust method for tackling a variety of nonlinear evolution equations and enhancing our understanding of complex dynamical systems. Future research may explore further applications of this framework across different types of PDEs and refine its capabilities to improve efficiency and accuracy in diverse scientific contexts. The researchers [11] propose a new Deep Neural Network (DNN) method for solving Partial Differential Equations (PDEs). The method simultaneously constructs an optimal r-adapted mesh and solves the PDE over the constructed mesh. The node locations are optimized over a set of 1D boundary nodes, and the corresponding 2D quadrilateral meshes are built using tensor product. The method supports the definition of fixed interfaces to create conforming meshes and allows nodes to jump across them, permitting topological variations. The authors apply their method in combination with other numerical methods including collocation, Least Squares, and the Deep Ritz method. They solve one- and two-dimensional problems whose solutions are smooth, singular, and/or exhibit strong gradients. Results consistently show the outperformance of employing r-adaptivity, while in some cases the improvement is limited by the tensor-product structure 2 of the mesh.

The paper [12] propose a framework for recovering/approximating unknown time-dependent partial differential equations (PDEs) using its solution data. Instead of identifying the terms in the underlying PDE, they seek to approximate the evolution operator of the underlying PDE numerically. The evolution operator of the PDE, defined in infinite-dimensional space,

maps the solution from a current time to a future time and completely characterizes the solution evolution of the underlying unknown PDE. Their recovery strategy relies on approximation of the evolution operator in a properly defined modal space, i.e., generalized Fourier space, in order to reduce the problem to finite dimensions.

The finite dimensional approximation is then accomplished by training a deep neural network structure, which is based on residual network (ResNet), using the given data. Error analysis is provided to illustrate the predictive accuracy of the proposed method. A set of examples of different types of PDEs, including inviscid Burgers' equation that develops discontinuity in its solution, are presented to demonstrate the effectiveness of the proposed method. Another research [13] proposes using Deep Neural Networks (DNNs) to approximate the solution of Partial Differential Equations (PDEs) in Computational Mechanics. DNNs are function approximation machines that can be used to solve PDEs with great flexibility and efficiency. DNNs are particularly well-suited for PDEs that are nonlinear, have complex geometries, or involve multiple scales. For example, DNNs can be used to solve PDEs that model fluid flow, heat transfer, and structural mechanics. The author focuses on mechanical problems and analyzes the energetic format of the PDE. The energy of a mechanical system seems to be the natural loss function for a machine learning method to approach a mechanical problem. The author deals with several problems and explores the capabilities of the method for applications in engineering. The author finds that DNNs are able to accurately approximate the solution of PDEs in a variety of mechanical problems. For example, DNNs can be used to predict the deformation of structures, the flow of fluids, and the heat transfer in solids.

Conclusion

In summary, the discussed deep neural network (DNN) method for solving partial differential equations (PDEs) represents a significant advancement in computational techniques for handling complex mathematical models. By integrating an r-adaptive mesh optimization process, this approach not only enhances the accuracy of PDE solutions but also allows for flexible topological variations through optimized node placements. The method\'s capability to adaptively refine mesh structures addresses the challenges associated with singularities and strong gradients in solutions, leading to improved performance across various numerical methods, including collocation, Least Squares, and the Deep Ritz method. As demonstrated through numerical experiments, the integration of r-adaptivity consistently yields superior results, showcasing its potential in practical applications. This innovative framework opens new avenues for future research and development in scientific computing, particularly in fields where accurate modeling of physical processes is crucial.

References

[1] J. Sirignano and K. Spiliopoulos, “DGM: A deep learning algorithm for solving partial differential equations,” J. Comput. Phys., vol. 375, pp. 1339–1364, Dec. 2018, doi: 10.1016/j.jcp.2018.08.029. [2] Y. Guo, X. Cao, B. Liu, and M. Gao, “Solving Partial Differential Equations Using Deep Learning and Physical Constraints,” Appl. Sci., vol. 10, no. 17, Art. no. 17, Jan. 2020, doi: 10.3390/app10175917. [3] M. Raissi, “Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Di?erential Equations”. [4] J. Han, A. Jentzen, and W. E, “Solving high-dimensional partial differential equations using deep learning,” Proc. Natl. Acad. Sci., vol. 115, no. 34, pp. 8505–8510, Aug. 2018, doi: 10.1073/pnas.1718942115. [5] M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” J. Comput. Phys., vol. 378, pp. 686–707, Feb. 2019, doi: 10.1016/j.jcp.2018.10.045. [6] W. E, J. Han, and A. Jentzen, “Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations,” Commun. Math. Stat., vol. 5, no. 4, pp. 349–380, Dec. 2017, doi: 10.1007/s40304-017-0117-6. [7] “DeepXDE: A Deep Learning Library for Solving Differential Equations | SIAM Review.” Accessed: Oct. 15, 2024. [Online]. Available: https://epubs.siam.org/doi/abs/10.1137/19M1274067 [8] Y. Li and F. Mei, “Deep learning-based method coupled with small sample learning for solving partial differential equations,” Multimed. Tools Appl., vol. 80, no. 11, pp. 17391–17413, May 2021, doi: 10.1007/s11042-020-09142-8. [9] X. Chen et al., “Transfer learning for deep neural network-based partial differential equations solving,” Adv. Aerodyn., vol. 3, no. 1, p. 36, Dec. 2021, doi: 10.1186/s42774-021-00094-7. [10] J. Li and Y. Chen, “Solving second-order nonlinear evolution partial differential equations using deep learning*,” Commun. Theor. Phys., vol. 72, no. 10, p. 105005, Oct. 2020, doi: 10.1088/1572-9494/aba243. [11] Á. J. Omella and D. Pardo, “r-Adaptive deep learning method for solving partial differential equations,” Comput. Math. Appl., vol. 153, pp. 33–42, Jan. 2024, doi: 10.1016/j.camwa.2023.11.005. [12] K. Wu and D. Xiu, “Data-driven deep learning of partial differential equations in modal space,” J. Comput. Phys., vol. 408, p. 109307, May 2020, doi: 10.1016/j.jcp.2020.109307. [13] E. Samaniego et al., “An energy approach to the solution of partial differential equations in computational mechanics via machine learning: Concepts, implementation and applications,” Comput. Methods Appl. Mech. Eng., vol. 362, p. 112790, Apr. 2020, doi: 10.1016/j.cma.2019.112790.

Copyright

Copyright © 2024 Nishchal Sharma, Som Krishan Sharma. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.