ChemNODE: A Neural Ordinary Differential Equations Approach for Chemical Kinetics Solvers

The main bottleneck when performing computational fluid dynamics (CFD) simulations of combustion systems is the computation and integration of the highly non-linear and stiff chemical source terms. In recent times, machine learning has emerged as a promising tool to accelerate combustion chemistry, involving the use of regression models to predict the chemical source terms as functions of the thermochemical state of the system. However, combustion is a highly nonlinear phenomenon, and this often leads to divergence from the true solution when the neural network representation of chemical kinetics is integrated in time. This is because these approaches minimize the error during training without guaranteeing successful integration with ordinary differential equation (ODE) solvers. In this work, a novel neural ODE approach to combustion modeling, ChemNODE, is developed to address this issue. The source terms predicted by the neural network are integrated during training, and by backpropagating errors through the ODE solver, the neural network weights are adjusted accordingly to minimize the difference between the predicted and actual ODE solutions. It is shown that even when the dimensionality of the thermochemical manifold is trimmed to remove redundant species, the proposed approach accurately captures the correct physical behavior and reproduces the results obtained using the full chemical kinetic mechanism.


Introduction
Chemical kinetic mechanisms for practical hydrocarbons fuels can contain hundreds of species and thousands of chemical reactions that describe the evolution of the species in time.
Solving for these species in space and time is computationally challenging due to the wide Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 4 February 2021 range of spatio-temporal scales over which turbulent combustion occurs. Therefore, solving chemistry remains the main bottleneck in computational fluid dynamics (CFD) simulations of combustion systems. For practical engine-size geometries of interest to the automotive and aero-propulsion applications, very simplified reduced or skeletal kinetic mechanisms are typically used to maintain computational tractability. However, these models sacrifice accuracy and predictive capability.
Several studies have applied machine learning to solve these issues. Christo et al. [1] used artificial neural networks (ANNs) to represent a 4-step H 2 /CO 2 chemical mechanism to perform simulations of a turbulent jet diffusion flame. Sen at al. [2] employed ANNs for modeling chemical kinetics by using them to predict the subgrid species source terms in the large eddy simulation linear eddy model (LES-LEM). In addition to these studies, neural networks have also been used to predict the chemical source terms of data-derived scalars within low-dimensional manifolds [3][4][5]. More recently, Ranade et al. [6] used neural networks to capture the process of pyrolysis of complex hydrocarbons, and Wan et al. [7] applied convolutional neural networks to the direct numerical simulation of a turbulent nonpremixed flame interacting with a cooled wall. The overall methodology of source term predictions using machine learning methods in these previous studies is as follows. First, data is generated by running a simple problem using a complex chemical mechanism. By learning from the examples presented in the dataset from lower-dimensional simulations, the machine learning model learns to predict the source terms as functions of the thermo-chemical state. If the mean squared error between the predicted and actual source terms are below an acceptable threshold with limited scatter on the parity plots, it is assumed that the neural network can be coupled with a numerical solver and integrated to recover the true solution. However, this is seldom the case in practice. Oftentimes, the predicted solution diverges from the true solution and becomes unstable when coupled with a numerical solver. Since combustion is a highly nonlinear phenomenon, even small errors in the predictions of the source terms, especially if occurring during an early time instance, can lead to very erroneous solutions.
In this study, a novel approach toward machine learning-based calculation of chemical kinetics is proposed. As opposed to separating the learning and numerical validation phases, the approach used in the study combines them. This novel approach, termed as ChemNODE, captures chemical kinetics using a recent class of deep learning models known as neural ordinary differential equations (NODEs) [8]. ChemNODE calculates the loss function based on the actual and predicted solutions and directly learns to predict the source terms that lead to an accurate ODE solution. A description of the ChemNODE approach and demonstration studies for a canonical homogeneous auto-ignition problem are presented in this paper. The paper ends with some directions for future studies.

ChemNODE approach for chemical kinetics
This study uses NODEs to learn to predict the evolution of chemical species. During training, an ANN is used to learn the chemical source terms of species as functions of the temperature and species mass fractions. A neural network is a machine learning model that is loosely inspired by biological neural networks. It consists of successive layers that are obtained by performing linear operations and non-linear transformations of the preceding layers. The value of the jth hidden layer is given by: In Eq. (1), W and b represent the weights and biases of the neural network, and f is the activation function. These weights and biases are initialized using a desired probability distribution, and are progressively tuned during training. In other words, the process of training a neural network is an optimization problem, where the goal is to find the weights and biases that minimize a loss function of interest. For regression-type problems, this loss function is typically a measure of the error between the predicted and actual values of the target variable.
The conventional practice when developing machine learning models for reacting flow simulations starts by generating data that covers a space of interest. While many earlier proofof-concept studies performed training and validation studies using the same CFD configuration, more recent studies have generated data using approaches such as stochastic micro-mixers [7] and a variant of the pairwise mixing stirred reactor [5,9]. Using these simpler and less expensive simulations, snapshots of data are collected at different points during the simulations. This data is used to build a database that consists of various thermochemical species and their corresponding source terms. After performing some process of manifold dimensionality reduction, either by combining or eliminating variables, a neural network is trained to learn the source terms as closed-form functions of the thermochemical state by using the database generated from inexpensive simulations as an example. After training, the neural network library is coupled with a numerical solver to compute the source terms during unsteady CFD simulations. The problem is one of finding the weights and biases, and minimize the following loss function: In the above equation, ̇ refers to the chemical source terms obtained from the full chemical mechanism, while ̇ corresponds to the chemical source terms predicted by the machine learning model. On the other hand, the approach used in this study differs in that it involves finding the weights and biases that minimize the following loss function: In Eq. (3), and are vectors containing the solutions to the thermo-chemical system of equations, as obtained from the full chemical mechanism and ChemNODE, respectively. As There are a number of ways in which this has been done in the literature. One approach involves a method known as adjoint sensitivity analysis [11], which involves framing an auxiliary ordinary differential equation whose solution gives the derivatives of with respect to the neural network parameters. The solution can be obtained by solving this auxiliary ODE backward in time [8], but this approach suffers from extreme errors under certain conditions.
The ODE can also be solved by performing multiple forward passes [12,13], a process that can be made more efficient by using a checkpointing scheme [14]. In this study, due to the small size of the neural networks, the sensitivity is calculated using a forward mode automatic differentiation [15] using the implementation in Julia's ForwardDiff.jl package [16].
Numerical integration was performed using an A-L stable stiffly-accurate 4th order ESDIRK method from Julia's DifferentialEquations.jl library [17]. The neural networks were optimized using an in-house Julia code, which implements a variant of the Levenberg-Marquardt algorithm [18]. The ChemNODE approach used in this study is illustrated in Fig.   1. Figure 1. Illustration of machine learning approach used in this study.

Results
As an initial validation study to test the capabilities of ChemNODE to accurately capture chemical kinetics, the problem of a simple homogenous zero-dimensional reactor at constant pressure is considered in this study. This system has no convective and diffusion terms, and is described by: In this work, ̇ , which is obtained from chemical mechanisms, is replaced ̇ = ( , , ) where represents a neural network. Separate neural networks, each consisting of a single layer and 10 neurons, were trained for each species. In this study, the results are based on hydrogen-air combustion at 1 atm. The composition space used for training of the neural network involved running the reactor to steady-state. The initial temperature, Ti, was varied between 950 K and 1200 K, while the equivalence ratio was varied from 0.5 to 1.5. The chemical kinetic mechanism used for training comprised of 9 species and 19 chemical reactions [19]. Before training, the species H, HO 2 , and H 2 O 2 were excluded due to these species having the lowest maximum mass fractions. This led to a thermochemical vector of = ( , , , , , , ) .        Finally, Fig. 6 shows a comparison of the time taken to advance the solution to steady-state using ChemNODE and the full chemical mechanism with initial conditions selected across various values of Ti and Φi. The heights of the bars in the plot represent how long the different methods for calculating source terms take to reach the final time step. Even for a small mechanism involving hydrogen-air combustion used in this study, ChemNODE leads to a speed-up of about 2.3, compared to using the full mechanism that uses the Arrhenius reaction rates. It can, therefore, be expected that for higher hydrocarbon fuels with several species and chemical reactions, much more significant savings can be achieved with the NODEembedded chemical kinetic solver. Future work will involve validation of this approach for complex hydrocarbon fuels, and demonstration in three-dimensional CFD simulations of internal combustion engines and gas turbine combustors.

Conclusion
In this study, a novel neural ordinary differential equation approach to predicting the evolution of chemical species in combustion systems was presented. The approach employs a neural network to learn the appropriate source terms that lead to the correct ODE solution.
By calculating the sensitivities of the ODE solution to the neural network parameters, the weights and biases of the neural network were progressively adjusted to obtain an accurate solution. The ChemNODE approach was used to learn the source terms for a zerodimensional homogeneous constant pressure reactor with hydrogen-air combustion. The results showed that ChemNODE was able to capture the correct time evolution for all species under the conditions considered. It was also shown that the ignition delay, and its variation as a function of initial equivalence ratio and temperature, was well predicted. Lastly, the results demonstrated that ChemNODE was about 2.3 times faster than full hydrogen-air chemical mechanism, indicating its promise for providing even more significant savings if applied to higher hydrocarbon fuels with more complex chemistry and larger kinetic mechanisms. Future work will involve validation of the approach for complex hydrocarbon fuels, and demonstration in three-dimensional CFD simulations of internal combustion engines and gas turbine combustors.