Lightweight Explainable Physics-Informed Neural Networks by Learnable Activation Function and Contextual Modulation

1. Introduction

Partial differential equations (PDEs) are fundamental for describing physical phenomena across science and engineering. Physics-informed neural networks (PINNs) have emerged as a scientific machine learning framework for solving forward and inverse problems involving nonlinear PDEs [1,2,3]. By embedding governing physical laws directly into the loss function, PINNs can approximate PDE solutions and assimilate sparse observations without relying entirely on traditional grid-based numerical solvers.

Despite their success, improving PINN accuracy often relies on scaling the multilayer perceptron (MLP) backbone by increasing depth or width. This can introduce substantial computational overhead and optimization pathologies, including stiff gradients, numerical instability, and convergence failures [4]. Recent studies also show that compact or shallow PINN architectures can maintain favorable training efficiency and avoid some optimization plateaus that affect deeper networks [5,6]. This motivates lightweight PINN designs that improve expressivity without aggressive architectural scaling.

The goal of this study is to investigate how to improve performance while keeping the size of the PINN MLP small. Instead of pursuing deeper networks, we explore whether performance and explainability can be improved by providing fine-grained, dynamic modulation to compact networks.

We introduce two mechanisms for lightweight PINNs. First, we use the Cannistraci-Muscoloni-Gu Generalized Logistic-Logit Function (CMG-GLLF) [7] as a layer-wise learnable activation function. To make CMG practical in the PINN context, we reformulate the implicit logit-phase approximation into a fully explicit and differentiable expression through a one-step Newton approximation. Second, we integrate transponder contextual modulation [8], which modulates hidden-layer outputs with respect to the network inputs. Together, these mechanisms allow compact PINNs to learn task-adaptive activation and modulation patterns with a small parameter overhead.

2. Methodology and Training Setups

To evaluate whether compact PINNs can be improved without aggressive depth or width scaling, we compare standard multilayer perceptrons (MLPs) with variants using the CMG-GLLF learnable activation function [7], transponder contextual modulation [8], and the PirateNet baseline [4]. Experiments use three standard PDE benchmarks: Burgers’ equation, the Allen-Cahn equation, and a diffusion-reaction equation [1]. These tasks cover transport with sharp gradients, phase-field interface dynamics, and coupled diffusion with local reaction kinetics.

2.1. Network Architectures

The eight network structures evaluated in this study are:

1.

Vanilla MLP: The original MLP with no additional structure.

2.

Vanilla MLP (match): A widened MLP matching the parameter count of Transponder-NS, used as a parameter-matched baseline.

3.

CMG layer-wise: An MLP where the standard tanh activation is replaced by explicit CMG-GLLF. All neurons in the same hidden layer share one CMG curve, adding only two trainable parameters per hidden layer: the inflection rate $\mu$ and the deviate inflection point I.

4.

Transponder-NS: An MLP with transponder modulation using both node-wise and scalar-wise modulators. The modulator hidden dimension is 8. Scalar-wise modulation provides layer-level control, while node-wise modulation provides channel-level adjustment of hidden representations.

5.

Transponder-S: An MLP with transponder modulation using scalar-wise modulation only.

6.

CMG-TNS: An MLP combining layer-wise CMG activation with Transponder-NS modulation.

7.

CMG-TS: An MLP combining layer-wise CMG activation with Transponder-S modulation.

8.

PirateNet: A physics-informed residual adaptive network using global U and V modulators and adaptive residual connections controlled by a trainable parameter $\alpha$ initialized to zero [4].

2.2. Optimization Process

PINNs are commonly trained with full-batch optimization, and MLPs are the standard neural function approximators in many PINN implementations. Following this convention, we use a two-stage training process. The first stage uses Adam [9] to reach a favorable loss basin, and the second stage uses a fixed-budget L-BFGS optimizer [10] to refine the solution. Learning rates for standard network weights and special CMG parameters are searched independently over the same grid. The training configurations are based on DeepXDE PINN tutorials [11]. Each experiment is run with five random seeds, and the reported metric is the mean relative L2 error.

Experiments are conducted on a platform with an Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz and four NVIDIA A6000 GPUs.

Table 1. Hyperparameter configurations for PINN training. Diffusion-Reaction uses the updated three-hidden-layer network from the write guide.

Task	Vanilla MLP structure	Adam iters	L-BFGS iters	Learning rates
Burgers	[2, 20, 20, 20, 1]	15,000	15,000	2e-3, 1e-3, 5e-4
Allen-Cahn	[2, 20, 20, 20, 1]	40,000	15,000	2e-3, 1e-3, 5e-4
Diffusion-Reaction	[2, 30, 30, 30, 1]	20,000	None	2e-3, 1e-3, 5e-4

Table 1. Hyperparameter configurations for PINN training. Diffusion-Reaction uses the updated three-hidden-layer network from the write guide.

Task	Vanilla MLP structure	Adam iters	L-BFGS iters	Learning rates
Burgers	[2, 20, 20, 20, 1]	15,000	15,000	2e-3, 1e-3, 5e-4
Allen-Cahn	[2, 20, 20, 20, 1]	40,000	15,000	2e-3, 1e-3, 5e-4
Diffusion-Reaction	[2, 30, 30, 30, 1]	20,000	None	2e-3, 1e-3, 5e-4

3. Results

3.1. Derivation of the Explicit CMG-Newton Formulation

The Cannistraci-Muscoloni-Gu Generalized Logistic-Logit Function (CMG-GLLF) provides a unified mathematical framework for trainable activation functions, allowing control over steepness, asymmetry, and input/output boundaries [7]. The original logit-phase formulation lacked a compact explicit expression and relied on implicit approximation to invert the logistic curve. In PINNs, where PDE residuals require repeated differentiation with respect to spatiotemporal inputs, implicit inversion can increase computational cost and introduce instability in gradient computation.

To address this bottleneck, we use a one-step Newton approximation to obtain an explicit, differentiable CMG formulation. Let $x\in [x_{min},x_{max}]$ be the input and $y\in [y_L,y_R]$ be the mapped output limits. Define the normalized input $t\in [0,1]$ as

$$t={\displaystyle \frac{x-x_{min}}{x_{max}-x_{min}}}.$$

(1)

The unified explicit CMG expression, with inflection rate $\mu \in [0,1]$ and deviate inflection point $I\in [0,1]$, is

$$CMG\left(x\right)=\left\{\begin{array}{cc}{y}_L+(y_R-y_L){\displaystyle \frac{1}{1+\frac{1-t}{t}exp\left[-\left(\frac{1}{\mu }-2\right)(t-I)\right]}},\hfill & 0<\mu \le 0.5,\hfill \\ y_L+(y_R-y_L){\displaystyle \frac{t+\left(\frac{1}{1-\mu }-2\right)It(1-t)}{1+\left(\frac{1}{1-\mu }-2\right)t(1-t)}},\hfill & 0.5<\mu <1.\hfill \end{array}\right.$$

(2)

This formulation preserves the flexible steepness and asymmetry control of the implicit CMG framework while allowing gradients to be computed by standard automatic differentiation. The derivation is provided in Appendix Section 5.1.

3.2. Empirical Validation of CMG-Explicit

We first validate CMG-Explicit as a node-wise hidden-layer activation function on the Burgers’ equation benchmark. Table 2 shows that CMG-Implicit degrades performance relative to tanh, while CMG-Explicit substantially decreases mean relative L2 error. This supports using CMG-Explicit as the default CMG implementation in the remaining experiments.

We then compare node-wise and layer-wise CMG-Explicit configurations. In the layer-wise configuration, all neurons in a hidden layer share the same trainable CMG parameters.

Table 3. Performance comparison of CMG node-wise and layer-wise configurations on Burgers’ equation.

Method	Mean relative L2 error
CMG Node-wise	6.34E-03
CMG Layer-wise	8.83E-03
Tanh	2.10E-02

Table 3. Performance comparison of CMG node-wise and layer-wise configurations on Burgers’ equation.

Method	Mean relative L2 error
CMG Node-wise	6.34E-03
CMG Layer-wise	8.83E-03
Tanh	2.10E-02

Both CMG configurations outperform tanh. Since the layer-wise configuration adds only two trainable parameters per hidden layer, it provides the better accuracy-parameter trade-off and is used in the following experiments.

3.3. Evaluation of Transponder and CMG on Different PINN Tasks

Table 4 reports relative L2 errors on Burgers, Allen-Cahn, and Diffusion-Reaction, along with mean rank, worst rank, and parameter counts. Ranks are computed within each task according to relative L2 error, where lower error receives a better rank.

Compared with Vanilla MLP, CMG layer-wise improves Burgers and Allen-Cahn while adding only six parameters to the three-hidden-layer network. Transponder-NS consistently improves over the parameter-matched Vanilla MLP across all three tasks, achieves the best Burgers error, and obtains the best mean and worst ranks. Transponder-S also improves over the matched Vanilla MLP on Burgers and Allen-Cahn, but its gains are smaller than those of Transponder-NS, consistent with the removal of node-wise modulation.

Combining CMG with Transponder-NS gives the best Allen-Cahn result and the second-best ranking profile overall. PirateNet obtains the best Diffusion-Reaction error, but it does not outperform the matched Vanilla MLP on Burgers and has one of the lowest worst ranks. These results indicate that CMG can reduce error at extremely low parameter cost, while transponder modulation provides consistent gains for lightweight PINNs.

3.4. Explainability of CMG and Transponder Design

To examine why CMG and transponder modulation can outperform Vanilla MLP and PirateNet, we analyze learned CMG activation parameters and transponder gate values in the CMG-TNS model on Allen-Cahn. This claim is restricted to Allen-Cahn and should not be read as proving that the same parameter distributions occur for all PDE systems.

Figure 1 shows that the learned CMG parameters differ across layers, indicating that different depths require different activation-value modulation. A fixed tanh activation cannot provide this layer-specific functional adaptation. The transponder gate values also vary across layers and nodes: scalar-wise modulation separates layers, while channel-wise modulation varies within each layer. This fine-grained, local modulation explains why CMG and transponder designs can improve over Vanilla MLP and over global modulation mechanisms such as PirateNet in the Allen-Cahn setting.

4. Conclusion

This study investigated a route for improving PINN accuracy and robustness without relying on aggressive MLP scaling. We introduced an explicit, differentiable CMG-GLLF formulation as a parameter-efficient layer-wise learnable activation function, and combined it with transponder contextual modulation for input-dependent hidden-state modulation.

Across Burgers, Allen-Cahn, and Diffusion-Reaction benchmarks, these mechanisms improved lightweight PINNs at low parameter cost. CMG layer-wise improved over Vanilla MLP on Burgers and Allen-Cahn while adding only two trainable parameters per hidden layer. Transponder-NS consistently improved over the parameter-matched Vanilla MLP and achieved the best overall ranking profile, while CMG-TNS achieved the best Allen-Cahn performance. PirateNet obtained the best Diffusion-Reaction error but was less stable across tasks.

The Allen-Cahn parameter analysis further supports the explainability of the design. Learned CMG parameters and transponder gate values differed across layers and nodes, indicating localized activation and modulation requirements that fixed activations and global modulation cannot provide. These results suggest that learnable activation functions and contextual modulation offer a practical path toward PINNs that are lightweight, accurate, and explainable.

The main limitation of this study is that the empirical evaluation covers only three PINN benchmark tasks. Although Burgers, Allen-Cahn, and Diffusion-Reaction equations represent different PDE behaviors, broader validation across more PINN tasks, higher-dimensional systems, inverse problems, and real-world scientific computing settings is needed before making stronger claims about generality. The broader impact of this work is primarily methodological: more accurate lightweight PINNs could reduce the computational cost of scientific machine learning and make PDE-based modeling more accessible.

A. Technical Appendices

A.1. Detailed Derivation of the CMG-Explicit Formulation

When $0<\mu \le 0.5$, CMG-GLLF is in the logistic phase and admits an explicit analytical expression. When $0.5<\mu <1$, CMG enters the logit phase, which is defined as the inverse of the corresponding logistic-phase mapping. Because an exact inverse in a simple closed form is not available, we derive an explicit approximation by applying one Newton step [12].

Normalization.

Let $x\in [x_{min},x_{max}]$ be the input and $y\in [y_L,y_R]$ be the mapped output limits. We normalize both axes to $[0,1]$:

$$t={\displaystyle \frac{x-x_{min}}{x_{max}-x_{min}}},s={\displaystyle \frac{y-y_L}{y_R-y_L}}.$$

(3)

Logistic-phase expression.

In normalized coordinates, the logistic-phase CMG can be written as

$$s=\sigma (logit(t)+\alpha (t-I\left)\right),$$

(4)

where $\sigma \left(z\right)=1/(1+exp(-z\left)\right)$, $logit\left(t\right)=log(t/(1-t\left)\right)$, and $I\in [0,1]$ is the deviate inflection point. In the logit phase,

$$\alpha ={\displaystyle \frac{1}{1-\mu }}-2,0.5<\mu <1.$$

(5)

Inverse problem.

To derive the logit-phase expression, we invert the normalized function:

$$t=\sigma (logit(s)+\alpha (s-I\left)\right).$$

(6)

Applying the logit transform gives

$$logit\left(t\right)=logit\left(s\right)+\alpha (s-I).$$

(7)

Define

$$F\left(s\right)=logit\left(s\right)+\alpha (s-I)-logit\left(t\right).$$

(8)

Solving $F\left(s\right)=0$ gives the logit-phase output.

One-step Newton approximation.

Using $s_0=t$ as the initial guess, one Newton step gives

$$s_1=s_0-{\displaystyle \frac{F\left(s_0\right)}{F^{\prime }\left(s_0\right)}}.$$

(9)

At $s_0=t$,

$$F\left(t\right)=\alpha (t-I),F^{\prime }\left(t\right)={\displaystyle \frac{1}{t(1-t)}}+\alpha .$$

(10)

Therefore,

$$s_1=t-{\displaystyle \frac{\alpha (t-I)}{\frac{1}{t(1-t)}+\alpha }}={\displaystyle \frac{t+\alpha It(1-t)}{1+\alpha t(1-t)}}.$$

(11)

Final expression.

Returning to the original coordinates yields the explicit logit-phase approximation:

$${CMG}_{\mathrm{logit}}\left(x\right)\approx y_L+(y_R-y_L){\displaystyle \frac{t+\left(\frac{1}{1-\mu }-2\right)It(1-t)}{1+\left(\frac{1}{1-\mu }-2\right)t(1-t)}}.$$

(12)

This expression avoids grid-based implicit inversion and supports direct automatic differentiation in PINN training.