Machine Learning Approaches for Inverse Problems and Optimal Design in Electromagnetism

1. Introduction

Computerized analysis is frequently used to recover field sources or device structure from measurements using numerical computations [1,2]. This process implies the repeated resolution of an electromagnetic problem, under different trial values of inputs (either sources or system parameters). The presence of measurement noise, the consideration of manufacturing and assembly tolerances, the inclusion of ferromagnetic materials or the consideration of complex, three-dimensional geometries add relevant computational efforts to the process. Several measures have been proposed to simplify or speed-up the analysis, basically trading off accuracy in the evaluation of the distance of trial data from the actual measurements with promptness.

Alternatively, Machine Learning (ML) and Deep Learning (DL) models can be used for a straightforward resolution of inverse problems or optimal design ones, trying to train NN to build an (approximated) relationship between the desired output (e.g. sensors reading) and the trial values of the degrees of freedom (e.g. radii or currents of coils, when considering magnets) starting from available examples of the desired output [3,4,5,6,7]. Examples can be obtained by solving a reduced set of instances for the computationally demanding problem, or even be extracted for experimental data.

Note that the construction of an approximate model from available experimental data could reveal the sole possibility in cases where a theoretical problem formulation is not available. This could happen either when a model based on the laws of physics is not available or too complex to be reduced to a set of equations manageable in the due course of an iterative process.

In addition, the resulting problem, either the inverse one or the design one, shows an ill posed nature, being prone to multiple solutions [8,9]. This aspect has been counteracted in many ways, and some precaution must be taken also when using data-driven models.

In previous works the authors proposed several ML models both for solving the direct [10,11,12,13,14] model and the inverse model [15,16,17], focusing the attention on the optimization through the use of the direct model, and highlighting the difficulty behind the inverse models.

Data-driven models can be obtained following different approaches [8,10], including the classical statistical regressions or the most modern Deep Neural Networks. Each approach presents advantages and drawbacks, which must be comparatively assessed. In addition, the hyper-parameters shaping the approach (e.g. the number of hidden neurons in the NN case) must be carefully chosen to obtain the best balance between promptness and accuracy. This work presents a detailed analysis of the learning process for a set of NN models of a benchmark inverse problem. The data used to train the model are generated by FEM or analytical formulations.

The main aspects investigated in this work include:

• Training and testing ML approaches to solve direct and indirect electromagnetic problems;
• Selection of the ML model;
• Selection of the model hyper—parameters;
• Dataset generation;
• Regularization approaches;
• How machine learning treats ill posed inverse problems.

In the following, with reference to a simple yet representative benchmark problem, we will compare the standard Shallow Neural Networks (SNN) [18,19] with a more recent model, namely Convolutional Neural Networks (CNN) [20,21]. For the sake of comparison, also a support vector machine, as an example of regression model, will be introduced.

2. The Benchmark Problem

To compare different approaches, we adopt here the benchmark problem TEAM 35 [22]. A multi-turn air-cored coil is considered. The coil is composed of n_t=20 independent turns. The width of each turn is w = 15 mm and the height is h = 10 mm. Hollow turns are assumed to allow for water circulation. In the following we will assume symmetric currents distribution among the uppermost 10 coils and the lowermost ones. Consequently, only half of the model is needed to compute the field (see Figure 1a). For evaluating the field, a two-dimensional controlled region is considered, (delimited by curves S₁, … S₃ in Figure 1a). The two components B_r and B_z along axes r and z of the flux density field B are sampled on a grid evenly spaced in a square region with a side length of 60 mm, which is denoted as the Region Of Interest (ROI). Underlying sensors could be, e.g., Hall Probes.

To train the shallow neural network and the support vector regressor, we firstly consider a grid of n_p=10×10 field points, taking the values along the boundaries S1, … S3 and the line γ, while for the training of the convolutional neural network we consider a grid of n_p=20×20 field points, taking all the internal values of the square region of interest.

The magnetic analysis is based on the Finite Element (FE) method. An example of field map is shown in Figure 1b.

In the TEAM 35 version considered here, the aim of the inverse problem is to identify the radii and the currents in the coils to generate a prescribed flux density map B(r,z). In the original formulation, the map is uniform, with $B=B_0\widehat{z}$, within a region adjacent to the symmetry plane z=0 (lines S₁, S₂ and S₃ in Figure 1), and with an amplitude as small as possible along an external segment of the symmetry axis (line γ in Figure 1). To evaluate the field uniformity in the inner ROI, the magnitude of B is “measured” in N_p = 30 field points, evenly spaced on the boundary of the ROI. On the other hand, in order to guarantee the minimum field amplitude on γ, the field is “measured” on N_k = 10 points along γ. In particular, the set of measurements includes 60 real values, which correspond to 10 B_r and 10 B_z measurements evenly spaced along lines S₁ and S₂ respectively, 10 B_z values measured along line S₃, and 10 B_z values along line γ. In this paper, we maintain a similar formulation of the inverse problem, where we aim to identify the radii and the currents correspondent to randomly generated flux density measured in the measurement’s points.

Figure 2 shows the main characteristics of the variables involved in the problem. Note that when radii of all coils are known, the relationship between currents and flux density values is linear. The matrix mapping the current in each coil onto each single measurement is known as Lead Field Matrix (LFM). On the other hand, when the radii are among the degrees of freedom, the inverse problem gets non-linear, and the LFM must be, in principle, be re-assembled for each trial configuration. One of the advantages of the ML approaches is that the reassembly of the LFM is not required, as the NN directly extracts from data the relationship between field values (measurements) and the degrees of freedom (radii and currents).

More details on the benchmark problem geometry can be found in [10].

3. Considered Machine Learning Models

The machine learning models analyzed in this work will be used in different modalities:

-

a “Direct Problem” (DP), where examples of the radii/currents set and of the corresponding flux density values are used to create a model able to generate the target field map, hence replicating the LFM from the radii. The direct model is then used within traditional optimization or inverse problem resolution algorithms;

-

a first class of “Inverse Problem” (IP#1), where measurements (inputs) and currents (outputs) are used to create a model of the underlying linear map. In this case, radii are assumed known, and the linear model is related to the (pseudo-)inverse of the LFM.

-

a second class of “Inverse Problem” (IP#2), where the currents are known and provided as input together with the measurements, while the outputs are the radii;

-

finally, a third class (IP#3), where both currents and radii (the outputs) must be recovered from field measurements, which are the only inputs in this case.

Figure 3 shows the input and output of each defined problem. As anticipated, different ML approaches are available to create data-driven behavioral models. In this study, we have considered the following ones:

• Shallow Neural Networks (SNN) [18,19]: this is the standard approach: an artificial neural network with a single hidden layer with logistic activation functions. In the DP, Ni input neurons (corresponding to the radii), Nh hidden neurons and No output neurons (the elements of the LFM) are trained to provide the output. While Ni=10, the LFM will be represented by a reduced number of real numbers, corresponding to the main components in a PCA analysis based on the correlation analysis of the measurements. The full matrix is then recovered by exploiting the components identified in the PCA. In the inverse problems IP#1, IP#2 and IP#3, the SNN is used a straightforward solver, and the meaning of output neurons depends on the type of considered problem. This point will be further discussed below.
• The number Nh of hidden neurons is varied to assess model capabilities. The different SNN’s are trained using Levenberg-Marquardt Bayesian Regularization approach [23], which minimizes the weights together with the discrepancy on the data. Early topping is performed by means of worsen performance on a validartion set.
• Convolutional Networks (CNN) [20,21]. This class of NN will be considered for IP#3 only. A higher number of hidden layers is present here, and the network can be named “deep”. The inner layers are classified as “convolutional” and “pooling”, with different associated actions on the data. The activation functions are in this scheme the “ReLU” functions. This model has an intrinsic capability of building a reduced order inner model, which can be exploited to cope with the highly correlated nature of the input, represented by the flux density map in the ROI. The ADAM algorithm is used to train the CNN.
• In order to compare ML with more traditional statistical regression approaches, we have also considered Support-Vector Regression (SVR) [24,25] for the inverse problems. SVR training algorithm builds a linear model in a higher dimensional space exploiting the so called “kernel trick” by minimizing a quadratic objective function which is a combination of the Euclidean norm of the weights of the linear model and the sum of the so called slack variables, which represent a threshold of the maximum absolute deviation between the predicted and target values. The LIBSVM implementation of the SVR was employed

The hyper-parameters of the SNN and SVR models where selected using an exhaustive K-Fold Grid Search Cross Validation approach, using K=10 and a defining adequate ranges for the optimized hyper-parameters. The CNN architecture was selected using a trial and error heuristic approach, which is common for complex deep learning models.

Figure 4. Representation of a random geometry, coils numbering and ROI.

Figure 4. Representation of a random geometry, coils numbering and ROI.

4. Results

To assess the different models, 26000 examples have been generated using a FEM solver to compute the flux density in the measurement points. Examples have been arranged according to radii values, to identify different subsets with the same LFM, and divided into Training and Test sets, according to the 70/30 rule.

4.1. Direct Problem

In a first study we trained different SNN on the DP. The metric adopted here to compare the accuracy of the different predicted currents or radii is the Normalized Mean Absolute Error% (NMAE%).

$$\mathrm{N}\mathrm{M}\mathrm{A}\mathrm{E}\mathrm{\%}=100\frac{\frac{1}{N_{test}}\sum _{N_{test}}\left|Y_{pred}-Y_{true}\right|}{\mathrm{m}\mathrm{a}\mathrm{x}\left(Y_{true}\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(Y_{true}\right)}.$$

NMAE% is an alternative metric that overcomes the limitations of more classical Mean Absolute Percentage Error in situations involving data that can be negative or close to zero. NMAE normalizes the error by dividing it by the range of the actual values, providing a more balanced measure of accuracy.

Figure 5a shows that the training of a SNN with 36 hidden neurons for the direct problem required 10 hours of computation in a 12 cores processor, while the convergence was obtained after 4 hours. The similar trend of the training and validation errors suggests that no the SNN is not overfitting. Figure 5b shows a test set pattern prediction, with respect to the target value, indicating that the prediction is qualitatively good. The NMAE% for the test set of the direct problem was 0.15%, indicating that the direct problem can be solved with good accuracy.

4.2. Inverse Problem #1

We then solved the inverse problem of type IP#1 by computing the Truncated-Pseudo-Inverse (TSVD) of either the SNN LFM and of the actual LFM, computed using FEM. Just 8 singular modes are used in this investigation. The currents obtained using the two inverse matrices are used to compute the flux density in the measurement points, and the results are compared with the original measurements.

Table I reports the test set NMAE% (normalized to 80A) of the 10 currents for the IP#1, solved by means of the SNN.

From Table 1 it can be observed that the NMAE% increases with the distance of the coil from the sampling curves S₁..S₃. The averaged NMAE% value is 1.66% In this case a SNN with 27 hidden neurons performed best. The average NMAE for SVR is 2.35%. SVR with Gaussian kernel has 3 real hyperparameters and searching them is difficult because a single training requires hours: better results were obtained with SNN.

The flux density in the “measurement” points corresponding to the different approaches are reported in Figure 6. The scatter plots shown in Figure 7 shows that the problem of predicting the currents can be solved by the neural networks accurately, and the dispersion increases with the distance for them sampling curves.

4.3. Inverse Problem #2

Very similar results are obtained for the inverse problem IP#2, reported in Table 2, again showing an increase of NMAE% with the distance from the measurement region. The averaged value is 10.34% for the SNN and 14.1% for the SVR. In this case a SNN with 35 hidden neurons performed best.

Figure 8. Regression plots of the predicted vs true radii for IP#2.

Figure 8. Regression plots of the predicted vs true radii for IP#2.

Figure 5 evidences that the problem of predicting the radii is more difficult, as expected.

4.4. Inverse Problem #3

Finally, for the inverse problem IP#3, the results for the currents confirmed the influence of the distance from the ROI, but with a slight recover in coils 9 and 10 when influence of γ line starts to be relevant. On the other hand, the radii show a completely different behavior (see Table 3).

It is interesting to show the scatter plot of true vs. predicted radius for coil 1 for all the test cases (Figure 9). Basically, the SNN keeps the estimate of the radius equal to the average value, regardless of the actual value. This is due to the mutual role of current and radius in this problem. If considering the expression of the flux density on the axis due to a filamentary coil:

$$B_{axis}=\frac{{\mu }_0}{2\pi }\frac{I}{R},$$

(1)

where I is the coil current and R its radius, we can conclude that the relevant figure is the ratio of current and radius. Starting from this, we can hypothesize that the SNN tries to regularize this ill-posed problem by keeping the radius constant, and leveraging on the currents only to fit the measurements.

Figure 9. catter plot of true and predicted values of current and radius of coil 1.

Figure 9. catter plot of true and predicted values of current and radius of coil 1.

The above results for IP#3 where obtained with a SNN with 35 hidden neurons. We addressed the same problem using a CNN, but taking as inputs the whole Br and Bz images inside the ROI, considering a grid of 20x20.

The total number of inputs of the CNN is then the whole 2D ROI region 800 (as shown in Figure 10), compared to the 60 inputs along the ROI border used by the SNN, while the output of both methods are 10 radii values and 10 current values (as shown in Table 3). The architecture of the CNN is shown in Figure 11. The training of the CNN required 20 hours, while the training of the SNN and the SVR for the previous problems required 10 hours and 4 hours respectively. The results obtained with the CNN were almost identical to the results of the SNN, shown in Table 3 and Figure 6. Considering that the SNN adopts only the boundaries of the ROI as inputs and that the CNN considers the whole 2D domain, this result can be interpreted as a confirmation of the Dirichlet theorem for harmonic functions.

4.4. Inverse Problem #3: forward solution of inverted patterns

In order to assess the effectiveness of the radii and currents solutions obtained by the neural network for inverse problem IP#3, apart from calculating the errors in Table 3, we used these solutions as the input of the direct solver by means of a Finite Element Method, and compared the resulting field distribution with the input field distribution.

As a result, the NMAE% was 4.05%, where the normalization was performed against the mean absolute value of field inside the ROI.

Figure 12 shows three test examples of the true field values compared to the reconstructed field values using the FEM solution of the geometry obtained solving the inverse problem IP#3 with the SNN. From these results it can be stated that even if the error of the predicted radii is large, as can be observed in Figure 6 and Table 3, the combination of the predicted radii and currents represent a good source identification by means of the resulting field that is obtained applying the direct problem. The leverage of the SNN (and the CNN as well) consisting in applying the nonlinearities only to the current estimation appears to be an automatic way for regularizing the difficult inverse problem IP#3.

5. Discussion

The availability of different machine learning models for the prompt resolution of electromagnetic problems, as required in the inverse problems from electromagnetic measurements or in the optimization of electromagnetic devices, calls for a reciprocal comparison in terms of accuracy and promptness of the models, both in the training and operation phases. We proposed here a comparison among a few neural models on a well-known benchmark problem, considered here as an inverse problem.

The correlation among measurements from each sensor location (assessed on 26000 randomly distributed examples) shows that in the benchmark problem redundancy is present, although this redundancy is well managed by the machine learning and deep learning models, as shown by the results.

In the linear direct problem DP, the reconstruction of lead field matrix is straightforward, while for the three inverse problems, care must be taken in the choice of the regression algorithm. In the latter case IP#3, an interpolating model is achieved anyway, but generalization capability could be probably improved using “noisy” data. Neural network approach to linear inverse problem also requires careful choice of the learning strategy. As a remark, an accurately selected and trained SNN was shown to be a powerful model to solve the inverse problems, both linear and non-linear. The results obtained with the SNN and CNN for IP#3 show that a similar regularization was performed by two different models to approach the problem of infinite solutions of the inverse problem. Even if the radii and currents solutions were strongly regularized, they possessed both feasibility (they were inside the design bounds), and they turned to give good forward solutions when used as input of the direct problem. These results might be imputed more to the dataset itself than to the ML approach, confirming the results obtained in previous works [13]: the choice of the distribution of design parameters used to generate the training set is crucial for the performance of the ML approaches that are, of course, data driven.

Another quite remarkable result, which is new in the literature, is that the inversion of the electromagnetic problem gives the same results when using as input the field distribution in the boundary of the domain and the field distribution in the whole domain. Being the distribution inside the domain in theory dependent on the boundary conditions, that shall not be a complete surprise from a mathematical point of view, but it is not automatically expected that a neural network is able to incorporate this knowledge based on learning from data. The results obtained for the IP#3 solved by means of the SNN, which uses the boundary of the ROI, and the CNN, which uses the whole ROI; show that the SNN was able to acquire from the boundary values the same information that the CNN obtained from internal values. These results find confirmation in the recent development of the physics informed neural networks. For this reason, we are still working on the resolution of electromagnetic inverse and optimization problems with physics informed [26] and generative adversarial (game theory) neural networks [27].

Future work will be focused on the tuning of the NN for the inverse problem with a custom training loop using automatic differentiation. The idea is to back-propagate the error of the reconstructed field obtained by feeding the direct solver with the result of the NN.