Fidex and FidexGlo: From Local Explanations to Global Explanations of Deep Models

1. Introduction

Within Artificial Intelligence (AI), XAI stands for Explainable AI and includes methods that aim to explain the responses of black-box models. The number of publications in XAI is constantly increasing. Extensive reviews of XAI methods have been presented in [1,2,3,4,5]. Deep learning models are characterized by many neurons grouped together in many successive layers. Even though they have been very successful over the last decade, they remain complex and difficult to explain. Rule-based systems offer transparency and clarity in decision-making. Models defined by explicit rules, enable human users to trace each decision back to the specific conditions that triggered it. Unlike black-box machine learning models, the behavior of a rule-based system is comprehensible not only to AI practitioners but also to a broader audience, including domain experts. In certain areas of application, such as medicine, it is essential to explain the decisions made by deep models so that physicians and patients can have more confidence in them.

The precursor to deep connectionist models is the Multi-Layer Perceptron (MLP). When it was introduced, researchers raised the question of explainability and developed new techniques to provide explanations using propositional rules. A nomenclature of all methods to explain the responses of neural networks by propositional rules was introduced by Andrews et al. [6]. Before this century, Decision Trees (DTs) were a direct competitor to MLPs in classification problems. Their answers are easy to explain and accurate in many applications. Furthermore, models that are combined to form an ensemble are often more accurate than a single model. Several algorithms for learning ensembles were proposed, such as bagging [7] and boosting [8], and have been applied to DTs and MLPs. However, even DTs lose their explainability in ensemble combination.

A key element of explanatory approaches in the context of deep models is the focus on the neighborhood of a sample [9]. Such approaches are defined as local. A global technique, on the other hand, considers all the data that defines a problem. In addition, many other image classification techniques visualize areas that are most relevant to the result [10].

The Discretized Interpretable Multi Layer Perceptron (DIMLP) is a neural network model from which responses can be explained by propositional rules. The rules are generated by an algorithm that determines axis-parallel discriminative hyperplanes [11]. The same algorithm was used to generate rules from ensembles of DIMLPs [12]. Later it was shown that ensembles of Decision Trees (DTs) are functionally special cases of DIMLP ensembles, so that rule extraction was applied to Random Forests (RFs) and Gradient Boosted (GB) Trees [13]. One question that arose at this point was how to apply rule extraction to deep models, but the complexity of the DIMLP rule extraction algorithm, which is global, was not favorable in practice (polynomial of degree four with respect to the size of the input vectors [12])

We present two novel algorithms that have been recently introduced in [14] to explain the responses of a number of black-box machine learning models for data classification problems. The first is Fidex, which is local and thus applied to a single sample. The second, called FidexGlo, is global and uses Fidex. Both algorithms generate explanations by means of propositional rules. In our framework, the discriminative boundaries are parallel to the input variables and their location is precisely determined. Fidex is a heuristic algorithm that establishes at each step where is the best hyperplane that has increased fidelity the most. Fidelity is a measure of the extent to which the explanation is in agreement with the model response. The algorithmic complexity of Fidex is proportional to the maximum number of iterations, the number of hyperplanes, which is finite by construction, and the number of samples. In this work, we first applied FidexGlo to ensembles of DTs, ensembles of MLPs and support vector machines (SVMs) to show that its performance on three benchmark problems is competitive in terms of complexity, fidelity and accuracy. Our main contribution here is its application to Convolutional Neural Networks (CNNs), as rule extraction has rarely been applied to this deep model. We obtained accurate results on three classification problems based on images. The characteristics of the generated rules were shown, and several examples of explanations were illustrated with their corresponding pictures. In the following sections we present a brief state of the art on explainability methods, the used models and algorithms, then the experiments with tabular and non-tabular data, followed by a discussion and the conclusion.

2. Explainability Techniques

In this section, we provide a brief description of explainable techniques used with supervised learning models, including deep neural networks. Specifically, we describe a number of visualisation methods, some of which allow the determination of feature relevance and finally rule-based explanations.

2.1. Explainability by Visualization

Many techniques aimed at explaining deep NN decisions in image classification are based on visualizing the areas that are most relevant to the result [15], but as stated by Rudin [16]: “it does not explain anything except where the network is looking”. Grad-CAM, which stands for Gradient-weighted Class Activation Mapping, is a technique designed to highlight the most critical regions of an image used in a deep learning model’s decision-making process [17]. The heat-map shows which pixels from the image are required by the model to perform the classification task. Furthermore, Grad-CAM only shows features that contribute positively to the class of interest. It does not show features that the model uses as evidence against a particular class. Finally, this technique assumes a linear relationship between the final convolutional layer and the output of the model. This may not always be the case, especially for complex models.

Layer-wise Relevance Propagation (LRP) can provide more detailed and accurate explanations than Grad-CAM [18], because it can handle non-linearities in the model. It first calculates the output for a given input; then, during a backward pass, LRP propagates the output of the network (the relevance) back to the input layer. Specifically, the relevance of each neuron is distributed to its inputs according to their contribution to its activation. The resulting relevance scores indicate how much each feature in the input contributed to the final output.

2.2. Explainability by Feature Relevance

An important technique used to determine the relevance of the input features is using Shapley Values [19]. These values are calculated as the average marginal contribution across all possible input combinations. The marginal contribution is the difference between the prediction with and without the feature. Shapley values have desirable properties such as efficiency (the sum of the Shapley values is equal to the difference between the actual and average prediction) and symmetry. A major drawback of this technique is the algorithmic complexity, which is exponential in the number of input variables. However, it is possible to use approximation algorithms. Finally, Shapley scores assume that the features are independent. In real datasets, features can often be correlated, which can affect the calculation and interpretation of Shapley values.

LIME (Local Interpretable Model-Agnostic Explanations) creates a local surrogate model around the prediction to be explained [9]. This local model is a simpler, interpretable model, such as a linear model or a decision tree. LIME generates a new dataset by perturbing the instance to be explained. It then uses the original classifier to predict the outcomes for these perturbed samples. Each perturbed instance is assigned a weight according to its proximity to the original instance. The closer the perturbed instance is to the original instance, the higher its weight. The local surrogate model is fitted to the perturbed dataset and is then used to explain the prediction of the original classifier. At the end, a local surrogate model is created for each prediction to be explained.

2.3. Rule-Based Explanations

The formalism of propositional rules is natural with respect to the logic of human reasoning. Specifically, rules are given as: “if $(x_1<t_1)$ and $(x_2>t_2)$ and … $(x_n<t_n)$ then DECISION”; with $x_i$ corresponding to input variables $(i=1\dots n)$, $t_i$ being real values, and DECISION corresponding to a decision, such as “patient healthy”. For instance, we can say: “this patient is sick, because input variable i is greater than threshold $t_i$ and input variable j is great than threshold $tj$”.

The first taxonomy describing the general characteristics of all techniques for extracting rules from connectionist models was introduced in [6]. Specifically, they distinguished three main categories of methods, namely: pedagogical; decompositional; and eclectic. Pedagogical techniques use a transparent model, such as Decision Trees (DTs) [20] to learn input/output associations, without considering weight values of neural networks. Decompositional methods generate propositional rules by analyzing the weight values. However, most algorithms in this category present exponential algorithmic complexity. To alleviate this difficulty, a pruning step is helpful to reduce the number of weights. Eclectic techniques are both pedagogical and decompositional. Golea proved that the problem of rule extraction from Multi-Layer Perceptrons (MLPs) is NP-hard [21]. Finally, for Support Vector Machines(SVMs), which are functionally equivalent to MLPs with a single hidden layer, a review of many explainability techniques were proposed in [22].

Note that the majority of decompositional techniques suffers from exponential algorithmic complexity [12]. Examples of pedagogical techniques are reported in [23], and [24]. Deep architectures are even more complex, so their opacity with respect to MLPs has increased. Unfortunately, the simplest pedagogical approach of adapting transparent models such as decision trees to deep neural networks generally yields unsatisfactory results in terms of accuracy and fidelity [25].

Although new rule extraction algorithms are always being proposed based on a single MLP, little work has been achieved for ensembles of connectionist models. For example, in REFNE a trained ensemble generated additional samples and then extracted propositional rules [26]. Here, the input variables were discretized during rule extraction. The rules were limited to only three antecedents. Using genetic programming, Johansson presented a technique for generating rules from ensembles of 20 neural networks [27]. Specifically, the problem of rule extraction has been considered as an optimization problem, where a trade-off between accuracy and comprehensibility has to be considered. Moreover, Hara and Hayashi introduced a rule extraction technique for a limited number of MLPs in an ensemble [28,29]. Sendi et al. trained DIMLP ensembles by optimizing their diversity [30]. Rule extraction was then performed on each individual network, and for each sample, the rule with the highest confidence score was selected. Finally, in RE-E-NNES (Rule Extraction with Ensemble of NN Ensembles) the C4.5 decision tree algorithm was used to extract rulesets from MLP ensembles trained by a boosting algorithm (NNBOOST); the number of MLPs in an ensemble was limited to ten [31]. RE-E-NNES generated rules from the first MLP, then these rules were refined with the second MLP and so on with the other MLPs.

Ensembles of decision trees are very good models for data classification. Furthermore, a single tree can be explained by the rules that are generated from it, but many trees in an ensemble lose their transparency and require special explanation techniques. In particular, it is always possible to generate all the rules from all the trees in an ensemble, but since there can be hundreds of trees, the number of rules is too large. Therefore, several techniques try to remove as many rules as possible. For example, “RuleFit” is an algorithm in which a linear function containing rules and features approximates the entire ensemble of DTs, with a large number of coefficients equal to zero [32]. The rule extraction problem in [33] is also viewed as a regression problem, where the rules represent features, and by trying to remove as many rules as possible with the constraint of getting good fidelity and accuracy. Providing propositional rules that verify precision and recall conditions is the main goal of the “Skope-Rules” technique [34]. Similar or duplicate rules are removed based on a similarity threshold of their supports. The final rules are assigned weights that are simply proportional to their out-of-bag accuracy. Finally, the RuleCOSI technique first generates the rulesets from the individual trees of an ensemble and then uses a particular method to prune unnecessary conditions from the rules [35]. This results in a simpler model because the final ruleset is smaller with fewer conditions per rule.

One quality of propositional rules over all other methods described above is that the former are both explanatory and predictive, whereas the latter cannot be used as predictors, but only as descriptors. For example, when classifying numbers represented in images, heat-maps will often give importance to pixels in the region of the number, but this applies to all numbers without being able to differentiate between them. Another example is the diagnosis of melanoma. Specifically, for its diagnosis, the heat-maps will highlight the area of a mole for negative or positive cases, but we will not know what characterizes the positive or negative decision. We believe that a good explanation must be both descriptive and predictive, i.e. it can replace the original model to some extent, so the notion of fidelity between the explanatory model and the original model is essential. The explainability of deep models by propositional rules has rarely been addressed in the state of the art, because it is a very complex problem that we are tackling in this work.

3. Materials and Methods

In this part we present the models used in this work, which are ensembles of DTs and MLPs, SVMs, and CNNs. We then describe our local/global rule extraction algorithm, which is applied to all the models presented.

3.1. DIMLPs

The Discretized Interpretable Multi Layer Perceptron (DIMLP [11]) is a model that allows us to generate propositional rules using the FidexGlo algorithm described in Section 3.5. In a general Multi Layer Perceptron (MLP), we denote $x^{\left(0\right)}$ as the vector for the input layer. For layer $l+1(l\ge 0)$, the activation values $x^{(l+1)}$ of the neurons are

$$x^{(l+1)}=F(W^{\left(l\right)}{x}^{\left(l\right)}+b^{\left(l\right)}).$$

(1)

$W^{\left(l\right)}$ is a matrix of weight parameters between two successive layers l and $l+1$; $b^{\left(l\right)}$ is a vector also called the bias and F is an activation function. Usually, F is a sigmoid $\sigma \left(x\right)$:

$$\sigma \left(x\right)={\displaystyle \frac{1}{1+exp(-x)}}.$$

(2)

In a DIMLP, $W^{\left(0\right)}$ is a diagonal matrix. Moreover, the activation function applied to $x^{\left(1\right)}$ is a staircase function $S\left(x\right)$. Here, $S\left(x\right)$ approximates with $\mathsf{\Theta }$ steps a sigmoid function:

$$\begin{array}{cc}\hfill S\left(x\right)=\sigma \left({\alpha }_{min}\right),& \mathrm{if}x\le {\alpha }_{min};\end{array}$$

(3)

${\alpha }_{min}$ represents the abscissa of the first step. By default ${\alpha }_{min}=-5$.

$$\begin{array}{cc}\hfill S\left(x\right)=\sigma \left({\alpha }_{max}\right),& \mathrm{if}x\ge {\alpha }_{max};\end{array}$$

(4)

${\alpha }_{max}$ represents the abscissa of the last step. By default ${\alpha }_{max}=5$. Between ${\alpha }_{min}$ and ${\alpha }_{max}$, $S\left(x\right)$ is:

$$\begin{array}{c}\hfill S\left(x\right)=\sigma ({\alpha }_{min}+\left[\mathsf{\Theta }·{\displaystyle \frac{x-{\alpha }_{min}}{{\alpha }_{max}-{\alpha }_{min}}}\right]\left({\displaystyle \frac{{\alpha }_{max}-{\alpha }_{min}}{\mathsf{\Theta }}}\right));\end{array}$$

(5)

with [ ] designating the integer-part function. For $l\ge 2$, DIMLP and MLP are the same model.

In a DIMLP it is possible to determine the location of boundaries between classes. Specifically, these boundaries are axis-parallel hyperplanes. Consider the example of a step activation function, which is a special case of a staircase function. The step function is

$$t\left(x\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}x>0;\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(6)

Figure 1 shows a DIMLP with an input neuron a hidden neuron and an output neuron. Below the network are samples belonging to two classes: circles and squares. Due to the hidden neuron’s step activation function, we have a potential discriminant hyperplane in $-b/w$ (equal to three). The reason is that the step function defines a possible boundary where $w\ast x+b=0$.

After activating h, if the value of $w^{\prime }$ is zero, the circles and the triangles are indistinguishable, since the signal entering y will always be negative. With $w^{\prime }$ equal to 10 and $b^{\prime }$ equal to -6, if h is 0, the signal entering y is negative; otherwise, if h is equal to one, the signal entering y is positive. We therefore have a hyperplane that allows us to define two propositional rules:

• if $x>3$ then class is triangle;
• if $x\le 3$ then class is circle.

Regardless of the number of hidden layers, the first always determines the discriminative hyperplane location. When using the staircase activation function, the maximum number of discriminating hyperplanes per input variable is equal to the number of steps $\mathsf{\Theta }$ in the equation 5. Our previous rule extraction algorithm [12] was based on constructing a DT with the location of a hyperplane at each node. Each path from the root to a leaf represented a propositional rule. Finally, a greedy algorithm progressively removed rule antecedents and rules [12].

The position of the hyperplanes $h{p}_i$ for each input variable $x_i$ is

$$h{p}_i={\displaystyle \frac{1}{w_i}}·(({\alpha }_{min}-b_i)+{\displaystyle \frac{k}{\mathsf{\Theta }}}·({\alpha }_{max}-{\alpha }_{min}));$$

(7)

with $k=0,\cdots ,\mathsf{\Theta }$.

3.2. QSVM

A Quantified Support Vector Machine (QSVM) is a DIMLP network with two hidden layers [36]. First, the neurons in the first hidden layer perform a normalization of the input variables, so that the number of neurons in these two layers is the same. For each hidden neuron i, the weights are frozen during training, and their values are given as:

• weights: $w_i=1/{\gamma }_i$, with ${\gamma }_i$ as the standard deviation of input variable $x_i$ on the training set;
• bias: $b_i=-{\mu }_i/{\gamma }_i$, with ${\mu }_i$ as the average of input variable $x_i$ on the training set.

After normalization, quantization is achieved using the staircase activation function. In the second hidden layer, the activation function depends on the SVM kernel used. For example, with a dot kernel the corresponding activation function is the identity, while with a Gaussian kernel the activation function is a Gaussian. The number of neurons in this layer is equal to the number of support vectors, with the incoming weight connections corresponding to the components of the support vectors (see [37] for more details). During training, weights above the first hidden layer are calculated according to the SVM training algorithm [38].

3.3. Ensembles

In this work, we train ensembles of DIMLPs with bagging [7]. Bagging builds an ensemble of models with slightly different training datasets. It is very often the diversity of the models that is an important factor in improving accuracy over a single model. Specifically, with s training samples, bagging selects for each classifier s samples drawn with replacement from the original training set. So, for each model, the training data has missing or repeated data.

Random Forests (RFs [39]) are ensembles of DTs trained by bagging. Similar to bagging, boosting [8] constructs slightly different training data for each model. Specifically, for one model, samples are drawn randomly with replacement, but samples that have been misclassified by previous predictors are given a higher probability. In this work, we train and we apply rule extraction to ensembles of DTs trained by gradient boosting (GB) [40] and RFs. We showed in previous work that these models are special cases of DIMLP ensembles [13], and so rule extraction can be performed with the algorithm used for DIMLPs.

3.4. Convolutional Neural Networks

Convolution is an operation that calculates many scalar products between a coefficient mask and a sample. It can be represented by an MLP with layers of weights that are not completely connected. Here we consider the two-dimensional convolution operator applied to images. Therefore, given a two-dimensional kernel $w_{pq}$ of size PxQ and a data matrix of elements $m_{ab}$, the calculation of an element $c_{ij}$ of the convolutional layer is

$$\begin{array}{c}\hfill c_{ij}=f(\sum _p^P\sum _q^Q{w}_{pq}·m_{i-p,j-q}+b_{pq});\end{array}$$

(8)

with f a transfer function and $b_{pq}$ the bias. As a transfer function we use the ReLU (Rectified Linear Unit):

$$\begin{array}{c}\hfill ReLU\left(x\right)=Max(0,x).\end{array}$$

(9)

In this work, we extract propositional rules from CNNs by adding a frozen layer after the input layer whose role is to normalize and quantize the data as has been achieved with QSVMs (cf. Section 3.2). This allows us to generate propositional rules relative to the input layer.

3.5. The Fidex Algorithm

Fidex is a local rule extraction algorithm, which means that it generates an explanation for a single sample. Specifically, it is a greedy algorithm that aims to maximize the fidelity criterion. Fidelity refers to how well the extracted rules mimic the behavior of a model. For example, with s samples in a training set and $s^{\prime }$ samples for which the classifications of the rules match the classifications of the model, the fidelity is $s^{\prime }/s$.

Let us describe Fidex; we assume that we have d input variables $x_1\cdots x_d$. Furthermore, let us denote $L_{xi}$ the list of hyperplanes for input variable $x_i$; $L_{xi}$ is calculated according to eq. (7). The purpose of Fidex is to determine a propositional rule R with respect to sample S. The fidelity of R to be reached in $\rho$ steps is $\varphi$, and is calculated with all the samples of the training set. The Fidex algorithm is the following:

• Given sample S and rule $R=\varnothing$;
• for $n=1\cdots \rho$ do
•  for all input variables $x_i$ randomly selected and for all $L_{xi}$ do
•   find the hyperplane $h^{\ast }$ allowing the highest increase of fidelity;
•  end for
•  if the increment of fidelity is strictly positive then $R=R\cup h^{\ast }$;
•  if fidelity reaches value $\varphi$ then exit;
• end for

Fidex computational complexity depends mainly on its loops and corresponds to $O(\rho ·d·\mathsf{\Theta }·s)$; with s the number of training samples. Note that the s factor appears as for each new rule antecedent Fidex must determine whether a sample is covered by R or not. Finally, $\mathsf{\Theta }$ is due to the number of hyperplanes in each list of possible hyperplanes $L_{xi}$. As an example of the number of hyperplanes, if we have 100 input variables and 100 steps in the staircase function in the first hidden layer of a DIMLP (or a QSVM) then $L_{xi}$ has 10000 hyperplanes, but if we have an ensemble of 25 DIMLPs this number becomes 250000. With a CNN trained on images of size $64\times 64$ and 255 steps, we obtain 1044480 hyperplanes. As this number is quite large, we have two parameters to reduce the number of hyperplanes. Specifically, the execution of Fidex is managed by five parameters:

• $\mu$: minimal number of samples covering the final rule;
• $\rho$: maximal number of Fidex steps (which is strongly related to the number of rule antecedents);
• $\varphi$: fidelity of the rule;
• p: dropout of input variables;
• q: dropout of hyperplanes.

The parameter $\mu$ is the minimum number of samples in the final rule, with a default value of two. The maximum number of iterations in Fidex which is related to the maximum number of rule antecedents is $\rho$; its default value is ten. The default value of $\varphi$, which corresponds to the fidelity that has to be reached, is 100%. The execution time of Fidex can be accelerated by considering two dropout parameters: p and q. Essentially, p determines at each step the proportion of input variables that will not be considered. The excluded variables are selected randomly. Parameter q is similar, but with respect to excluded hyperplanes in $L_{xi}$. Default values for p and q are equal to 0. If variables $x_i$ are selected with always the same order, for instance from 1 to d (see step 3) and without dropout ($p=0$ and $q=0$), Fidex is deterministic. Since a DT is a collection of nodes with conditions related to the input variables, the list of hyperplanes for an ensemble is the union of all the conditions of the trees. As a result, with GB, which typically uses shallow trees, fewer hyperlanes are brought into play compared to RFs and DIMLPs, so the rule extraction algorithm tends to be faster.

3.6. The FidexGlo Algorithm

FidexGlo is a global rule extraction algorithm1 that generates a ruleset for a training set of size s. It corresponds to a covering technique that calls Fidex s times. It therefore first generates s rules (see Figure 2) and then uses a heuristic to select a subset of the rule base that covers all s samples.

A simple heuristic would be to select the rules randomly and stop when all the training samples are covered. Another heuristic consists of ranking the rules in descending order according to the number of samples covered, then selecting the rules in descending order until all the samples are covered. As we are using the latter heuristic, the computational complexity is $O(\rho ·d·\mathsf{\Theta }·s^2)$. The essential reason is that Fidex is called s times. When compared to our old rule extraction algorithm (OREA [12]), which is global, FidexGlo is faster, since the algorithmic complexity of the former was $O(d^4·{\mathsf{\Theta }}^4·s^2)$ [12] (polynomial of degree four with respect to the number of variables and steps).

Rule extraction can be performed with an ensemble of DIMLPs, since it can be viewed as a unique DIMLP with an additional hidden layer (the one that averages all single network responses). Hence, the list of the hyperplanes for a DIMLP ensemble is the union of all the lists related to each single network (cf. eq. 7). Hence, with $\nu$ networks in an ensemble, the computational complexity is $O(\nu ·\rho ·d·\mathsf{\Theta }·s^2)$.

4. Experiments with Tabular Data

In this section we present several experiments performed on tabular data. First, we describe the characteristics of the classification problems, then we explain the evaluation measures and the values of the learning parameters and finally we describe the cross-validation experiments. In this part, we would like to find out if the proposed technique for the extraction of rules gives good results with respect to other known algorithms. Note that it is not our intention to determine the best possible performance by varying the parameters of the models.

4.1. Datasets

We retrieved four classification problems from the Machine Learning Repository at the University of California, Irvine (https://archive.ics.uci.edu/ml/index.php) [41]. Table 1 describes their main characteristics.

4.2. Learning Parameters and Assessment Measures

The following models were used in this work:

• DIMLP ensembles trained by bagging (DIMLP);
• Quantized Support Vector Machines (QSVM);
• Random Forests (RF);
• Shallow Decision Trees trained by Gradient Boosting (GB);

To train the DIMLPs, the data were normalized using Gaussian normalization. Specifically, during cross-validation trials, the means and standard deviations of each variable were calculated for the training datasets, and then these values were used to normalize the training data and the testing data. Normalization was not applied to the training of ensembles of DTs (RF and GB) as this is not necessary. Finally, for the QSVMs, the normalization was encoded in the weight values of the first layer (see Section 3.2), which was frozen during training.

In the experiments we tested two different architectures with DIMLPs. The first has a single hidden layer with a number of neurons equal to the number of inputs. In the second architecture, we added a second hidden layer of ten neurons to each DIMLP. All DIMLP ensembles had 25 networks. The unfrozen layers of QSVMs were trained with the Scikit Python library [42]. Specifically, we used default learning parameters with two different kernels: the linear kernel and the RBF kernel. Ensembles of 100 DTs were also trained using the Scikit Python library. For RFs we configured trees of depth ten or unconstrained depth, while for GBs we used shallow trees of depth one or three.

DIMLPs were trained by back-propagation with default learning parameters:

• Learning parameter = 0.1;
• Momentum = 0.6;
• Flat Spot Elimination = 0.01;
• Number of steps in the staircase activation function = 50.

Finally, out-of-bag samples were used with DIMLPs to avoid the overtraining phenomenon, by applying an early-stopping technique [43]. Specifically, the out-of-bag set is a subset of the training data that is not used to adjust the weights of the neural networks, but to decide when to stop training. To generate propositional rules, the Fidex parameters were:

• $\mu =2$ (minimal number of samples in the final rules);
• $\rho =20$ (maximal number of Fidex steps);
• $\varphi =100$% (fidelity of the rules);
• $p=0$ (dropout of input variables);
• $q=0$ (dropout of hyperplanes).

Prediction accuracy being defined as the ratio of correctly classified samples on the total number of samples of a testing set, columns from left to right in the following Tables designate:

• average prediction accuracy of the model in the testing set divided by the number of samples;
• average fidelity on the testing set, which is the degree of matching between the generated rules and the model. Specifically, with P samples in the testing set and Q samples for which the classification of the rules corresponds to the classification of the model, fidelity is $Q/P$.
• average prediction accuracy of the rules;
• average prediction accuracy of the rules when rules and model agree. Specifically, it is the proportion of correctly classified samples among the Q samples defined above.
• average number of extracted rules;
• average number of rule antecedents.

4.3. An Introductory Experiment

With the “Iris” classification problem belonging to three classes, we performed experiments based on ten repetitions of 10-fold cross validation trials. Table 2 illustrates the results, with the standard deviations in brackets. QSVMs were not trained here, as our current implementation is for datasets belonging to two classes. The first row of this table refers to DIMLP architectures without a second hidden layer, the second to DIMLPs with a second hidden layer of ten neurons, the third to RFs of depth ten, the fourth to RFs of unlimited depth, the fifth to GBs of depth one, and the last to GBs of depth three.

The highest average prediction accuracy of the rules (Acc. R. a) was provided by the DIMLPs with 95.3%. The same model also produced the lowest average number of rules (4.5) and antecedents per rule (1.6). RFs and GBs tended here to generate a higher number of rules. Finally, the average fidelity on the testing sets was over 99% for all models. Figure 3 shows an example of ruleset including four rules generated from a DIMLP ensemble. In cross-validation trials, its accuracy was 96% on the training set and 100% on the testing set. It is worth presenting in Figure 4 a more complex ruleset generated from a GB with a depth of three containing ten rules. The prediction accuracy of this second ruleset was 100% on both training and testing sets during cross-validation trials.

The rules generated by the FidexGlo algorithm are not ordered. In other words, the rules are not linked by an else statement so that one or more rules are activated for each sample. If a sample activates the rules that belong to more than one class, only the rules of the class that is provided by the model are kept, while the others are discarded. The first ruleset, due to its small size, makes it easy to understand how an iris is classified. The second ruleset is longer and more detailed and includes rules covering a small number of samples. It is perhaps more interesting for understanding iris classification in detail.

4.4. Comparisons to Other Classification Problems

We conducted experiments based on ten repetitions of 10-fold cross validation trials. Table 3 illustrates the results for the “Breast Cancer” dataset, with the last four rows showing other methods. The highest average prediction accuracy of the rules (Acc. R. a) was provided by the DIMLPs (97.3%), with QSVMs close behind (97.2% and 97.1%). Thus, FidexGlo has been shown to provide accurate rulesets in comparison with other techniques. On average, more rules were produced by RFs and GBs of depth three than DIMLPs and QSVMs. Furthermore, the average fidelity of the rulesets generated by our models was over 99%. Finally, the lowest average number of rules was obtained by RuleCOSI with an average of 10.8 rules, but the number of antecedents per rule is not reported by the referenced work.

Table 4 shows the results for the “Divorce” classification problem, with the last two rows representing other methods. The highest average prediction accuracy of the rules was obtained again by the DIMLPs with 98.0%, but all our models were very close. Furthermore, the lowest average number of generated rules was provided by QSVMs with the RBF kernel (2.4). Again, the average fidelity of the rulesets generated by our models was over 99%.

In Table 5 we present the results for the “Ionosphere” classification problem, the last four rows showing other methods. The highest average prediction accuracy of the rules was provided by RFs with 94.3%, but with more generated rules than RuleFit (28.8 versus 25). Again, FidexGlo has been shown to provide accurate rulesets in comparison with other techniques in the state of the art. The lowest average number of generated rules was provided by GB with 13.8 rules, on average. Finally, except for QSVM using the linear kernel, the average fidelity was over 97%.

For this classification problem with 34 variables, we wondered how the complexity of the rules varies as a function of the Drop-Var and Drop-Hyp parameters for DIMLP ensembles trained by bagging. Figure 5 shows the variation of several characteristics of the rules as a function of the drop-out parameters (p and q) with the same values varying from 0 to 0.99 (last bar). Each bar is the result of a cross-validation. We observe that the average number of rules is stable up to 0.5 on the x-axis and then starts to decrease. Finally, there is a further increase from the penultimate bar. The top right shows the average number of antecedents per rule on the vertical axis. This variable is relatively stable until it reaches a value of 0.8 on the x-axis, after which it increases to compensate for the scarcity of variables (and their hyperplanes) available for constructing rules. The bottom left corner shows the average fidelity, and the bottom right corner shows the average prediction of the rules. We can see that the average fidelity is stable and that the average prediction accuracy of the rules has slightly decreased.

We present in Figure 6 the variation of several characteristics of the rules as a function of the drop-out parameters for RFs. The results for the fidelity and prediction accuracy of the rules are similar to those obtained for the DIMLP ensembles. We observe that the average number of rules on the top left decreases progressively up to the abscissa of 0.8, then increases and in the last bar, which has an abscissa of 0.99, decreases sharply. The top right shows the average number of antecedents on the vertical axis. The values of this variable increase progressively until the penultimate bar, after which there is a very marked decrease, as with the average number of rules. It’s as if there were not enough hyperplanes left to build the rules, which are still faithful and perform well in terms of prediction. However, we noticed that the average coverage rate on the test data is 40.9%, while for the other values of Drop-var and Drop-Hyp it is over 95%. Overall, the performance in terms of fidelity, accuracy and complexity of the rules does not change too much with drop-out values up to 0.7-0.8.

5. Experiments with CNNs

We conducted a series of experiments on three classification problems based on images learned by CNNs. The first problem is to recognize handwritten numbers, the second is to identify facial emotions and the last problem purpose is to detect cracks in buildings. All the datasets are in the public domain (see after the conclusion). Our aim is to show that we obtain accurate explanations by means of propositional rules.

5.1. Handwritten Numbers

For handwritten number recognition, we used the MNIST dataset, a benchmark for data classification [44]. Numbers between zero and nine are represented by $28\times 28(=784)$ pixels. The training and testing sets of this classification problem consist of 60000 and 10000 samples, respectively. To avoid over-training, we used 10% of randomly selected training samples as the tuning set for early stopping [43]. We first trained ten small CNNs (CNN-Small) with the following architecture:

• an input layer;
• a normalization layer with a staircase activation function;
• a convolutional layer with 32 kernels of size $5\times 5$ and ReLU activation function, the stride parameter being equal to 1;
• a max-pooling layer, the stride parameter being equal to 2;
• a convolutional layer with 32 kernels of size $5\times 5$ and ReLU activation function, the stride parameter being equal to 1;
• a max-pooling layer, the stride parameter being equal to 2;
• a fully connected layer of 256 neurons;
• an output layer of 10 neurons.

As with QSVMs, the first hidden layer performs a quantified normalization. We then fine-tuned ten deeper architectures using pre-trained weights defined by the VGG neural network [45] (CNN-VGG). VGG has sixteen layers, on top of which we defined two fully connected layers with 256 and 10 neurons respectively. FidexGlo was run with the following parameter values:

• $\mu =2$;
• $\rho =20$;
• $\varphi =1$;
• $p=0.8$;
• $q=0.8$.

Table 6 shows the results. It is worth noting that CNN-VGG achieved slightly higher accuracy, on average. For both CNN architectures, the average test fidelity is over 96%. In addition, the average prediction accuracy of the rules, when the rules and the network match, is higher than that of the model. The number of rules appears high compared to the results obtained with the tabular data (see Section 4), but this is due to the number of training examples, which is much larger here.

We exhibit several visualizations of rules and samples. Figure 7 presents four pictures with some colored dots. Green dots represent antecedents given as $a_i>t_i$, where $a_i$ is a rule antecedent and $t_i$ a constant. Note that with images, $a_i$ is a pixel intensity. Similarly, red dots indicate antecedents given as $a_j\le t_j$. At the top left is shown the centroid of the represented rule, which covers 1426 samples of the training set. This centroid is calculated by summing all pixels and dividing by the number of samples covered. The other three pictures represent three samples from the testing set. Most of the red dots are near the edges, while the green dots are in the number itself. The represented rule is correct throughout the test set, with 242 examples covered by it. Moreover, the centroid (top left) is tilted to the right, indicating that this is the tendency of this rule. However, it can take into account cases that are straighter, such as the one on the bottom right.

Similarly, Figure 8 represents a rule with a centroid (top left) and three samples for class ’0’. This rule covers 183 samples of the test set, and its accuracy is 100%. The green dots are on the left-hand arc of the pictures and the red dots are in the hole. Note the variability of the examples with a large or small hole, but also of the left arc (pictures at the bottom). Overall, this rule says that a certain number of samples of class ’0’ are grouped together with a certain shape of left arc and with some flexibility in the size of the hole in the middle.

In Figure 9 we have a rule of class ’4’ covering 103 samples of the test set and with an accuracy of 100%. Again, the centroid of the covered samples in the training set (565 samples) is shown at the top left. The centroid is slightly tilted to the right. There are two red dots in the empty space between the two bars in the upper part and three green dots in the number. In the lower part of the centroid, we have two red dots to the left of the stem going down and one to the right. The top right sample and the bottom left sample have a noticeable difference in the width of the stems going up or down. We can therefore understand that the antecedents of the associated rule allow a certain amount of flexibility in covering the samples in terms of the thickness of the rods and in terms of the size of the upper empty space.

5.2. Facial Expression Recognition

The Facial Expression Recognition (FER) dataset that we retrieved consists of $48\times 48$ grayscale images. Seven categories of emotions have been defined, but in this work, we only focused on two classes: ’Happy’, and ’Unhappy’ (represented by angry, disgust, fear, sad, surprise, and neutral). The number of training samples is 28709 and an independent test set of 7178 samples was used for testing. Moreover, the number of ’happy’ class samples is 7215 for training and 1774 for testing. Again, to avoid over-training, we used 10% of randomly selected training samples as the tuning set for early stopping. FidexGlo was run with the following parameter values:

• $\mu =2$;
• $\rho =20$;
• $\varphi =1$;
• $p=0.9$;
• $q=0.9$.

The results of the fine-tuning of a VGG architecture, averaged over ten trials, are shown in Table 7. Similarly to the previous classification problem, we defined two fully connected layers on top of the VGG network, with 256 and two neurons respectively. The average number of rules is similar to that obtained in the MNIST problem and the average fidelity is 93.06%. Again, the average predictive accuracy of the rules, when the rules and the network agree, is higher than that obtained by the model.

The following Figures illustrate several examples of rules for the ’Happy’ class. Figure 10 shows four pictures with at the top left the centroid of the represented rule, which covers 27 samples of the training set. The other three pictures represent three covered samples from the testing set. In the picture representing the centroid and the three samples, the antecedents of the rules in red delineate the ends of the mouth in a smiling posture. In addition, a green antecedent is positioned over the teeth and three others in the lower part of the face. Generally, an expression of joy can be strongly marked by a smile.

In Figure 11, the centroid was calculated using 31 samples from the training set. Again, it shows us that the teeth are targeted, as well as the corner of the mouth on the right. Note that the left cheek fold is targeted by a red spot. We can also see a rule antecedent on the eyebrow and a little higher up on the forehead. Finally, an antecedent is used for the chin and, at the bottom left, an antecedent is used to select images with a bright area. The pictures on the right are relatively consistent with the features highlighted by the centroid, while the picture on the bottom left has a slightly different orientation and shows the antecedents of the rules in slightly different places; in particular, the spot on the teeth is on the lower lip and the spot on the eyebrows is in the darker part of the nose.

A final example of the ’Happy’ class is shown in Figure 12. The centroid has rule antecedents on: the left eye; teethes; above the upper lip (left and right); below the lower lip (left); and on the bottom right. The lower left picture has a different orientation, but the features pointed to by the rule antecedents are similar.

5.3. Cracks Recognition

The “Cracks” dataset contains 40000 images, half of which show cracks in buildings [46]. Therefore, we defined a classification problem with two classes: cracks present or absent. We downsampled the grayscale images to a resolution of $64\times 64$ pixels. Then, we randomly selected 20% of the samples as a testing set. On the training set we used 10% of randomly selected samples as the tuning set for early stopping. FidexGlo was run with the following parameter values:

• $\mu =2$;
• $\rho =20$;
• $\varphi =1$;
• $p=0.95$;
• $q=0.95$.

The results obtained by fine-tuning a VGG architecture and averaged over ten trials are presented in Table 8. As in the previous classification problem, we defined two fully connected layers on top of the VGG network, with 256 and two neurons, respectively. The mean prediction accuracy of the model and rules was over 99%, as was fidelity. The mean number of extracted rules was 1104.3, with 14.2 antecedents, on average.

We noticed that for the rules belonging to the class ’No-Cracks’, the number of rule antecedents was significantly higher than for the class ’Cracks’. Figure 13 and Figure 14 present two rules with their centroid (top left) and three samples for class ’Cracks’. Both centroids have a dark spot and are very similar. A red spot is seen on the dark area of the centroids and a green spot on the background of the image. In a way, by averaging all the images covered by a rule, only a small dark area remains and everything else is quite bright. The VGG network therefore uses the strategy of finding a dark area that represents a crack in the central part of an image and a bright area elsewhere. This may sound simple, but it is the case with many rules.

6. Discussion

In this work we focused on explainability by propositional rules. We implemented two explainability algorithms: the first is local and the second is global and uses the local algorithm (Fidex). With tabular data, FidexGlo applied to ensembles and SVMs performed well compared to other existing methods. Then, for CNNs, it turned out that the number of rules generated was high compared to the problems with tabular data, but the number of training samples and input variables was also much larger, and so the explainability task is more complex with more samples to be covered. As an example of execution time, FidexGlo took approximately thirteen hours to extract global rules using 128 CPUs in parallel. Finally, once the final rules are computed, it can be time-consuming to examine thousands of rules to explain a model, but at least it is possible to determine precisely how a model has classified data. From there, we can even determine whether a model’s classification strategy is partially or completely incorrect, and whether new training examples can be created and fed to counterbalance the errors.

With linear and logistic regression, it is always possible to obtain some form of explanation thanks to the coefficients that define these models. For example, linear regression makes it easy to calculate the relevance of a given variable to the output. Note, however, that these models would give very poor results on the datasets learned in this work by CNNs. Our FidexGlo global algorithm may in some cases have no explanation for the test samples (no rules activated), because some regions of the input space may not have been explored with the training samples. Nevertheless, it is still possible to use our local algorithm, which will always give at least one explanation in the form of a propositional rule. Since the fidelity of samples not learned during the training phase is generally different from 100%, we could adopt a strategy where the samples that do not activate any rules of the correct class would run Fidex to obtain a faithful explanation.

Our explanations of the CNN classifications are low-level because they are given at the pixel level. In the experiments, we highlighted model strategies for classifying the data. Specifically, for the crack classification, we found that when there is a crack, one or more antecedents were located on the crack, so the interpretation is straightforward. Furthermore, for the classification of the joy emotion, we found that the rules indicated rule antecedents at the level of teethes or skin folds, which typically represent happy expressions. The many rules obtained highlight different elements. The examples given in the previous figures are just a small sample, but the thousands of rules obtained give us a lot of information. Determining explanations using high-level concepts is a current trend in research on the explainability of connectionist models. A disadvantage of this type of approach is that the concepts must be defined in advance before they can be learned. In our low-level approach we do not need to define any prior knowledge, and this is an advantage over a high-level approach. Finally, we highlighted the centroid images which, for a given rule, represent the average image of the samples covered. This can be seen as a prototype and in a way we get a higher level explanation compared to the pixel representation.

7. Conclusions

We presented FidexGlo, a new global rule extraction method that generates propositional rules from ensembles, SVMs, and CNNs. With ensembles and SVMs our results were good compared to other existing techniques, in terms of fidelity, rule complexity, and accuracy. But our primary goal here was to use FidexGlo with much more complicated models, such as deep neural networks. The quadratic algorithmic complexity of our rule extraction technique made this possible in three image-based classification problems by first fine-tuning the VGG CNN model and then running FidexGlo. We summarized the characteristics of the rules and provided several examples of rule visualizations by their centroids and rule antecedents, which in some way reveal the classification strategy. To the best of our knowledge, this is one of the few works showing a global rule extraction technique applied to both ensembles, SVMs and deep neural networks.