Why Do Tree Ensemble Approximators Not Outperform the Recursive-Rule eXtraction Algorithm?

1. Introduction

Artificial intelligence (AI) has made great advances and AI algorithms are currently being applied to solve a wide variety of problems. However, this success has been driven by accepting their complexity and adopting “black box” AI models that lack transparency. On the other hand, eXplainable AI (XAI), which enhances the transparency of AI and facilitates its wider adoption in critical domains, has been attracting increasing attention [1,2,3,4,5,6,7,8,9,10].

Rudin [11] pointed out the limitations of some approaches to explainable machine learning, suggesting that interpretable models should be used instead of black box models for making high stakes decisions. In the field of health care, for example, it is not sufficient for a medical diagnosis model simply to be accurate; it must also be transparent to health professionals who use the output to make decisions about a given patient [6,12,13]. Moreover, in the field of finance, recent regulations, such as the General Data Protection Regulation and the Equal Credit Opportunity Act, have increased the need for model interpretability to ensure that algorithmic decisions are understandable and consistent. These issues have been addressed by interpretable machine learning models, which are characterized as models that can be easily visualized or described in plain text for the end user [14].

Tree ensembles are often used for tabular data. Bagging [15] and random forests (RFs) [16] are known as independent ensembles, whereas gradient boosting machines (GBMs) [17] such as XGBoost [18], LightGBM [19], and CatBoost [20] are known as dependent ensembles. Tree ensembles are extensively utilized in academic and research contexts. They are also applied in practical scenarios across a wide array of domains [21]. Recently, these models have been effective in many classification tasks. In fact, these models are used by most winners of Kaggle competitions 1. However, the structure of these algorithms is considered complex and very difficult to interpret. The effectiveness of ensemble trees generally improves as the number of trees increases, and in some cases, an ensemble can contain thousands of trees.

Decision trees (DTs), rule-based approaches, and knowledge graph-based approaches are widely used as examples of interpretable models [22,23,24,25,26,27,28,29]. Techniques for approximating tree ensembles with DTs or rule sets have also been investigated [30,31,32,33,34,35,36]. However, although tree ensemble approximators focus on reducing the number of rules and conditions, they do not consider the interpretability of the rules. For example, handling categorical and numerical attributes separately is known to increase interpretability [28].

There are three main types of rule sets generated by these methods: DT-based, unordered-based (the last rule in the rule set is Else), and decision list-based. Figure 1 shows the concept of these three types of rule sets. However, previous studies have not provided a metric to distinguish and compare these different rule sets. In the present study, we newly propose an interpretability metric, Complexity of Rules with Empirical Probability ($CREP$), to allow comparisons of interpretability between different rule sets. $CREP$ enables a fair comparison of different types of rule sets, such as decision list- and DT-based rule sets. We also explore the interpretability of the rules generated by the tree ensemble approximators. Specifically, we present and compare not only objective metrics, but also rule sets generated by the tree ensemble approximators and the Recursive-Rule eXtraction algorithm (Re-RX) with J48graft [29].

2. Related Work

The Re-RX algorithm [28] is a rule-based approach that can handle categorical and numerical attributes separately and extract rules recursively. By separating categorical and numerical attributes, Re-RX can generate rules that are intuitively easy to understand. Re-RX with J48graft  [29] is the extended version of Re-RX. Numerous studies have conducted research on Re-RX [37,38,39,40,41,42]. RuleFit [30] is a method that employs a linear regression model with a DT-based model to utilize interactions. The rules generated by the ensemble tree are used as new features and fitted using Lasso linear regression. inTrees [31] extracts, measures, prunes, and selects rules from tree ensembles such as RFs and boosts trees to generate a simplified tree ensemble learner for interpretable predictions. DefragTrees [32] involves simplifying complex tree ensembles, such as RFs, to enhance interpretability by formulating the simplification as a model selection problem and employing a Bayesian algorithm that optimizes the simplified model while preserving prediction performance. Initially introduced for independent tree ensembles by Sagi and Rokach [33], Forest-based Trees (FBTs) were later extended to dependent tree ensembles. Combined within both bagging (e.g., RFs) and boosting ensembles (e.g., GBMs), FBTs construct a singular DT from an ensemble of trees. Rule COmbination and SImplification (RuleCOSI+) [36], a recent advance in the field, is a fast post-hoc explainability approach. In contrast to its precursor, RuleCOSI, which was limited to imbalanced data and Adaboost-based small trees according to Obregon et al. [35], RuleCOSI+ was designed as an algorithm that extends the capabilities of RuleCOSI to function effectively in both bagging (e.g., RFs) and boosting (e.g., GBMs) ensembles. DefragTrees, FBTs, and Re-RX generate DT-based rule sets, inTree generates an unordered-based rule set, and RuleCOSI+ generates a decision list-based rule set.

Given this background, in the present study, we aim to provide new insights into the reasons why DL-based and DL-inspired classifiers do not work well for categorical datasets mainly consisting of nominal attributes [43].

3. Materials and Methods

3.1. Datasets

We used 10 diverse datasets from the University of California, Irvine, Machine Learning Repository [44] to compare each method. The details of the datasets are shown in Table 1. For each dataset, we split the data into training:test at a ratio of 8:2. Consistent splits were applied to all methods, with a unique seed-based split for each iteration. Each iteration means $10\times$ in the $10\times 10$-fold cross-validation (CV) scheme described in Section 3.3.3.

3.2. Baseline

We used scikit-learn’s DT and J48graft  [45] as simple DT-based methods. J48graft is a grafted (pruned or unpruned) C4.5 [46] DT. DT generates a binary tree, while J48graft, capable of handling categorical attributes, generates a m-ary tree.

We used FBTs and RuleCOSI+ for the tree ensemble approximator. Figure 2 and Figure 3 show overviews of FBTs and RuleCOSI+, respectively. Both FBTs and RuleCOSI+ were implemented using the official code provided by the authors 2 3.

As a rule-based method, we used Re-RX with J48graft, an overview of which is shown in Figure 4. In learning a multilayer perceptron (MLP) in Re-RX with J48graft, we apply one-hot encoding 4 to categorical attributes to enable efficient learning. With the application of the one-hot encoding to categorical attributes, we modify the pruning algorithm for the MLP. In the original pruning algorithm, the attributes are removed from $\mathcal{D}$ when $w_{i,\ast }=0$, where $w_{i,\ast }$ represents all weights in the first layer of the MLP connected to the i-th attribute. Let $\mathcal{C}$ be the set of categorical attributes in the dataset $\mathcal{D}$, and ${\overline{\mathcal{C}}}_i$ be the set of one-hot encoded values for the i-th categorical attribute $c_i\in \mathcal{C}$. In this study, we modified the pruning algorithm as follows: $\forall j\in \{j=0,...,|{\overline{\mathcal{C}}}_i|-1\}$, if $w_{j,\ast }=0$, then $\mathcal{D}\leftarrow \mathcal{D}\setminus \left\{c_i\right\}$.

All fitted methods are converted to the RuleSet 5 module implemented by Obregon and Jung [36].

3.3. Experimental Design

In this section, we present the experimental design for comparing methods.

3.3.1. Data Preprocessing

We applied one-hot encoding to the categorical attributes because of the inability of FBTs, RuleCOSI+, and DT to handle categorical attributes. For the numeric attributes, we applied standardization only to the training and prediction of the MLP in Re-RX with J48graft.

3.3.2. Interpretability Metrics

The metrics of interpretability, such as the total number of rules ($N_{rules}$) and average number of conditions, are often used. However, these metrics cannot distinguish between decision list-based, unordered, and DT-based rule sets.

We newly propose an interpretability metric, $CREP$, to facilitate fair comparisons of interpretability between different rule sets. $CREP$ quantifies the complexity of rules based on their empirical probability (coverage on the training data), and is defined as follows:

$$\begin{array}{c}\hfill CREP=\sum _{r\in R}{N}_{con{d}_r}·cov(r,\mathcal{D})\end{array}$$

(1)

where $N_{con{d}_r}$ is the number of conditions in rule r, and $cov(r,\mathcal{D})$ is the coverage of rule r on training data $\mathcal{D}$. If the rule set is a decision list or an unordered rule set, the instances in the data refer to one or more rules in the rule set. Therefore, if the rule set is a decision list, $N_{con{d}_r}$ is accumulated from the top, and if the rule set is an unordered rule set, $N_{con{d}_r}$ of all rules is added together for the Else rule in the rule set. The operation of $N_{con{d}_r}$ accumulation allows $CREP$ to compare different rule sets fairly.

$CREP$ represents the expected value of the number of conditions in the rules using the empirical probability obtained from the training data. In other words, if rules with a high likelihood of being referenced have fewer conditions, $CREP$ decreases, and if they have more conditions, $CREP$ increases. Conversely, rules with a low likelihood of being referenced may have many conditions, but their impact is minimal. Compared with $N_{rules}$ and the average number of conditions, which evaluate the interpretability of the entire model, $CREP$ can be considered a more practical metric.

$CREP$ in Eq. (1) treats all classes equally and therefore underestimates the interpretability of minority class rules when the dataset is class-imbalanced. This problem can be solved by calcu-lating $CREP$ for each class. We redefine as follows:

$$\begin{array}{ccc}\hfill \mathrm{micro}-CREP& =& \sum _{r\in R}{N}_{con{d}_r}·cov(r,\mathcal{D})\hfill \\ \hfill CRE{P}_c& =& \sum _{r\in R_c}{N}_{con{d}_r}·cov(r,\mathcal{D})\hfill \\ \hfill \mathrm{macro}-CREP& =& \frac{1}{\left|C\right|}\sum _{c\in C}CRE{P}_c\hfill \end{array}$$

where C is the set of all classes and $R_c$ is the subset for each class in the rule set. micro-$CREP$ is useful for both unbalanced datasets and evaluating entire rule sets. In this paper, we refer to micro-$CREP$ as $CREP$.

3.3.3. Model Evaluation and Hyperparameter Optimization

We performed the experiment using a stratified $10\times 10$-fold CV 6 scheme, which is a 10-fold CV repeated 10 times. In each cv-fold, we performed hyperparameter optimization using Optuna [48]. First, the hyperparameters of the base model, which is an MLP in Re-RX with J48graft and XGBoost in FBTs and RuleCOSI+, were optimized to maximize the classification performance. Then, other hyperparameters were optimized using multi-objective optimization 7 to maximize both classification performance and interpretability simultaneously. For both DT and J48graft, the first step was skipped because these methods do not have a base model. We used the area under the receiver operating characteristics curve (AUC-ROC) [49] for the classification performance metric, and the inverse of $N_{rules}$ ($1/N_{rules}$) for the interpretability metric. See Section A for details on the hyperparameters for each method.

In the case of multi-objective optimization, the optimal hyperparameters are provided on the Pareto front. From the Pareto front, we selected the hyperparameters that maximize the following equation:

$$\begin{array}{c}\hfill k=-{log}_2{N}_{rules}+\alpha ·AUC\end{array}$$

(2)

where $\alpha$ is a parameter that controls the trade-off between classification performance and interpretability. This equation indicates that $AUC$ increases by $1/\alpha$ and $N_{rules}$ decreases by half, which are equivalent. A higher $\alpha$ prioritizes classification performance, whereas a lower $\alpha$ prioritizes interpretability. In this experiment, we set $\alpha =0.25$. In other words, the $AUC$ value increasing by 4 points and $N_{rules}$ decreasing by half are equivalent. We excluded the Pareto solution with $N_{rules}=1$. When $N_{rules}=1$, the rule set R classifies all instances into the same label, which is a meaningless rule set.

4. Results

In this section, we present the experimental results and their analyses.

4.1. Classification Results

The classification results are presented in Table 2. RuleCOSI+ outperformed the other methods in many datasets. DT was inferior to RuleCOSI+ but superior to the other baselines. FBTs and J48graft performed better than Re-RX with J48graft, but tended to generate more rules and less interpretability than the other baselines, as discussed in the next subsection. Re-RX with J48graft was inferior to the other baselines on average, but showed competitive results against RuleCOSI+ in some datasets.

4.2. Interpretability Results

The interpretability results and the number of rules in $N_{rules}$ and $CREP$ are presented in Table 3 and Table 4. RuleCOSI+ outperformed the other methods for many datasets in $N_{rules}$. In particular, the variance was considerably smaller than that of the other methods, resulting in stable rule set generation. On the other hand, $CREP$ was larger than other methods. Because RuleCOSI+ generated a decision list-based rule set, $CREP$ tended to be large. This is discussed in detail in Section 4.4 and Section 5.4. DT obtained superior $N_{rules}$ and $CREP$ for many datasets. FBTs and J48graft produced very large $N_{rules}$ for some datasets. FBTs had higher $N_{rules}$ variance in the tictactoe, german, biodeg, and bank-marketing datasets, indicating unstable rule set generation. Furthermore, $CREP$ was larger for FBTs, even though it is a tree-based method. Although Re-RX with J48graft resulted in $N_{rules}$ being slightly higher than the other methods, except FBTs on average, $CREP$ was much lower than the other methods. Re-RX with J48graft and J48graft tended to have a large $N_{rules}$ because they both handle categorical attributes.

4.3. Summary of Comparative Experiments

Table 5 shows a summary of all the classification and interpretability results presented in the previous subsections. RuleCOSI+ had the highest scores for $AUC$ and $N_{rules}$, a result that overwhelmed the other methods when $N_{rules}$ was emphasized as an indicator of interpretability. Although DT and Re-RX with J48graft were inferior to RuleCOSI+ in terms of classification performance, they outperformed the other methods in $CREP$. In other words, DT and Re-RX with J48graft are appropriate when the interpretability and classification frequency of the rules by which instances are classified are important. FBTs and J48graft were not significantly better than the other methods in any of the metrics, and were relatively unsuitable when interpretability was more important.

4.4. Two Examples

We present two examples of rules actually generated in the german and bank-marketing datasets and compare DT, Re-RX with J48graft, and RuleCOSI+. We excluded FBTs and J48graft from the comparison in this section because of the relatively large numbers of rules and the difficulty of analyzing the rules. For each method, rules with $N_{rules}$ matching the median were adopted.

In RuleCOSI+ and DT, categorical attributes were renamed columns by one-hot encoding. For example, the one-hot encoding for element a of attribute x generates the attribute x="a". In other words, a rule such as $\mathit{x}="\mathit{a}">0.5$ is equivalent to $\mathit{x}=\mathit{a}$, and a rule such as $\mathit{x}="\mathit{a}"\le 0.5$ is equivalent to $\mathit{x}\ne \mathit{a}$. To maintain consistency in notation, we converted all one-hot encoded categorical attributes in the rules to the format $\mathit{x}=\mathit{a}$ or $\mathit{x}\ne \mathit{a}$.

4.4.1. bank-marketing

Table 6 shows an example of the rule set generated by each method. The rule set generated by RuleCOSI+ consists of complex rules involving both numerical and categorical attributes. Furthermore, the rule for class 1 had extremely low interpretability because it was expressed as follows:

$$\begin{array}{c}\hfill r_{\mathrm{class}1}\neg r_0\wedge \neg r_1\to \left[1\right]\end{array}$$

(3)

By contrast, the rule set generated by Re-RX with J48graft consists exclusively of rules based on categorical attributes, resulting in relatively high interpretability. Furthermore, it is simpler by post-processing, as shown in Table 7. The rule set generated by DT, while containing numerical attributes, was composed of simple rules.

4.4.2. german

Table 8 shows an example of the rule set generated by each method. As in Section 4.4.1, the rule set generated by RuleCOSI+ contained a complex mixture of numerical and categorical attributes, and the rule for class 1 was Eq. (3), which had extremely low interpretability. On the other hand, the rule set generated by Re-RX with J48graft was relatively highly interpretable because the rules were composed of categorical attributes, except for $r_8$ and $r_9$. For $r_8$ and $r_9$, Re-RX with J48graft performed subdivision and added a numerical attribute duration to improve the accuracy of the rule for $\mathit{checking}\_\mathit{status}=\mathit{0}<=\mathit{X}<\mathit{200}$. Also, as shown in Table 9, the rule set can be simpler, as in Section 4.4.1. The rule set generated by DT, while containing numerical attributes, was composed of simple rules.

5. Discussion

5.1. Why Should We Avoid a Mixture of Categorical and Numerical Attributes?

Many methods that have been proposed to enhance interpretability cannot handle categorical and numerical attributes separately. If we could adequately express a rule using only categorical attributes, the use of numerical attributes would reduce interpretability. Conditions for categorical attributes are easy to understand intuitively because they are categorized into a finite group. On the other hand, conditions for numerical attributes are difficult to understand intuitively because there are an infinite number of thresholds, so the division is not deterministic. Furthermore, it is not common for the division to be performed convincingly. For example, in the german dataset, there are only four conditions for the attribute $\mathit{checking}\_\mathit{status}\in \{\mathit{no} \mathit{checking},<\mathit{0},\mathit{0}<=\mathit{X}<\mathit{200},>=\mathit{200}\}$, whereas the conditions for the attribute $\mathit{credit}\_\mathit{amount}\in \mathbb{N}$ are infinite. In the division of the condition $\mathit{credit}\_\mathit{amount}\le 10841.5$ in $r_2$ in RuleCOSI+ in Table 8, it is difficult to understand why the value $10841.5$ was chosen. Furthermore, rules that contain many numerical attribute conditions make it more difficult to understand how each condition relates to the other. Setting thresholds for numerical attributes is infinite, and an intuitive understanding of how such thresholds affect the other attributes and the overall rules is difficult. Combining the conditions of multiple numerical attributes exponentially increases the complexity of the rule. By contrast, the conditions for categorical attributes are clustered in a finite group, which makes their relationships and effects easier to understand. Therefore, to realize high interpretability, mixing categorical and numerical attributes should be avoided.

5.2. Optimal Selection of the Pareto Solutions

In this study, we selected the Pareto optimal solution from the Pareto front obtained by multi-objective optimization with Optuna using Eq. (2). However, the Pareto optimal solution selected by Eq. (2) is not always the optimal solution sought by the user. As shown in Figure 5, we observed that the Pareto front depends significantly on the dataset, method, and data splitting. In other words, to select the optimal Pareto solution in real-world applications, it is desirable to verify the Pareto front individually.

5.3. Decision Lists v.s. Decision Trees

DTs delineate distinct, nonoverlapping regions within the training data, which affects the depth of the tree when representing complex regions. Conversely, decision lists are superior to DTs in that they allow overlapped regions in the feature space to be represented by different rules, thereby generating more concise rule descriptions for decision boundaries for the classification problem.

On the other hand, when interpreting rules corresponding to an instance, DTs only require tracing the node corresponding to the instance, whereas decision lists require concatenating rules until the instance is classified, which makes the rules more complex. In other words, even if the apparent size of the decision list is small, such as $N_{rule}$ and the average number of conditions, it is actually a very complex rule set with less interpretability than DTs. Therefore, if interpretability is important, DTs or DT-based rule sets are the better choice.

5.4. $CREP$ as a Metric of Interpretability

$CREP$ measures the complexity of a rule set based on empirical probabilities and is an intuitive metric of the interpretability of the rule set. $CREP$ can be used to compare the interpretability of both decision list- and DT-based rule sets because it is calculated separately for these models. RuleCOSI+, which is a decision list-based rule set that was concluded to have low interpretability in Section 4.4.1 and Section 4.4.2, has a high $CREP$ value compared with the other methods in Table 4, observing that examples and the indicator are consistent. From the above, we conclude that $CREP$ is an appropriate metric for evaluating the interpretability of rule sets.

5.5. Limitation

In this study, all datasets were used for the evaluation of binary classification. We found the Pareto optimal solution from the Pareto front obtained by multi-objective optimization using Eq. (2), but this equation is specialized for binary classification. In multi-class classification, effective results were not obtained using Eq. (2). For example, a solution in which there are no rules corresponding to a certain class was sometimes selected as the best solution. To solve this issue, it will be necessary to devise an equation specialized for multi-class classification.

6. Conclusions

In the present study, we compared the tree ensemble approximator with Re-RX with J48graft and showed the importance of handling categorical and numerical attributes separately. RuleCOSI+ obtained high interpretability on the measure with a small number of rules. However, the rules that are actually used to classify instances are complex and have quite low interpretability. On the other hand, Re-RX with J48graft obtained low interpretability on the measure with a large number of rules. However, it can handle categorical and numerical attributes separately, has simple rules, and achieves high interpretability, even when the number of rules is large. We newly proposed $CREP$ as a metric for interpretability, which is based on the empirical probability of the rules and measures their complexity. $CREP$ can be used for a fair comparison of decision list- and DT-based rule sets. Furthermore, by using macro-$CREP$, interpretability can be evaluated appropriately, even for class-imbalanced datasets. Few studies have considered handling categorical and numerical attributes separately, and we believe that this is an important issue for future work. Existing tree ensembles do not distinguish between categorical and numerical attributes because they prioritize accuracy. Therefore, they cannot be distinguished by tree ensemble approximators such as RuleCOSI+ and FBTs. In the future, we plan to develop a tree ensemble that distinguishes categorical from numeric attributes and serves as an approximator with even better interpretability.   

Appendix A.1. XGBoost

Implementation. We used XGBoost  10 for the base tree ensemble for FBTs and RuleCOSI+. We fixed and did not tune the following hyperparameters:

• $early\_stopping\_rounds=10$
• $n\_estimators=250$

In Table A1, we provide the hyperparameter space.

Table A1. XGBoost hyperparameter space.

Parameter	Space
$max\_depth$	UniformInt(1, 10)
$\eta$	LogUniform(1e-4,1.0)
# Iterations	50

Table A1. XGBoost hyperparameter space.

Parameter	Space
$max\_depth$	UniformInt(1, 10)
$\eta$	LogUniform(1e-4,1.0)
# Iterations	50

Appendix A.2. FBTs

Implementation. We used the official implementation of FBTs  11. We fixed and did not tune the following hyperparameters:

• $min\_forest\_size=10$
• $max\_number\_of\_conjunctions=1000$

In Table A2, we provide the hyperparameter space.

Table A2. FBTs hyperparameter space.

Parameter	Space
$max\_depth$	UniformInt(1, 10)
$pruning\_method$	{auc, None}
# Iterations	50

Table A2. FBTs hyperparameter space.

Parameter	Space
$max\_depth$	UniformInt(1, 10)
$pruning\_method$	{auc, None}
# Iterations	50

Appendix A.3. RuleCOSI+

Implementation. We used the official implementation of RuleCOSI+  12. In Table A3, we provide the hyperparameter space.

Table A3. RuleCOSI+ hyperparameter space.

Parameter	Space
$conf\_threshold$	Uniform(0.0, 0.95)
$cov\_threshold$	Uniform(0.0, 0.5)
c	Uniform(0.1, 0.5)
# Iterations	50

Table A3. RuleCOSI+ hyperparameter space.

Parameter	Space
$conf\_threshold$	Uniform(0.0, 0.95)
$cov\_threshold$	Uniform(0.0, 0.5)
c	Uniform(0.1, 0.5)
# Iterations	50

Appendix A.4. Re-RX with J48graft

Implementation. We used the repository 13 that we implemented for Re-RX with J48graft. We used $batch\_size=2^{\lfloor log\left(d\right)+0.5\rfloor }$, where d is the number of training data. In addition, we fixed and did not tune the following hyperparameters in the MLP:

• $epochs=200$
• $early\_stopping=10$
• $optimizer=AdamW$ [51]

In Table A4, we provide the hyperparameter space of the MLP. We searched for the optimal parameters of the MLP and then the other parameters of Re-RX with J48graft. In Table A5, we provide the hyperparameter space for the other parameters of Re-RX with J48graft.

Table A4. MLP hyperparameter space.

Parameter	Space
$dim$	UniformInt(1, 5)
$learning\_rate$	LogUniform(5e-3, 0.1)
$weight\_decay$	LogUniform(1e-6, 1e-2)
# Iterations	50

Table A4. MLP hyperparameter space.

Parameter	Space
$dim$	UniformInt(1, 5)
$learning\_rate$	LogUniform(5e-3, 0.1)
$weight\_decay$	LogUniform(1e-6, 1e-2)
# Iterations	50

Table A5. Re-RX with J48graft hyperparameter space.

Parameter	Space
$j48graft.min\_instance$	{2, 4, 8, …, 128}
$j48graft.pruning\_threshold$	Uniform(0.1, 0.5)
$pruning\_lamda$	LogUniform(0.001, 0.25)
${\delta }_1$	Uniform(0.05, 0.4)
${\delta }_2$	Uniform(0.05, 0.4)
# Iterations	50

Table A5. Re-RX with J48graft hyperparameter space.

Parameter	Space
$j48graft.min\_instance$	{2, 4, 8, …, 128}
$j48graft.pruning\_threshold$	Uniform(0.1, 0.5)
$pruning\_lamda$	LogUniform(0.001, 0.25)
${\delta }_1$	Uniform(0.05, 0.4)
${\delta }_2$	Uniform(0.05, 0.4)
# Iterations	50

Appendix A.5. DT

Implementation. We used scikit-learn’s DT 14. In Table A6, we provide the hyperparameter space. We used the default hyperparameters of scikit-learn for the other parameters.

Table A6. DT hyperparameter space.

Parameter	Space
$max\_depth$	UniformInt(1, 10)
$min\_samples\_split$	Uniform(0.0, 0.5)
$min\_samples\_leaf$	Uniform(0.0, 0.5)
# Iterations	100

Table A6. DT hyperparameter space.

Parameter	Space
$max\_depth$	UniformInt(1, 10)
$min\_samples\_split$	Uniform(0.0, 0.5)
$min\_samples\_leaf$	Uniform(0.0, 0.5)
# Iterations	100

Appendix A.6. J48graft

Implementation. We used J48graft as implemented in the rerx repository 15. Table A7 shows the hyperparameter space.

Table A7. J48graft hyperparameter space.

Parameter	Space
$min\_instance$	{2, 4, 8, …, 128}
$pruning\_threshold$	Uniform(0.1, 0.5)
# Iterations	100

Table A7. J48graft hyperparameter space.

Parameter	Space
$min\_instance$	{2, 4, 8, …, 128}
$pruning\_threshold$	Uniform(0.1, 0.5)
# Iterations	100