A Novel Tsetlin Machine with Enhanced Generalization

1. Introduction

The Tsetlin Machine (TM) is a novel machine learning algorithm first proposed in 2018 by Granmo et al. [1]. It is based on the Tsetlin Automaton (TA). The TM is able to find complex patterns using propositional logic. It learns by performing an action that can trigger a reward or penalty. The TM can have applications in not only pattern recognition but also in other domains and tasks, such as regression, healthcare, and natural language processing. TMs have the advantage of low memory usage and increased learning speed while achieving competitive predictive performance in several benchmark datasets [2,3]. The TM can easily be implemented using simple low-energy consumption hardware [4,5].

Using propositional logic in the TM has many added advantages over traditional machine learning algorithms. Propositional logic-based learning is more interpretable, and the results are easy for humans to understand. Hence, the TM has important applications in the field of explainable AI, where interpretability and transparency are important for machine learning algorithms [6,7]. Propositional logic-based algorithms are also much faster in execution time than traditional neural networks that are based on multiple layers using vector multiplication. In fact, some traditional machine learning algorithms have proposed using propositional logic based algorithms for learning [8,9].

In the TM, propositional logic is used to create clauses. The clauses are made from literals which are features of the data. The literals within the clauses are learned using Type I or Type II feedback based on the feedback from the classification result. The predicted class is based on the majority sum from the clauses which is then obtained by a step-function defined by the parameter T. The number of clauses and the parameter T are the main hyperparameters that increase the discriminative power of the TM and enable patterns to be learned more efficiently.

Increasing the number of clauses and having a high value of T, may lead to better predictive power of the TM but it may also lead to overfitting and an increase in resources used and a reduction in training speed.

In this paper, we propose a novel variant of the TM called Regularized TM (RegTM). RegTM is based on the idea of using regularizers and smoothness functions, just like in traditional machine learning algorithms [10]. We use two different variants of regularizers. The first regularizer, called the Moving Average Regularizer (MAR), uses the sum of clauses from previous iterations and samples and incorporates them into the current clause sum calculation. The second type of regularizer, called the Weighted Regularizer (WER), adds the sum of the weights to improve generalization. By using these regularization techniques, the generalization capability of the TM can improve, increasing their predictive power, especially in cases where the data is low in count.

Furthermore, we also propose the use of the Sigmoid function [11,12]. The sigmoid function can make the TM differentiable, and it can be further optimized with standard methods like gradient descent.

In summary, the contributions of our paper are as follows:

• We propose a novel variant of the Tsetlin Machine called Regularized Tsetlin Machine (RegTM) that implements regularization based on past clause sums and weights to improve the accuracy of the TM by increasing its generalizability, making it highly suitable for few-shot learning and meta-learning optimization tasks.
• We propose to use the Sigmoid function as an alternative to the unit-step function.
• We perform different experiments using benchmark image datasets to show the efficiency of RegTM with the different regularizers and sigmoid function.

2. Related Works

Granmo et al. proposed the TM in 2018 as a way to provide solutions to the bandit algorithm [1,13,14]. Since then, there has been a lot of significant work in this domain to improve upon the existing idea.

The TM was recast as a contextual bandit algorithm in [15]. Two contextual bandit learning algorithms were proposed: TM with $ϵ$-greedy arm selection and TM with Thompson sampling.

TM was extended for image classification and was called the Convolution Tsetlin Machine (CTM) [16,17]. CTM employed clauses as a convolution filter on each image as opposed to one clause for each image.

In addition to image classification, TM has also been used for regression [18]. The Regression TM is able to output in a continuous form rather than as distinct categories. The Regression TM employs a new voting mechanism to analyze complex patterns.

For multi-output problems, a coalesced Tsetlin Machine (CoTM) was proposed that simultaneously learns both the weights and the composition of each clause by employing interacting Stochastic Searching on the Line (SSL) and Tsetlin Automata (TA) [19].

TM has also shown great potential for natural language processing, e.g., text encoding [20], text classification [21,22,23], sentiment analysis [24], question classification [25], and fake news detection [26].

TM has also been implemented in the domain of healthcare. For example, a TM architecture was developed for the identification of premature ventricular contraction (PVC) by analyzing long-term ECG signals [27]. The capability of TM using continuous data was proposed for forecasting disease outbreaks [18]. Intelligent prefiltering of healthcare measurement data using Principal Component Analysis and Partial Least Squares Regression was added to TM to improve performance [28].

A label-critical TM was proposed that used a twin of label-critical Tsetlin Automata (TA). Label-TA learned the correct labels of the individual samples guided by a logical self-corrected TM clause [29].

There has also been substantial work on improving the feasibility of TM. One of the ways of improving the TM was by focusing on the clauses. The accuracy of the TM was improved with the addition of a drop clause probability [30]. The drop clause probability dropped clauses with a specific probability to increase the robustness, accuracy, and training time of TM, similar to a dropout layer in neural networks.

Another improvement to TM was the addition of weights called the Weighted Tsetlin Machine (WTM) [31]. The Weighted Tsetlin Machine (WTM) adds weights to the clauses to reduce the computation time and memory usage.

Focused negative sampling (FNS) was introduced to discriminate between classes that are not easily distinguishable from each other to improve TM precision [32].

A method to more efficiently prune clauses using the Markov boundary was proposed [33]. The new scheme aimed to add an additional feedback scheme to the TM, called the Type III feedback.

Another proposed scheme added the absorbing automata to the Tsetlin Machine [34]. Another variant of TM focusing on improving the efficiency of clauses was called the Clause Size Constrained TM (CSC-TM) [35]. In CSC-TM, a soft constraint on the clause size is placed and as soon as the clause includes more literals, that is, more than the allowed constraint, the literals are removed. Another research work looked at finding the optimal clause count by proposing a run-time pruning system [36].

3. Preliminary: Vanilla Tsetline Macnhine

The TM is able to solve complex pattern recognition problems using proposition-based conjunctive logic statements [1]. A standard Tsetlin Automaton (TA) consists of a collection of TAs [37] to find solutions to the multi-armed bandit problem [38,39] and the Finite State Learning Algorithm (FSLA) [40]. The TM uses conjunctive clauses composed of a collection of TAs.

In this section, we provide an overview of the TM to help us understand the algorithm introduced in subsequent sections. The TM takes a vector X of n propositional variables as input within the domain 0,1 and is defined as:

$$\begin{array}{c}\hfill \mathbf{X}=x_k\in {0,1}^n\end{array}$$

(1)

For example, if n=2, the variables are $x_1$ and $x_2$, and the domain is (0,0),(0,1),(1,0),(1,1).

For example, if $n=2$, the variables are $x_1$ and $x_2$, then the domain is $\left\{\right(0,0),(0,1),$$(1,0),(1,1)\}$.

The TM determines patterns using conjunctive clauses. A conjunctive clause consists of literals and is meant to describe the dataset. A literal consists of the variables and their negations $\mathbf{L}=l_k\in {0,1}^{2n}$ where each literal $l_k$ is defined as:

$$\begin{array}{c}\hfill l_k=\left\{\begin{array}{cc}{x}_k\hfill & if1\le k\le n,\hfill \\ \neg x_{k-n}\hfill & ifn+1\le k\le 2n\hfill \end{array}\right.\end{array}$$

(2)

The first n literals are the actual variables and the next n variables are the negated variables. Using the example of $n=2$, $L=\{x_1,x_2,\neg x_1,\neg x_2\}$.

Typically, the number of clauses is a user set parameter. It is an important parameter that determines how well training is done [36]. Let the number of clauses be defined as m and each clause be denoted as $C_j$ where $j\in \{1,...,m\}$, then the clause is a conjunction of a subset of literals $L_j\subseteq L$:

$$\begin{array}{c}\hfill C_j\left(\mathbf{x}\right)=\underset{l_k\in L_j}{\bigwedge }{l}_k=\prod _{l_k\in L_j}{l}_k\end{array}$$

(3)

$C_j\in \{0,1\}$. Let $m=2$ and the user defined clauses are $C_1\left(\mathbf{x}\right)=x_1\wedge \neg x_2$ and $C_2\left(\mathbf{x}\right)=x_1\wedge x_2$. Then, from the previous example, the literals have the index $L_1=\{1,4\}$ and $L_2=\{1,2\}$.

Each clause is also assigned a polarity $p\in \{-1,+1\}$. The polarity decides whether the clause output is negative or positive. Positive polarity clauses, denoted by $C_j^{+}\in {\{0,1\}}^{{\displaystyle \frac{m}{2}}}$, are assigned to class $y=1$ and negative polarity clauses, denoted by $C_j^{-}\in {\{0,1\}}^{{\displaystyle \frac{m}{2}}}$, are assigned to class $y=0$. Typically, half of the clauses are positive and half are negative. The positive polarity clauses vote for classifying the input as the target class and the negative polarity clauses vote against the target class.

The final classification decision is done by summing the clause outputs:

$$\begin{array}{cc}\hfill s\left(\mathbf{x}\right)& =\sum _{j=1}^{m/2}{C}_j^{+}-\sum _{j=1}^{m/2}{C}_j^{-}\hfill \\ \hfill \widehat{y}& =u\left(s\right(\mathbf{x}\left)\right)\hfill \\ \hfill \end{array}$$

(4)

where $u\left(v\right)$ is the unit step function defined as:

$$u\left(v\right)=\left\{\begin{array}{cc}1\hfill & ifv\ge 0\hfill \\ 0\hfill & otherwise\hfill \end{array}\right.$$

(5)

In the end, the output of the sum of summation decides the class $\widehat{y}\in \{0,1\}$.

Equation (4) was modified in the by adding weights to the clauses Weighted Tsetlin Machine [31]:

$$\begin{array}{c}\hfill s\left(\mathbf{x}\right)=\sum _{j=1}^{m/2}{w}_j^{+}{C}_j^{+}-\sum _{j=1}^{m/2}{w}_j^{-}{C}_j^{-}\end{array}$$

(6)

For example, Figure 1 shows two positive polarity clauses, $C_1\left(x\right)=x_1\wedge \neg x_2$ and $C_3\left(x\right)=\neg x_1\wedge \neg x_2$ (negative polarity clauses are not shown). Both clauses evaluate to zero, and leading to a prediction of $\widehat{y}=1$.

For training, each clause is given feedback with a probability $\widehat{p}$.

$$\begin{array}{c}\hfill \widehat{p}=\left\{\begin{array}{cc}{\displaystyle \frac{T+clip\left(s\right(x),-T,T)}{2T}}\hfill & ify=0\hfill \\ {\displaystyle \frac{T-clip\left(s\right(x),-T,T)}{2T}}\hfill & ify=1\hfill \end{array}\right.\end{array}$$

(7)

In the above equation, $s\left(x\right)$ is the sum of the clause of Equation (6). T is a user-defined parameter. The $clamp$ function scales the $s\left(x\right)$ between $-T$ and T.

The above random clause selection is crucial as it guides the clauses to distribute across the sub-patterns in a more efficient manner. The frequency of each clause becomes proportional to $\pm T$. The amount of feedback depends on this and more feedback is only generated when $s\left(x\right)$ is far from T. In this way, only a few clauses are activated, thus reducing the overhead during training.

The TA is depicted in the upper part of Figure 1. The TA determines the actions to take during learning. The TA can decide to include or exclude the literal in the clause based on the received feedback. The feedback can be categorized as Type I or Type II. Figure 2 depicts a two-action TA with 2N states. For states $1\cdots N$, the TA performs exclude (Action 1) and for states $N+1\cdots 2N$, include (Action 2) actions are performed. Based on the feedback, a TA can receive a reward or a penalty. If a TA receives a reward, then the literal moves deeper into being included or excluded. If a TA receives a penalty, then the literal moves towards the middle until it moves from action 1 to action 2 and vice versa.

Type I feedback reduces false negatives and overfitting by frequently recognizing patterns. Type II feedback suppresses false positives by increasing the discrimination power of the learned patterns.

Type I feedback: In Type I feedback, clauses of positive polarity receive a positive feedback when $y=1$ and negative feedback for clauses with negative polarity and when $y=0$. There are two types of Type I feedback: Type 1a and Type1b. Type 1a feedback reinforces the TA include actions. Type 1b feedback is activated when either the clause output or the literal is 0. Type 1b causes the TA to exclude the literal.

Type I feedback is defined by a user-defined parameter s where $s\ge 1$. Let the TA state be defined by $a_{j,k}$ for action on the $k^{th}$ literal and the $j^{th}$ clause. The TA state $a_{j,k}$ receives Type 1a feedback with probability ${\displaystyle \frac{s-1}{s}}$ and increases $a_{j,k}$ by 1. Consequently, the TA receives Type 1b feedback with probability ${\displaystyle \frac{1}{s}}$ and reduces $a_{j,k}$ by 1. In this way, Type 1a pushes to include the literal and Type 1b pushes to exclude the literal.

Type II feedback: Type II feedback is given to clauses with positive polarity when $y=0$ and to clauses with negative polarity when $y=1$. When the clause output is 0, the TA states are not updated. When the clause output is 1, all 0-valued literals that make the output 0 in TA state $a_{j,k}$ are increased by 1. Thus, this feedback focuses on all literals that differentiate between $y=0$ and $y=1$.

4. Methodology

In this section, we introduce our novel TM variant titled the Regularized Tsetlin Machine (RegTM).

RegTM is based on the novel idea of using regularization and smoothing for Tsetline Machines. Regularization has been used extensively in machine learning algorithms. It is a way of infusing knowledge about data and parameters into the learning algorithm and provides a good balance between bias and variance. Regularization adds bias to the learning process and helps reduce the variance.

Regularization can reduce overfitting and improve the generalizability of the model [41].

One of the most common methods of regularization is to constrain the capacity of a model by adding some penalty function to the original objective function and making a new objective function. In TM we consider the objective function as the weighted clause sum equation represented in Equation (6). Hence, the new clause summation equation in RegTM can be written as:

$$\begin{array}{c}\hfill s_{RegTM}\left(x\right)=\sum _{j=1}^{m/2}(w_j^{+}{C}_j^{+}+b_j^{+})-\sum _{j=1}^{m/2}(w_j^{-}{C}_j^{-}-b_j^{-})\end{array}$$

(8)

where $b_j^{+}$ and $b_j^{-}$ are the two regularization terms for the positive and negative classes.

Next we will describe the two regularization methods we propose for implementation in RegTM.

Moving Average Regularizer (MAR): The first regularization method we explore is inspired by the moving average term used in time series analysis [42]. In this case we assume that the the short-term fluctuations between clause sums can cause the model to be less generalizable. To remove the fluctuation, we propose a bias function that is dependent on the past values of the clause sums. We call this moving average regularizer (MAR).

Consequently, $b_j^{+}$ and $b_j^{-}$ from Equation (8) can be written as:

$$\begin{array}{c}\hfill b_j^{+}=p({\displaystyle \frac{1}{k}}\sum _{n=1}^k{\mathtt{C}}_{j-n}^{+})\end{array}$$

(9)

$$\begin{array}{c}\hfill b_j^{-}=p({\displaystyle \frac{1}{k}}\sum _{n=1}^k{\mathtt{C}}_{j-n}^{-})\end{array}$$

(10)

where p is a weighting parameter to define how much weight we want to put on the regularizer terms, k is a parameter that determines how far into the past we want to consider the clause sums, and ${\mathtt{C}}_{j-k}^{+}$ and ${\mathtt{C}}_{j-k}^{-}$ are the positive and negative clause sums of the sample ${(j-k)}^{th}$:

$$\begin{array}{c}\hfill {\mathtt{C}}_{j-k}^{+}=\sum _{j=1}^{m/2}{w}_j^{+}{C}_j^{+}\end{array}$$

(11)

$$\begin{array}{c}\hfill {\mathtt{C}}_{j-k}^{-}=\sum _{j=1}^{m/2}{w}_j^{-}{C}_j^{-}\end{array}$$

(12)

Weighted Regularizer (WER): The second method of regularization is based on weight decay [41], which we call Weighted Regularizer (WER). For WER, the bias terms from Equation (8) are expressed as:

$$\begin{array}{c}\hfill b_j^{+}=p\sum _{j=1}^{m/2}{\left(w_j^{+}\right)}^k\end{array}$$

(13)

$$\begin{array}{c}\hfill b_j^{-}=p\sum _{j=1}^{m/2}{\left(w_j^{-}\right)}^k\end{array}$$

(14)

where p is a parameter to define how much weight we want to put on the regularizer, k is the power of the weights, and ${\left(w_j^{+}\right)}^k$ and ${\left(w_j^{-}\right)}^k$ are weights from Equation (8) for the $k^{th}$ sample.

Sigmoid: In this paper, we also introduce an alternative to Equation (7) for the feedback mechanism of the classes. One of the drawbacks of Equation (7) is that it is non-differentiable and cannot be optimized via standard methods such as gradient descent. This might be problematic if the TM is to be integrated with traditional neural networks for hyperparameter optimization and for providing explainability to neural networks.

Hence, we propose the use of the Sigmoid function. The Sigmoid function is a popular mathematical function that has an S-shaped curve and works in a similar way to the clamp function [9]. The sigmoid function maps values between $0,1$. The sigmoid function is defined as:

$$\begin{array}{c}\hfill \sigma \left(s\left(x\right)\right)={\displaystyle \frac{1}{1+ex{p}^{-s\left(x\right)}}}\end{array}$$

(15)

5. Experiments

In this section, we introduce the experimental parameters, data used for the experiments, and the experimental results.

5.1. Experimental Design

All of our experiments are performed using the Github repository of the TM 1.

We use the two standard benchmark datasets (MNIST and CIFAR10) for vision classification. The MNIST database contains 60,000 training images and 10,000 testing images. The dataset is a set of 28x28 grayscale images of 10 digits. The CIFAR10 dataset consists of 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.

The benchmark methods for comparison are the weighted Tsetlin Machine with literal and drop clauses [30,31]. The RegTM implementation includes using MAR and WER with and without the sigmoid function.

5.2. Experiment Results

The accuracy of the test dataset after 60 epochs is summarized in Table 1 and Table 2. The column TM reports the results with vanilla weighted TM. The column with drop probability reports the results with drop clause and drop literal probabilities included in the algorithm. The results are reported for different values of clause counts (C).

The drop probabilities are obtained by using the literal and clause drop probability that results in the best accuracy. For MNIST, the best accuracy is obtained when both clause drop and literal drop probabilities are 0.1. For CIFAR10, the best accuracy is obtained when the clause drop probability and the literal drop probability are 0.4 and 0.0, respectively.

Similarly, we also use the values for k and p that give the highest accuracy. For the MNIST dataset, when considering MAR, the best accuracy is obtained when k=0 and p=0.03. For MAR with sigmoid, the best accuracy is obtained when k=1 and p=0.1. Similarly, for WER the best accuracy is obtained when k=2 and p=0.4 and for WER with sigmoid, by setting k=3 and p=0.01 gives the best accuracy.

For the CIFAR10 dataset, the best accuracies are given by $k=0,p=0.8$ for MAR, $k=3,p=0.5$, for MAR and sigmoid, $k=3,p=0.1$ for WER, and $k=2,p=0.7$ for WER and sigmoid.

In general, using RegTM regularizers improves the accuracy of the TM. In addition, the improvement in accuracy by using RegTM is better when compared to using drop literals and drop clauses. In fact, for the MNIST dataset, adding drop probability appears to have reduced the accuracy. Furthermore, combining RegTM with drop clauses improves accuracy even further, especially for the CIFAR10 dataset where the accuracy is already low.

For RegTM with only the sigmoid function (denoted by sigmoid), the accuracy is very close to the vanilla TM for both the MNIST and CIFAR10 datasets. This means that the sigmoid function can be used as a viable replacement for Equation (7), especially when there is a need to develop a differentiable model.

When comparing different datasets, the improvement in accuracy for MNIST is much lower compared to CIFAR10. This is because MNIST already has a high accuracy, so improvements in its accuracy are very small.

The results show that RegTM with WER has higher accuracy than MAR, and when it does have lower accuracy (e.g., in the case of the MNIST dataset), the difference is very low. Adding a sigmoid function to MAR and WER also appears to not affect the performance of MAR and WER and in some cases improves the accuracy.

5.3. Ablation Studies

Our ablation studies include observing the effect of the parameter k and p on the accuracy and comparing the runtime of RegTM with TM and TM with drop probabilities.

The effect of the parameters k and p on the accuracy for both the MNIST and CIFAR10 datasets is shown in Figure 3, Figure 4, Figure 5 and Figure 6. It is observed that for a specific k value and any value of p, the accuracy drops sharply. In the case of the MAR algorithm, significantly lower accuracy is obtained when $k=2$ and for the WER algorithm, when $k=0$ much lower accuracy is observed. We will study this in more detail in future work. Across the rest of the k and p values, there is very little variation in the accuracy values. For MNIST the accuracy is almost the same. For CIFAR10, the variation is slightly more which may be due to the complexity of the dataset.

We also compare the runtime over 60 epochs for the different regularizers. The run-time of RegTM with MAR and WER is comparable to that of the vanilla TM and better than the use of dropping clauses and literals. Adding the sigmoid function does increase the run-time compared to vanilla TM and TM with drop clauses, but the increase is not so significant.

6. Conclusion

In this paper, we introduce a novel variant of the Tsetlin Machine called the Regularized Tsetlin Machine (RegTM) by proposing the idea of using regularizers to improve the performance and generalization of the TM. Even though the improvement in the results is marginal, it opens up new potential for future research, where generalization is important, especially in the field of few-shot learning and meta-optimization [43,44].

Our work shows promise and improves upon the performance of TM. However, there are some limitations, epsecally in case of TM. In some cases, the explanatory advantages that TM provides are not intuitive. Understanding and making the interpretations provided by TMs are important. But this is an on-going research. In our paper, we do not specifically focus on the interpretability of TM but rather on increasing its generalizability through regularizers.

Another limitation of TM is its inefficiency in handling large-scale data. This is because of the increase in the Tsetlin Automata, which can cause memory usage problems and reduce computational efficiency. One way of dealing with this issue is by using Sparse Tsetlin Machine [45] and parallelization [1]. However, it is still an active area of research that still requires further attention. In our paper, we do not focus on how to handle large-scale data. However, we believe regularizers could be used in combination with Sparse Tsetlin Machines to improve performance.

Another concern regarding TM is how to select the different hyperparameters. Currently, the way to select hyperparameters is using grid search, random search, cross-validation or using Bayesian optimization [1,46]. We have shown that using the sigmoid function does not reduce the performance of TM. This means that gradient descent could be used in combination with TM, allowing us to apply the same optimization techniques as those from traditional neural networks. Hence, we can implement other hyperparameter optimization techniques in TM that are also used in neural networks, which would be an interesting area for future research.

Another limitation of TM is that it can only handle binary features. This means that continuous features can only be handled by discretization, which may lead to information loss. This has been discussed in [47] but it still requires further research and investigation of techniques that can reduce information loss.

The main focus of TM has been in the field of classification and regression tasks, but its application in other tasks (clustering, segmentation, and sequence modeling) is still limited. Research on the application of TM in other tasks is ongoing. Our work on regularizers aims to be a step towards the application of TM in other domains and tasks, more specifically in few-shot learning. The applicability of TM for other tasks would be another potential future research area.

Another interesting area of study can involve investigating more complex regularizers, such as tangentprop [48]. However, implementing such regularizers and even the sigmoid function might be challenging considering that TMs are mainly designed for low-energy hardware. This is something that could also be further studied.