Fuzzy Clustering with Uninorm‐Based Distance Measure

1. Introduction

In general, decision making starts with formulation of possible alternatives, collecting required data, data analysis, formulation of the decision criteria and choice of the alternatives based on the formulated criteria [1].

In simple cases of decision making with certain information and univariate data, decision making process is reduced to the choice of alternatives which minimize the payoff or maximize the reward. In cases of uncertain information, decision making is more complicated and deals with the expected payoffs and rewards using probabilistic or other methods for handling uncertainty. Finally, in cases of multivariate data and multicriteria choices, decision making is hardly solvable by exact methods and can be considered as a kind of art.

For example, Triantaphyllou in his book [2] describes several methods of multicriteria decision making without suggesting any preferred technique and indicates that each specific decision-making problem requires specific method for its solution. Such an approach is supported by the authors of the book [3] who stress “a growing need for methodologies that, on the basis of the ever-increasing wealth of data available, are able to take into account all the relevant points of view, technically called criteria, and all the actors involved” [3] (p. ix) and then present classification of the methods for solving different decision-making problems.

To apply the methods of decision making the raw data are preprocessed with the aim of recognition of possible patterns and consequently – to decrease the number of instances and, if it is possible, the number of dimensions [4]. The basic preprocessing procedure which decreases the number of instances used in the next steps of decision making is data classification [5,6] or, if any additional information is unavailable – data clustering [7,8].

The popular clustering method is the k-means algorithm suggested independently by Lloyd [9] and by Forgy [10] and is known as Lloyd-Forgy algorithm. This algorithm obtains $n$ instances and creates $k$ clusters such that each instance is included into a single cluster for which Euclidean distance between the instance and the cluster’s center is minimal. The clusters are formed from the instances for which the within-cluster variances are also minimal. The FORTRAN implementation of the algorithm was published by Hartigan and Wong [11]. Currently, this algorithm is implemented in most statistical and mathematical software tools; for example, in MATLAB^® it is implemented in the Statistics and Machine Learning Toolbox™ in the function kmeans [12].

Main disadvantage of the k-means algorithm and its direct successors is an inclusion of each instance only in a single cluster that often restricts recognition of the data patterns and interpretation of the obtained clusters.

To overcome this disadvantage Bezdek [13,14] suggested the fuzzy clustering algorithm widely known as the fuzzy c-means (FCM) algorithm. Following this algorithm, the clusters are considered as fuzzy sets, and each instance is included in several clusters with certain degrees of membership. In its original version, the algorithm uses Euclidean distances between the instances and the degrees of membership are calculated based on these distances. The other versions of the algorithm follow its original structure but use the other distance measures. For example, the Fuzzy Logic Toolbox™ of MATLAB^® includes the function fcm [15], which implements the c-means algorithm with three distance measures: Euclidean distance, distance measure based on Mahalanobis distance from the instances and the cluster centers [16] and exponential distance measure normalized by the probability of choosing the cluster [17]. For the overview of different versions of the c-means algorithm and the other fuzzy clustering methods see the paper [18] and the book [19].

Along with the advantages of the fuzzy c-means clustering techniques, it inherits the following disadvantage of the k-means algorithm: both algorithms search for the predefined number of clusters and separate the cluster centers even in cases where such separation does not follow from the data structures. For example, if the data is a set of normally distributed instances, then it is expected that the clustering algorithm will recognize a single cluster, and the centers of all clusters will be placed close to the center of distribution. However, the c-means algorithm with the known distance measures does not provide such a result.

The suggested c-means algorithm with the uninorm-based distance solves this problem. Similar to the other fuzzy c-means algorithms, the suggested algorithm has the same structure as original Bezdek algorithm, but, in contrast to the known methods, it considers the normalized instances as truth values of certain propositions and implements fuzzy logical distance between them. The suggested distance is based on the probability-based uninorm and absorbing norm [22] which already demonstrated their usefulness in a wide range of decision-making and control tasks [23].

The suggested algorithm can be used for recognition of the patterns in raw data and for data preprocessing in different decision-making problems.

The rest of the paper is organized as follows. In section 2.1 we briefly outline the Bezdek fuzzy c-means algorithm and in section 2.2 describe the uninorm and absorbing norm which will be used for construction the logical distance. Section 3 presents the main results: the distance measure based on the uninorm and absorbing norm (section 3.1) and the resulting algorithm for fuzzy clustering (section 3.2). In section 3.3 we illustrate the activity of the algorithm by numerical simulations with different data distributions and compare the obtained results with the results provided by the known fuzzy c-means algorithms. Section 4 includes some discussable issues.

2. Materials and Methods

The suggested algorithm follows the structure of the Bezdek fuzzy c-means algorithm [13,14] but, in contrast to the existing algorithms, it uses the logical distance based on the uninorm and absorbing norm [20]. Below we briefly describe this algorithm and the used norms.

2.1. Fuzzy c-Means Algorithm

Assume that the raw data is given by $d$-dimensional vectors of real numbers

$$X=\left(x_1,x_2,\dots ,x_n\right)\subset {ℝ}^d,$$

where $x_i=\left(x_{i1}, x_{i2},\dots ,x_{id}\right)$, $i=1, 2,\dots , n$, $n\ge 1$, $d\ge 1$, stands for the observation or instance of the data.

Given a number $m\le n$, denote by

$$Y=\left(y_1,y_2,\dots ,y_m\right)\subset {ℝ}^d$$

a vector of the cluster centers, where $y_j=\left(y_{j1}, y_{j2},\dots ,y_{jd}\right)$, $j=1, 2,\dots , m$, $m\ge 1$, $d\ge 1$, denotes the center of the $j$th cluster.

The problem is to define the centers $y_j$ of the clusters and to define the membership degree ${\mu }_{ij}\in \left[0, 1\right]$ of each instance $x_i$ to each cluster $j$.

The fuzzy c-means algorithm which solves this problem is outlined as follows [13] (see also [14,15,17]).

In the original version [13], the fuzzy c-means algorithm uses the Euclidean distance

$${\mathrm{dist}}^d\left(x_i,y_j\right)=\sqrt{{{\displaystyle \sum }}_{l=1}^d{\left(x_{il}-y_{jl}\right)}^2},$$

and in the succeeding versions [16,17] the other distance measures were applied.

In this paper, we suggest to use the logical distance measure based on the uninorm and absorbing norm which results in more accurate recognition of the cluster centers.

2.2. Uninorm and Absorbing Norm

Uninorm [22] and absorbing norm [23] are the operators of fuzzy logic which extend Boolean operators and are used for aggregating truth values.

Uninorm aggregator ${\oplus }_{\theta }$ with neutral element $\theta \in \left[0,1\right]$ is a function ${\oplus }_{\theta }:\left[0,1\right]\times \left[0,1\right]\to \left[0,1\right]$ satisfying the following properties [22]; $x,y,z\in \left[0,1\right]$:

a.

Commutativity: $x{\oplus }_{\theta }y=y{\oplus }_{\theta }x$,

b.

Associativity: $\left(x{\oplus }_{\theta }y\right){\oplus }_{\theta }z=x{\oplus }_{\theta }\left(y{\oplus }_{\theta }z\right)$,

c.

Monotonicity: $x\le y$ implies $x{\oplus }_{\theta }z\le y{\oplus }_{\theta }z$,

d.

Identity: $\theta {\oplus }_{\theta }x=x$ for some $\theta \in \left[0,1\right]$,

and such that if $\theta =1$, then $x{\oplus }_{1}y=x\wedge y$ is a conjunction operator and if $\theta =0$, then $x{\oplus }_{0}y=x\vee y$ is a disjunction operator.

An absorbing norm aggregator ${\otimes }_{\vartheta }$ with absorbing element $\vartheta \in \left[0,1\right]$ is a function ${\otimes }_{\vartheta }:\left[0,1\right]\times \left[0,1\right]\to \left[0,1\right]$ satisfying the following properties [23]; $x,y,z\in \left[0,1\right]$:

a.

Commutativity: $x{\otimes }_{\vartheta }y=y{\otimes }_{\vartheta }x$,

b.

Associativity: $\left(x{\otimes }_{\vartheta }y\right){\otimes }_{\vartheta }z=x{\otimes }_{\vartheta }\left(y{\otimes }_{\vartheta }z\right)$,

c.

Absorbing element: $\vartheta {\otimes }_{\vartheta }x=\vartheta$ for any $x\in \left[0,1\right]$.

Operator ${\otimes }_{\vartheta }$ is a fuzzy analog of the negated $xor$ operator.

Let $u_{\theta }:\left(0,1\right)\to \left(-\infty ,\infty \right)$ and $v_{\vartheta }:\left(0,1\right)\to \left(-\infty ,\infty \right)$ be invertible, continuous, strictly monotonously increasing functions with the parameters $\theta ,\vartheta \in \left[0,1\right]$ such that

a.

${\lim }_{x\to 0}{u}_{\theta }\left(x\right)=-\infty$, ${\lim }_{x\to 0}{v}_{\vartheta }\left(x\right)=-\infty$,

b.

${\lim }_{x\to 1}{u}_{\theta }\left(x\right)=+\infty$, ${\lim }_{x\to 1}{v}_{\vartheta }\left(x\right)=+\infty$,

c.

$u_{\theta }\left(\theta \right)=\theta$, $v_{\vartheta }\left(\vartheta \right)=\vartheta$.

Then, for any $x,y\in \left(0,1\right)$ hold [24]

$$\begin{gathered} x{\oplus }_{\theta }y=u_{\theta }^{-1}\left(u_{\theta }\left(x\right)+u_{\theta }\left(y\right)\right), \\ x{\otimes }_{\vartheta }y=v_{\vartheta }^{-1}\left(v_{\vartheta }\left(x\right)\times v_{\vartheta }\left(y\right)\right), \end{gathered}$$

and for the bounds $0$ and $1$ the values of the operators ${\oplus }_{\theta }$ and ${\otimes }_{\vartheta }$ are defined in correspondence to the results of Boolean operators.

If $\theta =\vartheta$ and $u\equiv v$, then the interval $\left[0,1\right]$ with the operators ${\oplus }_{\theta }$ and ${\otimes }_{\vartheta }$ form an algebra [20,25]

$$\mathcal{A}=\left[0,1\right],{\oplus }_{\theta },{\otimes }_{\vartheta }$$

in which uninorm ${\oplus }_{\theta }$ acts as a summation operator and absorbing norm ${\otimes }_{\vartheta }$ acts as a multiplication operator. In this algebra, neutral element $\theta$ is zero value for summation and absorbing element $\vartheta$ is a unit value for multiplication. Note that in contrast to the algebra of real numbers, in the algebra $\mathcal{A}$ holds $\theta =\vartheta$ despite different meanings of these values.

The inverse operators in this algebra are

$$\begin{gathered} x{\ominus }_{\theta }y=u_{\theta }^{-1}\left(u_{\theta }\left(x\right)-u_{\theta }\left(y\right)\right),x,y\in \left(0,1\right) \\ x{\oslash }_{\vartheta }y=v_{\vartheta }^{-1}\left(v_{\vartheta }\left(x\right)/v_{\vartheta }\left(y\right)\right),v\left(y\right)\ne 0. \end{gathered}$$

In addition, negation operator is defined as

$${\ominus }_{\theta }x=u_{\theta }^{-1}\left(-u_{\theta }\left(x\right)\right),x\in \left(0,1\right).$$

In the paper [20] it was demonstrated that the functions $u_{\theta }$ and $v_{\vartheta }$ satisfy the requirements of quantile functions of probability distributions. Along with that any function satisfying the indicated above requirements and its inverse are also applicable.

Here we will assume that the functions $u_{\theta }\equiv v_{\vartheta }$ are equivalent. In the simulations, we will use the functions

$$u_{\theta }\left(x\right)=\ln {\displaystyle \frac{x^{\alpha }}{1-x^{\alpha }}},x\in \left(0,1\right),$$

$$u_{\theta }^{-1}\left(\xi \right)={\left({\displaystyle \frac{e^{\xi }}{1+e^{\xi }}}\right)}^{1/\alpha },\xi \in \left(-\infty ,\infty \right),$$

where $\alpha =1/{\log }_2{\displaystyle \frac{1}{\theta }}$, $\theta \in \left(0,1\right)$.

For the other examples of the functions $u_{\theta }$ and $v_{\vartheta }$ see the paper [20] and the book [21].

3. Results

We start with the definition of logical distance based on the uninorm and absorbing norm and then present the suggested algorithm. In the last subsection we consider numerical simulations and compare the suggested algorithm with the known method.

3.1. Logical Distance Based on the Uninorm and Absorbing Norm

Let $x,y\in \left[0,1\right]$ be two truth values. Given neutral element $\theta$ and absorbing element $\vartheta$, fuzzy logical dissimilarity of the values $x$ and $y$ is defined as follows:

$${\mathrm{disim}}_{\theta ,\vartheta }\left(x,y\right)={\ominus }_{\theta }\left(\left(x{\oplus }_y\theta \right){\otimes }_{\vartheta }\left(\theta {\oplus }_{x}y\right)\right).$$

where the value

$${\mathrm{sim}}_{\theta ,\vartheta }\left(x,y\right)=\left(x{\oplus }_y\theta \right){\otimes }_{\vartheta }\left(\theta {\oplus }_{x}y\right)$$

is a fuzzy logical similarity of the values $x$ and $y$.

If $\theta =\vartheta$, we will write ${\mathrm{disim}}_{\theta }$ instead of ${\mathrm{disim}}_{\theta ,\vartheta }$ and ${\mathrm{sim}}_{\theta }$ instead of ${\mathrm{sim}}_{\theta ,\vartheta }$.

Lemma 1.

If $\theta =\vartheta$, then the function $disi{m}_{\theta }$ is a semi-metric in the algebra $\mathcal{A}$.

Proof. 

To prove the lemma, we need to check the following three properties:

a.

${\mathrm{disim}}_{\theta ,\vartheta }\left(x,x\right)=\theta =\vartheta$.

By commutativity and identity properties of the uninorm holds:

$$x{\oplus }_x\theta =\theta {\oplus }_{x}x=\theta .$$

Then, by the property of absorbing element holds

$$\theta {\otimes }_{\theta }\theta =\theta$$

and

$${\ominus }_{\theta }\theta =\theta .$$

b.

${\mathrm{disim}}_{\theta ,\vartheta }\left(x,y\right)>\theta =\vartheta$ if $x\ne y$.

If $x\ne y$, then either

$$x{\oplus }_y\theta >\theta \mathrm{and}\theta {\oplus }_{x}y<\theta$$

or

$$x{\oplus }_y\theta <\theta \mathrm{and}\theta {\oplus }_{x}y>\theta .$$

Thus,

$$\left(x{\oplus }_y\theta \right){\otimes }_{\theta }\left(\theta {\oplus }_{x}y\right)<\theta$$

and

$${\ominus }_{\theta }\left(\left(x{\oplus }_y\theta \right){\otimes }_{\theta }\left(\theta {\oplus }_{x}y\right)\right)>\theta .$$

c.

${\mathrm{disim}}_{\theta ,\vartheta }\left(x,y\right)={\mathrm{disim}}_{\theta ,\vartheta }\left(y,x\right)$.

It follows directly from the commutativity of the absorbing norm.

Lemma is proven. □

Using the dissimilarity function, we define the fuzzy logical distance ${\mathrm{dist}}_{\theta ,\vartheta }$ between the values $x,y\in \left[0,1\right]$ as follows ($\theta =\vartheta$):

$${\mathrm{dist}}_{\theta ,\vartheta }\left(x,y\right)={\displaystyle \frac{1}{\theta }}{\mathrm{disim}}_{\theta ,\vartheta }\left(x,y\right)-1.$$

Lemma 2.

If $\theta =\vartheta$, then the function $dis{t}_{\theta }$ is a semi-metric in the algebra of real numbers on the interval $\left[0, 1\right]$.

Proof. 

Since for any $x,y\in \left[0,1\right]$ and any $\theta =\vartheta \in \left[0,1\right]$ both $x{\oplus }_{\theta }y\in \left[0,1\right]$, $x{\otimes }_{\vartheta }y\in \left[0,1\right]$ and ${\ominus }_{\theta }x\in \left[0,1\right]$, and by the properties a and b of semi-metric, holds

$${\mathrm{disim}}_{\theta ,\vartheta }\left(x,y\right)\in \left[\theta ,1\right].$$

a.

If $x=y$, then ${\mathrm{disim}}_{\theta ,\vartheta }\left(x,y\right)=\theta =\vartheta$ and

$${\mathrm{dist}}_{\theta ,\vartheta }\left(x,y\right)=0.$$

b.

If $x\ne y$, then ${\mathrm{disim}}_{\theta ,\vartheta }\left(x,y\right)>\theta =\vartheta$. So ${\displaystyle \frac{1}{\theta }}{\mathrm{disim}}_{\theta ,\vartheta }\left(x,y\right)>1$ and

$${\mathrm{dist}}_{\theta ,\vartheta }\left(x,y\right)>0.$$

c.

Symmetry

$${\mathrm{dist}}_{\theta ,\vartheta }\left(x,y\right)={\mathrm{dist}}_{\theta ,\vartheta }\left(y,x\right)$$

follows directly from the symmetry of the dissimilarity ${\mathrm{disim}}_{\theta ,\vartheta }$. □

An example of the fuzzy logic distance ${\mathrm{dist}}_{\theta ,\vartheta }\left(x,y\right)$ between the values $x,y\in \left[0,1\right]$ with $\theta =\vartheta =0.5$ is shown in Figure 1.a. For comparison, Figure 1.b shows the Euclidean distance between the values $x,y\in \left[0,1\right]$.

It is seen that the fuzzy logical distance better separates the close values and less sensitive to the far values than the Euclidean distance.

Now let us extend the introduced fuzzy logical distance to multidimensional variables.

Let

$$\left(x_1,x_2,\dots ,x_n\right)\subset {\left[0, 1\right]}^d,n\ge 1,d\ge 1,$$

be $d$-dimensional vectors such that each vector $x_i=\left(x_{i1}, x_{i2},\dots ,x_{id}\right)$, $i=1, 2,\dots , n$, is a point in a $d$-dimensional space.

The fuzzy logical dissimilarity of the points $x_i$ and $x_j$, $i,j=1, 2,\dots , n$, is defined as follows:

$${\mathrm{disim}}_{\theta ,\vartheta }^d\left(x_i,x_j\right)={\ominus }_{\theta }\left\{{{\oplus }_{\theta }}_{l=1}^d\left[\left(x_{il}{\oplus }_{x_{jl}}\theta \right){\otimes }_{\vartheta }\left(\theta {\oplus }_{x_{il}}{x}_{jl}\right)\right]\right\},$$

where, as above, the value

$${\mathrm{sim}}_{\theta ,\vartheta }^d\left(x_i,x_j\right)={{\oplus }_{\theta }}_{l=1}^d\left[\left(x_{il}{\oplus }_{x_{jl}}\theta \right){\otimes }_{\vartheta }\left(\theta {\oplus }_{x_{il}}{x}_{jl}\right)\right]$$

is a fuzzy logical similarity of the points $x_i$ and $x_j$.

Let $\theta =\vartheta$. Then, as above, the fuzzy logical distance between the points $x_i$ and $x_j$, $i,j=1, 2,\dots , n$, is

$${\mathrm{dist}}_{\theta ,\vartheta }^d\left(x_i,x_j\right)={\displaystyle \frac{1}{\theta }}{\mathrm{disim}}_{\theta ,\vartheta }^d\left(x_i,x_j\right)-1.$$

Lemma 3.

If $\theta =\vartheta$, then the function $disi{m}_{\theta }^d$ is a semi-metric in the algebra $\mathcal{A}={\left[0, 1\right]}^d, {\oplus }_{\theta }, {\otimes }_{\vartheta }$, $d\ge 1$.

Proof. 

This statement is a direct consequence of lemma 1 and the properties of the uninorm.

a.

${\mathrm{disim}}_{\theta ,\vartheta }^d\left(x_i,x_j\right)=\theta =\vartheta$.

If $x_i=x_j$ then for each $l=1,2,\dots ,d$

$$\left(x_{il}{\oplus }_{x_{jl}}\theta \right){\otimes }_{\theta }\left(\theta {\oplus }_{x_{il}}{x}_{jl}\right)=\theta .$$

Then,

$${{\oplus }_{\theta }}_{l=1}^d\left[\left(x_{il}{\oplus }_{x_{jl}}\theta \right){\otimes }_{\vartheta }\left(\theta {\oplus }_{x_{il}}{x}_{jl}\right)\right]={{\oplus }_{\theta }}_{l=1}^d\theta =\theta$$

and finally

$${\ominus }_{\theta }\theta =\theta .$$

b.

${\mathrm{disim}}_{\theta ,\vartheta }^d\left(x_i,x_j\right)>\theta$ if $x\ne y$.

If $x_i\ne x_j$, then for each $l=1,2,\dots ,d$

$$\left(x_{il}{\oplus }_{x_{jl}}\theta \right){\otimes }_{\theta }\left(\theta {\oplus }_{x_{il}}{x}_{jl}\right)<\theta$$

Then,

$${{\oplus }_{\theta }}_{l=1}^d\left[\left(x_{il}{\oplus }_{x_{jl}}\theta \right){\otimes }_{\vartheta }\left(\theta {\oplus }_{x_{il}}{x}_{jl}\right)\right]<\theta$$

and

$${\ominus }_{\theta }\left\{{{\oplus }_{\theta }}_{l=1}^d\left[\left(x_{il}{\oplus }_{x_{jl}}\theta \right){\otimes }_{\vartheta }\left(\theta {\oplus }_{x_{il}}{x}_{jl}\right)\right]\right\}>\theta .$$

c.

${\mathrm{disim}}_{\theta ,\vartheta }^d\left(x_i,x_j\right)={\mathrm{disim}}_{\theta ,\vartheta }^d\left(x_j,x_i\right)$.

It follows directly from the symmetry of the dissimilarity for each $l=1,2,\dots ,d$ and commutativity of the uninorm. □

Lemma 4.

If $\theta =\vartheta$, then the function $dis{t}_{\theta ,\vartheta }^d$ is a semi-metric on the hypercube ${\left[0, 1\right]}^d$, $d\ge 1$.

Proof. 

The proof is literally the same as the proof of lemma2. □

The suggested algorithm uses the introduced function ${\mathrm{dist}}_{\theta ,\vartheta }^d$ as a distance measure between the instances of the data.

3.2. The c-Means Algorithm with Fuzzy Logical Distance

The suggested algorithm considers the instances of the data as truth values and uses Algorithm 1 with the fuzzy logical distance ${\mathrm{dist}}_{\theta ,\vartheta }^d$ on these values.

As above, assume that the raw data is represented by the $d$-dimensional vectors

$$X=\left(x_1,x_2,\dots ,x_n\right)\subset {ℝ}^d,$$

where $x_i=\left(x_{i1},x_{i2},\dots ,x_{id}\right)$, $i=1,2,\dots ,n$, $n\ge 1$, $d\ge 1$, is a data instance.

Since function ${\mathrm{dist}}_{\theta ,\vartheta }^d$ requires the values from the hypercube ${\left[0,1\right]}^d$, $\mathrm{d}\ge 1$, vector $X$ must be normalized

$$\tilde{X}=\mathrm{norm}\left(X\right),X\subset {ℝ}^d,\tilde{X}\subset {\left[0,1\right]}^d,$$

and the algorithm should be applied to the normalized data vector $\tilde{X}$. After definition of the cluster centers $\tilde{Y}$, the inverse normalization must be applied to the vector $\tilde{Y}$

$$Y={\mathrm{norm}}^{-1}\left(\tilde{Y}\right),\tilde{Y}\subset {\left[0,1\right]}^d,Y\subset {ℝ}^d.$$

Normalization can be conducted by several methods; in Appendix A we present a simple Algorithm 3 of normalization by linear transformation. The inverse transformation is provided by the Algorithm 4 also presented in Appendix A.

In general, the suggested Algorithm 2 follows the Bezdek fuzzy c-means algorithm 1, however differs in the distance function and in the initialization of the cluster centers ${\tilde{y}}_j$, $j=1, 2,\dots , m$, and, consequently, – in the definition of the number of clusters.

The main difference between the suggested Algorithm 2 and the original Bezdek fuzzy c-means Algorithm 1 [13,14] and the known Gustafson-Kessel [16] and Gath-Geva [17] is the use of the fuzzy logical distance ${\mathrm{dist}}_{\theta ,\vartheta }^d$. The use of this distance requires normalization and renormalization of the data.

The other difference is in the need of the initialization of the cluster centers as a grid in the algorithm’s domain. Such initialization is required because of quick convergence of the algorithm; thus, the even distribution of the initial cluster centers avoids missing the clusters. A simple algorithm 5 for creating a grid in the square ${\left[0,1\right]}^2$ is outlined in Appendix A.

Let us consider two main properties of the suggested algorithm.

Theorem 1. 

Algorithm 2 converges.

Proof. 

Convergence of the algorithm 2 follows directly from the fact that ${\mathrm{dist}}_{\theta ,\vartheta }^d$ is a semi-metric (see lemma 4).

In fact, in the lines 3 and 7 of the algorithm holds

$$0<{\mu }_{ij}\le 1,$$

$${\mathrm{dist}}_{\theta ,\vartheta }^d\left(x_i,x_j\right)<{\mathrm{dist}}_{\theta ,\vartheta }^d\left(x_p,x_q\right)\Rightarrow {\mu }_{ij}<{\mu }_{pq},$$

and the algorithm converges. □

Theorem 2.

Time complexity of the Algorithm 2 is $\mathcal{O}\left(dmnk\right)$, where $d$is the dimensionality of the data, $m$is the number of clusters, $n$is the number of instances in the data and $k$is the number of iterations.

Proof. 

At first, consider calculation of the fuzzy logical distances ${\mathrm{disim}}_{\theta ,\vartheta }^d\left(x_i,y_j\right)$, $i=1, 2, \dots , n$, $j=1, 2, \dots , m$. Complexity of this operation is $\mathcal{O}\left(1\right)$ for each dimension $l=1, 2, \dots , d$; thus, calculation of the distances has a complexity $\mathcal{O}\left(d\right)$.

Now let us consider the lines of the algorithm. Normalization of the data vector (line 1) requires $\mathcal{O}\left(dn\right)$ steps and initialization of the cluster centers (line 2) requires $\mathcal{O}\left(mn\right)$ steps (see Algorithm 3 in Appendix A).

Initialization of the membership degrees (line 3) requires $\mathcal{O}\left(mn\right)$ steps for each dimension that gives $\mathcal{O}\left(dmn\right)$.

In the do-while loop, saving the membership degrees (line 5) requires $\mathcal{O}\left(mn\right)$, calculation of the cluster centers given the membership degrees (line 6) requires $\mathcal{O}\left(mn\right)$ steps and calculation of the membership degrees (line 7) requires, as above, $\mathcal{O}\left(mn\right)$ steps for each dimension that gives $\mathcal{O}\left(dmn\right)$.

Finally, renormalization of the vector of the cluster centers (line 9) requires $\mathcal{O}\left(dn\right)$ steps.

Thus, initial (lines 1-3) and final (9) operations of the algorithm require

$$\mathcal{O}\left(dn\right)+\mathcal{O}\left(mn\right)+\mathcal{O}\left(dmn\right)+\mathcal{O}\left(dn\right)=\mathcal{O}\left(dmn\right).$$

steps and each iteration requires

$$\mathcal{O}\left(mn\right)+\mathcal{O}\left(mn\right)+\mathcal{O}\left(mn\right)+\mathcal{O}\left(dmn\right)=\mathcal{O}\left(dmn\right).$$

Then, for $k$ iterations it is required

$$\mathcal{O}\left(dmn\right)+k\mathcal{O}\left(dmn\right)=\mathcal{O}\left(dmnk\right)$$

steps. □

Note that in the considerations above we assumed that $\theta =\vartheta$ which supported the semi-metric properties of the function ${\mathrm{dist}}_{\theta ,\vartheta }^d$. Along with that, in practice these parameters can differ and, despite the absence of formal proofs, the use of the function ${\mathrm{dist}}_{\theta ,\vartheta }^d$ with $\theta \ne \vartheta$ can provide essentially better clustering.

3.3. Numerical Simulations

In the first series of simulations, we will demonstrate that the suggested Algorithm 2 with fuzzy logical distance results in more precise centers of the clusters than the original Algorithm 1 with Euclidean distance.

For this purpose, in the simulations we generate a single cluster as normally distributed instances with the known center and apply Algorithms 1 and 2 to these data. As a measure of the quality of the algorithms we use the mean squared error (MSE) in finding the center of the cluster center and the means of standard deviations of the calculated clusters centers.

To avoid the influence of the instance values, in the simulations we considered the normalized data. An example of the data (white circles) and the results of the algorithms (gray diamonds for Algorithm 1 and black pentagrams for Algorithm 2) are shown in Figure 2.

The simulations were conducted with different numbers of clusters by the series of $100$ trials. The results were tested and compared by the one-sample and two-sample Student’s $t$-tests.

In the simulations we assumed that the algorithm which for a single cluster provides less deviated cluster centers is more precise in the calculation of the cluster centers. In other words, the algorithm which results in the clusters centers concentrated near the actual cluster center is better than the algorithm which results in more dispersed cluster centers.

Results of simulations are summarized in Table 1. In the table, we present the results of clustering of two-dimensional data distributed around the mean $\mu =0.5$.

It is seen that both algorithms result in cluster centers close to the actual cluster center and the errors in calculating the centers are extremely small. Additional statistical testing by Student’s $t$-test demonstrated that the differences between the obtained clusters centers are not significant with $\alpha =0.95$.

Along with that, the suggested Algorithm 2 results in smaller standard deviation than the Algorithm 1 and this difference is significant with $\alpha =0.95$. Hence, the suggested Algorithm 2 results in more precise cluster centers than Algorithm 1.

To illustrate these results, let us consider simulations of the algorithms on the data with several predefined clusters.

Consider application of Algorithms 1 and 2 to the data with two predefined clusters with the centers in the points $\left(0.3, 0.3\right)$ and $\left(0.7, 0.7\right)$. The resulting cluster centers with $m=25$ and $m=9$ clusters are shown in Figure 3. Notation in the figure is the same as in Figure 2.

It is seen that clusters centers obtained by Algorithm 2 (black pentagrams) are concentrated closer to the actual cluster centers than the cluster centers obtained by Algorithm 1 (gray diamonds). Moreover, some of the cluster centers obtained by Algorithm 2 are located in the same points while all cluster centers obtained by Algorithm 1 are located in different points.

More clearly the observed effect is seen on the data with several clusters. Figure 4 shows the results of Algorithms 1 and 2 applied to the data with $10$ predefined clusters.

It is seen that the cluster centers calculated by Algorithm 2 are concentrated at the real centers of the clusters and, as above, several centers are located at the same points. Hence, the suggested Algorithm 2 allows more correct definition of the cluster centers and consequently, more correct clustering.

To illustrate the usefulness of the suggested algorithm in recognition of the number of clusters and analysis of the data structure, let us compare the results of the algorithm with the results obtained by the MATLAB^® fcm function [15] with three possible distance measures: Euclidean distance, Mahalanobis distance [16] and exponential distance [17]. These algorithms were applied to $n=200$ data instances distributed around $5$ centers; the obtained results are shown in Figure 5.

It is seen that the suggested algorithm (Figure 5.a) correctly recognizes the real centers of the clusters and locates the cluster centers close to these real centers. In contrast, the known algorithms implemented in MATLAB^® do not recognize real centers of the clusters and, consequently, do not define correct number of the clusters and locations of their centers. The function fcm with Euclidean and Mahalanobis distance measures (Figure 5.b,c) results in two cluster centers and the function fcm with exponential distance (Figure 5.d) results in three cluster centers. Note that the use of Euclidean and Mahalanobis distance measures leads to very similar results.

Thus, recognition of the real cluster centers which are the centers of distributions of the data instances can be conducted in two stages. At first, the cluster centers are defined by application of the suggested algorithm to the raw data, and at second, the cluster centers by application of k-means to the cluster centers found at the first stage. The resulting cluster centers indicate the real cluster centers.

4. Discussion

The suggested algorithm of fuzzy clustering follows the line of c-means fuzzy clustering algorithms and differs from the known methods in the used distance measure.

The suggested fuzzy logical distance is a semi-metric based on the introduced semi-metric in the algebra of truth values with uninorm and absorbing norm. The meaning of these semi-metrics is the following.

Assume that some statements $A$ and $B$ are considered by a group of $d$ observers and each of the observers expresses an opinion about the truthiness of these statements. Assuming that the observers are independent, the statements $A$ and $B$ can be considered as point in the $d$-dimensional space and then the suggested semi-metric is a distance between these statements.

In other words, the fuzzy logical distance measures allow comparing the statements based on their subjective truthiness.

In the paper, we considered the fuzzy logical distance based on the uninorm and absorbing norm which, in their turn, use the sigmoid generating functions. As a result, the fuzzy logical distance effectively separates the data instances which leads to more precise calculation of the cluster centers and more quick convergence of the algorithm.

An additional advantage of the algorithm is the possibility of tuning its activity by two parameters: neutral element $\theta$ and absorbing element $\vartheta$. As indicated above, better results of the clustering can be obtained using non-equal values $\theta$ and $\vartheta$. An inequality of these parameters slightly disturbs the semi-metric properties of the distance measures, however, leads to better separation of close but different points.

The weakness of the algorithm is the need of normalization of the data and then renormalization of the obtained clusters centers. However, since both operations are conducted in polynomial time, this disadvantage is not a serious drawback.

The suggested algorithm of fuzzy clustering can be used for solving the clustering problems instead of or together with the known algorithms, for recognition of the centers of distributions of the data instances and can form a basis for development of the methods of comparison of multivariate samples.